Asp Forum - memory pool?

Uli Kunkel

11/10/2008 9:06:00 AM

I have a situation where I'm getting a picture from a camera in small
chunks.
So the photo is a few MBytes and I have a callback function that returns
1KB until it finishes transfering the picture.
Any suggestions or examples how should I put a picture in one variable?
I heard something about using memory pools for this but I only need
something simple for this occasion.
Could I use CString?

Anyway thanks in advance for any suggestions.

21 Answers

pjb

11/10/2008 9:55:00 AM

Uli Kunkel <genijalac@yahoo.com> writes:

> I have a situation where I'm getting a picture from a camera in small
> chunks.
> So the photo is a few MBytes and I have a callback function that
> returns 1KB until it finishes transfering the picture.
> Any suggestions or examples how should I put a picture in one variable?
> I heard something about using memory pools for this but I only need
> something simple for this occasion.
> Could I use CString?

Not on comp.lang.c++. For CString, you should ask on comp.lang.c.

> Anyway thanks in advance for any suggestions.

typedef unsigned char byte;
const int KByte=1024;
const int MByte=1024*KByte;
const int few=6;
std::vector<byte> image(few*MByte,0);

Then you can receive copy small buffers into it:

std::vector<byte>::iterator pos=image.begin();
std::vector<byte> buffer(1*KByte);

while(receive(&buffer)){
pos=std::copy(buffer.begin(),buffer.end(),pos);
}

--
__Pascal Bourguignon__

Juan Antonio Zaratiegui Vallecillo

11/10/2008 10:03:00 AM

Uli Kunkel escribió:
> I have a situation where I'm getting a picture from a camera in small
> chunks.
> So the photo is a few MBytes and I have a callback function that returns
> 1KB until it finishes transfering the picture.
> Any suggestions or examples how should I put a picture in one variable?
> I heard something about using memory pools for this but I only need
> something simple for this occasion.
> Could I use CString?
>
> Anyway thanks in advance for any suggestions.

Use a vector, taking advantage from its reserve() function.

#include <vector>

(...)

std::vector<unsigned char> myPhoto;
myPhoto.reserve(1024*1024);

for (size_t part=0; part !=1024;++part) {
<read image data into &myPhoto(part*1024) >
}
(...)
<you now have your photo in myPhoto.size() sequential bytes at
&myPhoto[0]

Just customize such piece for your exact needs, substsituting all magic
constants with nice real constants.
This code skeleton takes as granted that char size is one byte, you may
need to correct the code it it is otherwise

best regards,

Zara

Hendrik Schober

11/10/2008 11:46:00 AM

Uli Kunkel

11/10/2008 11:49:00 AM

Juan Antonio Zaratiegui Vallecillo wrote:
> Uli Kunkel escribió:
>> I have a situation where I'm getting a picture from a camera in small
>> chunks.
>> So the photo is a few MBytes and I have a callback function that
>> returns 1KB until it finishes transfering the picture.
>> Any suggestions or examples how should I put a picture in one variable?
>> I heard something about using memory pools for this but I only need
>> something simple for this occasion.
>> Could I use CString?
>>
>> Anyway thanks in advance for any suggestions.
>
>
> Use a vector, taking advantage from its reserve() function.
>
> #include <vector>
>
> (...)
>
> std::vector<unsigned char> myPhoto;
> myPhoto.reserve(1024*1024);
>
> for (size_t part=0; part !=1024;++part) {
> <read image data into &myPhoto(part*1024) >
> }
> (...)
> <you now have your photo in myPhoto.size() sequential bytes at
> &myPhoto[0]
>
>
> Just customize such piece for your exact needs, substsituting all magic
> constants with nice real constants.
> This code skeleton takes as granted that char size is one byte, you may
> need to correct the code it it is otherwise
>
> best regards,
>
> Zara

Thanks for the answers.
I'm wondering if I could use reserve to allocate a few MB and then if
the size is not enough that the vector grows automatically?
I read that vector resizing is very costly so I presume that reserving
just 1KB is not enough.

peter koch

11/10/2008 12:22:00 PM

On 10 Nov., 12:49, Uli Kunkel <genija...@yahoo.com> wrote:
> Juan Antonio Zaratiegui Vallecillo wrote:
>
>
>
>
>
> > Uli Kunkel escribió:
> >> I have a situation where I'm getting a picture from a camera in small
> >> chunks.
> >> So the photo is a few MBytes and I have a callback function that
> >> returns 1KB until it finishes transfering the picture.
> >> Any suggestions or examples how should I put a picture in one variable?
> >> I heard something about using memory pools for this but I only need
> >> something simple for this occasion.
> >> Could I use CString?
>
> >> Anyway thanks in advance for any suggestions.
>
> > Use a vector, taking advantage from its reserve() function.
>
> > #include <vector>
>
> > (...)
>
> > std::vector<unsigned char> myPhoto;
> > myPhoto.reserve(1024*1024);
>
> > for (size_t part=0; part !=1024;++part) {
> > <read image data into &myPhoto(part*1024) >
> > }
> > (...)
> > <you now have your photo in myPhoto.size() sequential bytes at
> > &myPhoto[0]
>
> > Just customize such piece for your exact needs, substsituting all magic
> > constants with nice real constants.
> > This code skeleton takes as granted that char size is one byte, you may
> > need to correct the code it it is otherwise
>
> > best regards,
>
> > Zara
>
> Thanks for the answers.
> I'm wondering if I could use reserve to allocate a few MB and then if
> the size is not enough that the vector grows automatically?
> I read that vector resizing is very costly so I presume that reserving
> just 1KB is not enough.
std::vector resizes autmatically. As an alternative, you could also
use std::deque which memorywise is more optimal with regard to
reallocation but overall is slightly slower (does more allocations)
than a std::vector.

/Peter

Hendrik Schober

11/10/2008 12:58:00 PM

Uli Kunkel wrote:
> [...]
> I'm wondering if I could use reserve to allocate a few MB and then if
> the size is not enough that the vector grows automatically?
> I read that vector resizing is very costly so I presume that reserving
> just 1KB is not enough.

There's a difference between a vector's size (member functions
'size()'/'resize()') and its capacity ('capacity()'/'reserve()').
The former is related to the number of actual objects in the
vector, the latter to the number of objects the vector is able
to store without having to reallocate.
When you tell a vector to reserve memory for an amount of
elements, it allocates the memory needed (or more), but does
not create the objects. When you tell it to resize to a
specific amount of objects, it will actually contain this
amount of constructed objects (while its capacity might be
higher). The capacity concept allows to write code adding
objects to a vector in the most obvious way (keep appending)
and, if performance or memory fragmentation etc. are issues,
reserve before-hand to reduce or eliminate reallocations. It's
only there for performance-reasons and (besides performance)
has no influence on your algorithm.

So when you know how many objects you will need before-hand,
reserve that amount and then keep appending incoming objects.
If not, strategies depend on your specific problem.

Schobi

Uli Kunkel

11/10/2008 2:49:00 PM

Pascal J. Bourguignon wrote:
> Uli Kunkel <genijalac@yahoo.com> writes:
>
>> I have a situation where I'm getting a picture from a camera in small
>> chunks.
>> So the photo is a few MBytes and I have a callback function that
>> returns 1KB until it finishes transfering the picture.
>> Any suggestions or examples how should I put a picture in one variable?
>> I heard something about using memory pools for this but I only need
>> something simple for this occasion.
>> Could I use CString?
>
> Not on comp.lang.c++. For CString, you should ask on comp.lang.c.
>
>
>> Anyway thanks in advance for any suggestions.
>
>
> typedef unsigned char byte;
> const int KByte=1024;
> const int MByte=1024*KByte;
> const int few=6;
> std::vector<byte> image(few*MByte,0);
>
> Then you can receive copy small buffers into it:
>
>
> std::vector<byte>::iterator pos=image.begin();
> std::vector<byte> buffer(1*KByte);
>
> while(receive(&buffer)){
> pos=std::copy(buffer.begin(),buffer.end(),pos);
> }
>
>

I wrote some code and I have a problem that I can't open the picture.
Here is the simplified code:
----------------------------
//This part of the code pushes back every character to myPhoto vector

for(unsigned long i=0; i < Length; i++)
{
myPhoto.push_back(srcData[i]);
}
if(complete) //Write myPhoto vector to a file
{
ofstream outf("C:\\test_0.jpg");
for ( pos=myPhoto.begin()
; pos!=myPhoto.end()
; ++pos)
{
outf << *pos;
}
outf.close();
}

//This part of the code append every chunk to the file
//It's for testing only
ofstream myFile ("C:\\test_1.jpg", ios::out | ios::binary | ios::app);
myFile.write ((char *)pProgress->pbData, pProgress->lLength);
myFile.close();
-----------------------------
What happens is that the first photo has 8 or 9 KB more than the second
one and it can't be opened.
The second picture is ok but I don't want to do it this way.
Also I opened them and they seem the samo at the begining but the first
one is larger..

I tried 2 different ways to writing them but with the same effect.

Any suggestions?

Rolf Magnus

11/10/2008 3:15:00 PM

Uli Kunkel wrote:

> I wrote some code and I have a problem that I can't open the picture.
> Here is the simplified code:
> ----------------------------
> //This part of the code pushes back every character to myPhoto vector
>
> for(unsigned long i=0; i < Length; i++)
> {
> myPhoto.push_back(srcData[i]);
> }
> if(complete) //Write myPhoto vector to a file
> {
> ofstream outf("C:\\test_0.jpg");
> for ( pos=myPhoto.begin()
> ; pos!=myPhoto.end()
> ; ++pos)
> {
> outf << *pos;
> }
> outf.close();
> }
>
> //This part of the code append every chunk to the file
> //It's for testing only
> ofstream myFile ("C:\\test_1.jpg", ios::out | ios::binary | ios::app);
> myFile.write ((char *)pProgress->pbData, pProgress->lLength);
> myFile.close();
> -----------------------------
> What happens is that the first photo has 8 or 9 KB more than the second
> one and it can't be opened.

The first file is opened in text mode, while the second is opened in binary
mode. Depending on the platform you're using, that means that the first file
might get altered in some way while being written.

Uli Kunkel

11/10/2008 3:24:00 PM

Rolf Magnus wrote:
> Uli Kunkel wrote:
>
>> I wrote some code and I have a problem that I can't open the picture.
>> Here is the simplified code:
>> ----------------------------
>> //This part of the code pushes back every character to myPhoto vector
>>
>> for(unsigned long i=0; i < Length; i++)
>> {
>> myPhoto.push_back(srcData[i]);
>> }
>> if(complete) //Write myPhoto vector to a file
>> {
>> ofstream outf("C:\\test_0.jpg");
>> for ( pos=myPhoto.begin()
>> ; pos!=myPhoto.end()
>> ; ++pos)
>> {
>> outf << *pos;
>> }
>> outf.close();
>> }
>>
>> //This part of the code append every chunk to the file
>> //It's for testing only
>> ofstream myFile ("C:\\test_1.jpg", ios::out | ios::binary | ios::app);
>> myFile.write ((char *)pProgress->pbData, pProgress->lLength);
>> myFile.close();
>> -----------------------------
>> What happens is that the first photo has 8 or 9 KB more than the second
>> one and it can't be opened.
>
> The first file is opened in text mode, while the second is opened in binary
> mode. Depending on the platform you're using, that means that the first file
> might get altered in some way while being written.
>

That was the problem.
I didn't pay much attention on that part..
Thanks for the help.

Juan Antonio Zaratiegui Vallecillo

11/10/2008 4:01:00 PM

Juan Antonio Zaratiegui Vallecillo escribiÃ³:
> Uli Kunkel escribiÃ³:
>> I have a situation where I'm getting a picture from a camera in small
>> chunks.
>> So the photo is a few MBytes and I have a callback function that
>> returns 1KB until it finishes transfering the picture.
>> Any suggestions or examples how should I put a picture in one variable?
>> I heard something about using memory pools for this but I only need
>> something simple for this occasion.
>> Could I use CString?
>>
>> Anyway thanks in advance for any suggestions.
>
>
> Use a vector, taking advantage from its reserve() function.
>
> #include <vector>
>
> (...)
>
> std::vector<unsigned char> myPhoto;
> myPhoto.reserve(1024*1024);
>
> for (size_t part=0; part !=1024;++part) {
> <read image data into &myPhoto(part*1024) >
ThankÅ? to all for the comments.

I really meant to say
<append imaged data into myPhoto.end()>
But it was too early in the morning

> }
> (...)
> <you now have your photo in myPhoto.size() sequential bytes at
> &myPhoto[0]
>
>
(...)

And I keep on favouring the use of reserve, as it will be surely faster
than letting the container resize as needed, because we already know the
final size.

Best regards,

Zara

PS: Any error in this new message,, it is because it is too late in the
afternoon ;-)

comp.lang.c++

memory pool?

Uli Kunkel

pjb

Juan Antonio Zaratiegui Vallecillo

Hendrik Schober

Uli Kunkel

peter koch

Hendrik Schober

Uli Kunkel

Rolf Magnus

Uli Kunkel

Juan Antonio Zaratiegui Vallecillo

x Login to ForumsZone