jason.cipriani@gmail.com
12/5/2008 6:49:00 PM
On Dec 5, 10:49 am, Maxim Yegorushkin <maxim.yegorush...@gmail.com>
wrote:
> On Dec 5, 1:01 pm, fabian....@gmail.com wrote:
>
> > Hi all,
>
> > Im struggling with a memory allocation problem. I need to keep track
> > of the history of my computations, and I do so by filling up a growing
> > vector of elements, resizing it everytime. At some point
> > std::allocator will throw a std::bad_alloc exception, when it does the
> > array will be about size 40,000. Are they are ways to increase the
> > amount of memory my program can use?
>
> There certainly are. You may like to tell us a bit more about your
> problem.
>
> What is you platform/environment?
>
> What is the element type of your vector? 40000 elements does not seem
> to be a lot, but it depends on the size of the element.
Yes, this does not seem like a lot. More details are in order.
You could reserve() before resize()ing and you won't run the risk of
allocating twice as much space as you need, but you lose the
performance benefits of exponentially growing the capacity on resize
().
You could use a list instead of a vector, perhaps a highly fragmented
heap is leaving you without a large enough single block of memory, a
list has a bit more overhead (although from what I'm inferring about
the sizes of your elements it's probably negligible) but doesn't
require contiguous blocks.
You could handle a bad_alloc by committing the current contents to the
file system, for example, and then clearing them to free memory.
Although, unless your swap file is too small (or you're using
incredible amounts of memory), the OS should basically be doing this
for you anyways.
You could queue the data you're storing and have a second thread
running and committing it to disk in the background, this may or may
not give good performance depending on the nature of your
"calculations".
You could use a SQL database to store data in. A decent DBMS is
optimized for high performance, and while it's certainly slower than
sticking things in a vector, there may be useful benefits (perhaps
combined with commit on bad_alloc strategy). It may likely be faster
than committing to disk on your own as well. I'm not pushing that as a
solution, just illustrating that there are other options.
You could reduce the memory usage by "compressing" the data somehow,
depending on the nature of your computations. For example, if you are
performing computations on 100,000 objects, but typically only a small
fraction of those are actually modified each iteration, then you do
not need to store the complete state of every object each time -- you
can store information only about what has changed since the last
iteration. I've used techniques like this (+ asynchronous commit to
file system) before for recording and playback of massive amounts of
real-time data with great success -- the disadvantage, of course, is
that to reconstruct a specific frame you must analyze all frames
before it (but there are techniques to reduce this problem as well).
There's a lot of alternatives with a wide range of complexity. What
kind of computations are you performing? What is an "element"?
Jason