Eric Hodel
6/30/2008 7:07:00 AM
On Jun 29, 2008, at 23:58 PM, Greg Willits wrote:
> I have a pure Ruby project (no Rails) where I would like multiple
> "tasks" (ruby processes more or less) to run in parallel (collectively
> taking advantage of multiple CPU cores) while accessing a shared
> memory
> space of data structures.
>
> OK, that's a mouthful.
>
> - single machine, multiple cores (4 or 8)
>
> - step one: pre-load a number of arrays and hashes (could be a
> couple GB
> worth in total) into memory
>
> - step two: launch several independent Ruby scripts to search and read
> from the data pool in order to aggregate data in new sets to be
> written
> to text files.
>
> Ruby 1.8's threading would seem poorly suited to this. Can 1.9 run
> multiple threads each accesing the same RAM-space while using all
> cores
> of the machine?
At present, 1.9 has a global VM lock, so only one C thread can be
running ruby code at a time.
> I've looked at memcache, but it seems like it could store and retrieve
> one of my pool's arrays, but it cannot look inside that array and
> retrieve just a single row of it? It would want to return the whole
> array, yes? (not good if that array is 100MB).
memcache is just a cache and not designed to be used as a persistent
store. It may loose your data if you are not careful.
You're probably looking for something mmap and several forked
cooperative processes.