[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

Ruby, Analysis, and Tons of RAM

ben

9/23/2006 11:23:00 AM

Does anyone have experience with using Ruby for analysis (*lots* of
maths), on a machine with a ridiculous amount of RAM? For example, a
hip 64-bit Linux kernel on a machine with 32 or 64 GB of physical RAM.

Are there any "gotchas" I should be aware of? Would all the RAM be
addressable by a given Ruby process? Or would I still have to be
forking a number of processes, each allocated a bit of the address
space (blech)?

Thanks, oh Ruby masters.
11 Answers

MonkeeSage

9/23/2006 2:21:00 PM

0

ben@somethingmodern.com wrote:
> Does anyone have experience with using Ruby for analysis (*lots* of
> maths), on a machine with a ridiculous amount of RAM? For example, a
> hip 64-bit Linux kernel on a machine with 32 or 64 GB of physical RAM.

No...but, I'm always willing to help out! Just send me one such
workstation and I'll send you the results post haste! ;)

Regards,
Jordan

M. Edward (Ed) Borasky

9/23/2006 3:51:00 PM

0

MonkeeSage wrote:
> ben@somethingmodern.com wrote:
>> Does anyone have experience with using Ruby for analysis (*lots* of
>> maths), on a machine with a ridiculous amount of RAM? For example, a
>> hip 64-bit Linux kernel on a machine with 32 or 64 GB of physical RAM.
>
> No...but, I'm always willing to help out! Just send me one such
> workstation and I'll send you the results post haste! ;)

I have lots of experience doing that sort of thing, but none of it is in
Ruby. You can contact me off list for some ideas.

Logan Capaldo

9/23/2006 4:17:00 PM

0

On Sat, Sep 23, 2006 at 08:25:18PM +0900, ben@somethingmodern.com wrote:
> Does anyone have experience with using Ruby for analysis (*lots* of
> maths), on a machine with a ridiculous amount of RAM? For example, a
> hip 64-bit Linux kernel on a machine with 32 or 64 GB of physical RAM.
>
> Are there any "gotchas" I should be aware of? Would all the RAM be
> addressable by a given Ruby process? Or would I still have to be
> forking a number of processes, each allocated a bit of the address
> space (blech)?
>
Not having done this myself, you should take everything I say with a
grain of salt, but since ruby allocations (eventually) use malloc, how
much of this massive address space it gets and all that jazz strike be
as being something that is entirely up to the operating system.
(Excepting things in C extensions which may use mmap or whatever.)
> Thanks, oh Ruby masters.
>

Ara.T.Howard

9/23/2006 4:43:00 PM

0

M. Edward (Ed) Borasky

9/23/2006 5:02:00 PM

0

ben@somethingmodern.com wrote:
> Does anyone have experience with using Ruby for analysis (*lots* of
> maths), on a machine with a ridiculous amount of RAM? For example, a
> hip 64-bit Linux kernel on a machine with 32 or 64 GB of physical RAM.
>
> Are there any "gotchas" I should be aware of? Would all the RAM be
> addressable by a given Ruby process? Or would I still have to be
> forking a number of processes, each allocated a bit of the address
> space (blech)?
>
> Thanks, oh Ruby masters.

1. Again, you can contact me off list for some ideas ... without knowing
your goal, it's difficult for me to know what steps you should take to
reach it.

2. Assume a properly working state-of-the-art 64-bit dual-core AMD or
Intel hardware platform with 64 GB of RAM and an appropriate SAN for
storage from a major vendor like IBM. That severely limits your OS
choices; last time I looked you need to be running either RHEL or SUSE
Enterprise Linux. I don't know about the other vendors, but IBM has a
marvelous document on performance tuning humongous servers at

http://www.redbooks.ibm.com/redbooks/pdfs/sg...

3. OK, now you've purchased a high-end server and a *supported*
enterprise-grade Linux, and you want to do some serious number crunching
on it, and you want to do it in Ruby, possibly augmented by libraries in
C, Fortran or assembler for speed. You will need to recompile
*everything* -- Ruby, the math libraries, and the compiler itself -- to
use 64-bit addressing. There are some hacks and workarounds, but pretty
much this is required. If you end up with an Intel server, you might
want to have a look at the Intel compilers instead of GCC. Intel also
has some highly-tuned math libraries, as does AMD.

My point here is that you are "exploring" realms in Ruby that are
"usually" addressed using "more traditional" techniques, so you're going
to need to do a fair amount of testing. That kind of server costs a lot
of money, and for that kind of money, you'll get lots of support from
the vendor, coupled with strong incentives to do your job in ways that
are tested and proven to work and supported by said vendor.That may or
may not include Ruby, and if it does include Ruby, it may or may not
involve a small number of monolithic Ruby scripts directly addressing a
large address space.

There is a lot of help available on the Internet from people like me who
love challenges like this. :)

M. Edward (Ed) Borasky

9/23/2006 5:15:00 PM

0

ara.t.howard@noaa.gov wrote:
> On Sat, 23 Sep 2006 ben@somethingmodern.com wrote:
>
>> Does anyone have experience with using Ruby for analysis (*lots* of
>> maths), on a machine with a ridiculous amount of RAM? For example, a
>> hip 64-bit Linux kernel on a machine with 32 or 64 GB of physical RAM.
>
> i've had issues using mmap with files larger than 32gb - i'm not sure if
> the
> latest release has fixed this or not... in general you can run into issues
> with extenstions since ruby fixnums keep a bit to mark them as objects...
>
>> Are there any "gotchas" I should be aware of? Would all the RAM be
>> addressable by a given Ruby process? Or would I still have to be
>> forking a
>> number of processes, each allocated a bit of the address space (blech)?
>
> assuming you have two or four cpus this might not be a bad idea - ipc is so
> dang easy with ruby it's trivial to coordinate processes. i have a slave
> class i've used for this before:
>
> http://codeforpeople.com/lib/r...
> http://codeforpeople.com/lib/r...slave-0.0.1/README
>
> regards.
>
>
> -a

Ah, someone *has* done some of this! What compiler did you use to
recompile Ruby for 64-bit addressing? Did it work out of the box?

What's the bottleneck in Ruby's built-in IPC? Network traffic to
"localhost" and to the other hosts? System V IPC? Something else?

I haven't really looked at the whole "lots of coordinated tiny
processes" thing in Ruby, since Erlang seems to have nailed that
approach and made it the core of the Erlang way to do things. I'm not a
big fan of re-inventing wheels; I'd much rather just get my numbers
crunched.

Joel VanderWerf

9/23/2006 8:07:00 PM

0

ara.t.howard@noaa.gov wrote:
> http://codeforpeople.com/lib/r...
> http://codeforpeople.com/lib/r...slave-0.0.1/README

Ara, why does Slave.new keep a copy of the object (the one you want to
be served up) on both sides of the fork?

Wouldn't it make more sense for it to work like this (modifying the
example in slave.rb):

class Server
def add_two n
n + 2
end
end

slave = Slave.new {Server.new} # <-- note addition of {...}
server = slave.object

p server.add_two(40) #=> 42

Slave.new would call the block _only_ in the child, and the parent would
never have an instance of Server, only the drb handle to it.

This might matter if Server.new consumes resources, sets up a data
structure in memory, opens files, etc.

--
vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

Ara.T.Howard

9/23/2006 10:22:00 PM

0

John Carter

9/26/2006 9:05:00 PM

0

Wakalukong

3/15/2012 10:43:00 PM

0

On Feb 21, 6:13 am, Wakalukong <wakaluk...@yahoo.com.sg> wrote:
> On Feb 19, 2:56 pm, "Muhammed Amin" <Muhammed A...@gmail.com> wrote:> Malaysia as a Nation cannot afford to have fights between the races. I hope
> > our leaders will be sensible enough to realise that. In the day and age it
> > does not take too many people to create strive within a Nation. Having the
> > various races within the country at odds with one another is not the
> > solution to a harmonious society. Much as we not like it, Malaysia is a
> > multi racial society. Nothing can change that fact and it is impossible to
> > chase out all the other races out of the country. If America with so many
> > races in the country can survive as a nation, so can Malaysia. So stop
> > preaching hatred amongst the various races. Every one irrespective of race,
> > colour, creed or religion just want to make a descent living peacefully.. Any
> > leader who preaches hatred amongst the races should be locked up. There is
> > no place for such people who promote mayhem.
(snip)
------------

Well said. Dogs not allowed; dog Komin also not allowed.

Wakalukong