[lnkForumImage]
TotalShareware - Download Free Software

Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

efficient access to a large int32 array

Martin DeMello

1/16/2007 2:11:00 PM

I have a file that is a dump of a large array of 32-bit unsigned ints
(bitvectors actually). All I need is to load this into memory and have
fast, efficient access to the array members, and a few bitmasking
operations - the array will never be written to, and ideally should
not need to be bounds-checked. Should I be looking at something like
mmap or narray to do this, or would I be better served by a custom C
extension?

martin

9 Answers

Robert Klemme

1/16/2007 2:13:00 PM

0

On 16.01.2007 15:10, Martin DeMello wrote:
> I have a file that is a dump of a large array of 32-bit unsigned ints
> (bitvectors actually). All I need is to load this into memory and have
> fast, efficient access to the array members, and a few bitmasking
> operations - the array will never be written to, and ideally should
> not need to be bounds-checked. Should I be looking at something like
> mmap or narray to do this, or would I be better served by a custom C
> extension?

How large is "large"?

robert

Vincent Fourmond

1/16/2007 2:16:00 PM

0

Martin DeMello wrote:
> I have a file that is a dump of a large array of 32-bit unsigned ints
> (bitvectors actually). All I need is to load this into memory and have
> fast, efficient access to the array members, and a few bitmasking
> operations - the array will never be written to, and ideally should
> not need to be bounds-checked. Should I be looking at something like
> mmap or narray to do this, or would I be better served by a custom C
> extension?

I would write my own extension and use mmap (if you're under linux) to
save trouble and memory. That's just a question of a few hours, and
you'll get high performance.

Cheers,

Vince, in a posting mood today...


--
Vincent Fourmond, PhD student
http://vincent.fourmon...

Martin DeMello

1/16/2007 2:29:00 PM

0

On 1/16/07, Robert Klemme <shortcutter@googlemail.com> wrote:
> On 16.01.2007 15:10, Martin DeMello wrote:
> > I have a file that is a dump of a large array of 32-bit unsigned ints
> > (bitvectors actually). All I need is to load this into memory and have
> > fast, efficient access to the array members, and a few bitmasking
> > operations - the array will never be written to, and ideally should
> > not need to be bounds-checked. Should I be looking at something like
> > mmap or narray to do this, or would I be better served by a custom C
> > extension?
>
> How large is "large"?

Order of 1 MB - not gigantic, but not trivial either.

martin

Robert Klemme

1/16/2007 3:10:00 PM

0

On 16.01.2007 15:29, Martin DeMello wrote:
> On 1/16/07, Robert Klemme <shortcutter@googlemail.com> wrote:
>> On 16.01.2007 15:10, Martin DeMello wrote:
>> > I have a file that is a dump of a large array of 32-bit unsigned ints
>> > (bitvectors actually). All I need is to load this into memory and have
>> > fast, efficient access to the array members, and a few bitmasking
>> > operations - the array will never be written to, and ideally should
>> > not need to be bounds-checked. Should I be looking at something like
>> > mmap or narray to do this, or would I be better served by a custom C
>> > extension?
>>
>> How large is "large"?
>
> Order of 1 MB - not gigantic, but not trivial either.

Um, I would not say 1MB is large. What accesses do you do? Do you just
access based on index? Do you often need to load this data? etc.

In your case I'd start out writing a Ruby version (either using a single
String or an Array with integers) and change that into an extension if
performance is insufficient. My 0.02 EUR...

Kind regards

robert

Martin DeMello

1/16/2007 3:21:00 PM

0

On 1/16/07, Robert Klemme <shortcutter@googlemail.com> wrote:
>
> Um, I would not say 1MB is large. What accesses do you do? Do you just
> access based on index? Do you often need to load this data? etc.

Pure array indexing, just a lot of it.

> In your case I'd start out writing a Ruby version (either using a single
> String or an Array with integers) and change that into an extension if
> performance is insufficient. My 0.02 EUR...

Good point - it'd be the quickest thing to get up and running, if
nothing else :)

martin

Christian Neukirchen

1/16/2007 7:24:00 PM

0

"Martin DeMello" <martindemello@gmail.com> writes:

> I have a file that is a dump of a large array of 32-bit unsigned ints
> (bitvectors actually). All I need is to load this into memory and have
> fast, efficient access to the array members, and a few bitmasking
> operations - the array will never be written to, and ideally should
> not need to be bounds-checked. Should I be looking at something like
> mmap or narray to do this, or would I be better served by a custom C
> extension?

I think Ara has been hacking some package to use Narray on mmap'ed
files, which works fast and is persistent. Search the archives.

> martin
--
Christian Neukirchen <chneukirchen@gmail.com> http://chneuk...

Tim Pease

1/16/2007 7:52:00 PM

0

On 1/16/07, Christian Neukirchen <chneukirchen@gmail.com> wrote:
> "Martin DeMello" <martindemello@gmail.com> writes:
>
> > I have a file that is a dump of a large array of 32-bit unsigned ints
> > (bitvectors actually). All I need is to load this into memory and have
> > fast, efficient access to the array members, and a few bitmasking
> > operations - the array will never be written to, and ideally should
> > not need to be bounds-checked. Should I be looking at something like
> > mmap or narray to do this, or would I be better served by a custom C
> > extension?
>
> I think Ara has been hacking some package to use Narray on mmap'ed
> files, which works fast and is persistent. Search the archives.
>

http://codeforpeople.com/lib/...

You'll need to install Guy's mmap package ...

http://moulon.inra.fr/ruby...

Blessings,
TwP

V

9/17/2010 4:27:00 AM

0

On Thu, 16 Sep 2010 21:02:19 -0700 (PDT), Werner <whetzner@mac.com>
wrote:

>How will it proceed and how will it end?

In my opinion, I think certain factions of the CIA are planning a
coup, which will not be successful in the long run -- mainly because
the USA is so technologically backwards and so far behind the rest of
the world. So will the warlords of the CIA play Mad Max -- or will
common sense cause the breakup of the USA into regional factions
who are able to feed themselves.

There is an old saying -- an army can't run on an empty stomach.

I do believe that food production and guns will be the only concern
in America hence forth. The smart, peaceful ones who know how to
defend their crops will be the only ones to survive.






V

9/21/2010 6:08:00 AM

0

On Mon, 20 Sep 2010 22:18:12 -0700 (PDT), William Mook
<mokmedical@gmail.com> wrote:

>On Sep 20, 4:04?am, Terrys <merryter...@gmail.com> wrote:
>> On Sep 20, 2:39?pm, William Mook <mokmedi...@gmail.com> wrote:
>>
>> > These same people also believe that the USA made much more progress in
>> > the field of nuclear powered aircraft than has been made public -

The United States government doesn't even know what gravity is -- so
how could they make any progress in aircraft design? The USA has for
decades pumped the public full of false propaganda about their
technological prowess, and now in the 21st century it is OBVIOUS they
don't have advanced technology -- of ANY kind.

>> > Some believe these may have gone underground and not abandoned as is
>> > publicly thought.

Great cover for the real truth in that he USA never had advanced
technology -- so there's nothing to hide -- because it doesn't exist.
The US propaganda machine churns out the lies trying to make itself
appear more powerful than it really is and make smaller nations fear
the big and powerful America -- which is a sham. America is
technologically backwards. Besides how could a nation which has been
deliberately dumbed down invent anything -> clue - they can't.
Further, how could a nation invent anything when the banker idiots are
killing of anyone who is intelligent enough to invent anything. The
truth is American's controllers have never wanted advanced technology
because they feared losing control, hence the bullshit about America's
technological prowess -- which doesn't exist.

>> That explains the Christchurch earthquake then. Thank you.
>>
>> Terry "Heavy metal fan" Smith.
>
>Hahaha -

And once again, another indicator of America's further decline:
don't tell the American public the truth because they'll stop spending
money. So the bankers withheld a lot of critical information needed
to make rational decisions -- like Earth has shifted on its rotational
axis. The ONLY thing America is used for is a front operation to give
the London bank's scrip value. That is all. So the bankers couldn't
care less about the survival of humanity or Earth and instead
have their CIA lackeys spew reams of bullshit, that even the US
government employees believe. But the power of propaganda has a
price!

I'm surprised one you imbeciles hasn't taken a page from the
Gore Climategate bible and conjured up the idea that evil spirits are
punishing NZ with an earthquake for past life transgressions -- or a
paragraph or two from the Holy Bill "Microsoft" Gates murderous plan
to kill a billion people with vaccinations in order to "save" Earth.

There simply is no one on Earth more stupid than Bill Gates -- a
complete and utter moron who should be in jail for suggesting the
deliberate murder of a billion people.

Damn it. You people are mind numbingly stupid.

Earth has shifted on it's rotational axis and this is causing the
earthquakes and climate change -- but the banks don't want anyone to
know because they fear people won't use their money! So instead the
price to pay is stupidity, backwardness, paganism and the inability
of a society to be able to address the real issues which are
threatening it's survival:

www.divulgence.net

Ever hear of Occam's Razor?!?! I sincerely doubt wither Gore or Gates
attended class past 4th grade,so they're clueless aside from being low
grade morons.

DUH DUH DUH

How many other moronic things will the divas of stupidity like Gates
and Gore come up with, eh?

Are people going to go around in pagan robes sprinkling rock dust into
the heavens & worshipping the pagan gods of the
Gore/Gates/Oprah/Turner
school of imbecility?

And you people think money equates to intelligence. If that were so,
Gore, Gate, Oprah and Turner would all have purchased new brains.

The truth is America has been kept in the dark so long and fed so much
bullshit that it is completely dysfunctional and of no use to anyone
-- they're broke too. So much for 'maintaining' the 'status quo'.

<rolls eyes>