Asp Forum - problem with OpenMP in Ruby extension

Rudi Cilibrasi

10/3/2007 9:53:00 PM

Hi everybody,

I am the main developer of CompLearn, an open-source machine learning library
for scientific researchers. I have written a Ruby binding for CompLearn
called complearn-ruby. It has worked fine in the past. But it has a problem
ever since I converted the base library (libcomplearn) to use OpenMP for
multicore processor support. If I just try to use the simplest approach that
has worked in the past in my extconf.rb, I get the following problem when I
try to require the C-extension Ruby binding complearn4r.so and use the
multicore part of the system:

ruby: symbol lookup error: /usr/lib/libcomplearn.so.1: undefined symbol: GOMP_pa
rallel_start

To try to fix this error,

I adjusted the extconf.rb to include this line:

have_library('gomp', 'main')

And rebuilding works but running a simple test program (just trying to require
complearn) fails in a different way than before:

/home/cilibrar/rlc/projects/complearn/complearn-ruby/ext/complearn/complearn4r.s
o: libgomp.so.1: shared object cannot be dlopen()ed - /home/cilibrar/rlc/project
s/complearn/complearn-ruby/ext/complearn/complearn4r.so (LoadError)
from /home/cilibrar/rlc/projects/complearn/complearn-ruby/lib/complearn.
rb:2
from tests/t2.rb:1:in `require'
from tests/t2.rb:1

I don't know what else to try and wonder if anybody else can offer a clue?

I have tried for a long time experimenting with CFLAGS, CPPFLAGS, LDFLAGS,
and -fopenmp among other settings but I still cannot seem to figure out
a solution. I have put the source code for this problem up at

http://cilibrar.com/~cilibrar/projsup/rubbug/libcomplearn-1....
and
http://cilibrar.com/~cilibrar/projsup/rubbug/libcomplearn-ruby-1....

I would appreciate greatly any help anybody can offer in this problem. Thank
you for your attention,

Rudi

--
"We can try to do it by breaking free of the mental prison of
separation and exclusion and see the world in its interconnectedness
and non-separability, allowing new alternatives to emerge." -- after
Vandana Shiva

4 Answers

ara.t.howard

10/4/2007 4:48:00 AM

On Oct 3, 2007, at 3:53 PM, Rudi Cilibrasi wrote:

>
> I adjusted the extconf.rb to include this line:
>
> have_library('gomp', 'main')
>
> And rebuilding works but running a simple test program (just trying
> to require
> complearn) fails in a different way than before:
>

this sounds like it gets the link right (LD_LIBRARY_PATH) but fails
at runtime. try setting LD_RUN_PATH to the location of the gomp lib
*before* compiling

for example

export LD_RUN_PATH=/full/path/to/directory/with/gomp/lib
ruby extconf.rb && make && sudo make install

just a hunch...

cheers.

ps. long time time no hear!

a @ http://draw...
--
we can deny everything, except that we have the possibility of being
better. simply reflect on that.
h.h. the 14th dalai lama

Rudi Cilibrasi

10/4/2007 11:22:00 AM

Hi Ara,

Good to hear from you and I am glad to hear you are still thinking about dynamic
linking issues. Unfortunately I tried your suggestion and so far I
cannot detect any measurable difference in behavior. Same error
messages in both cases, with and without the #have_library(gomp...)
line.

Yes, it has taken years but I have rewritten complearn to use gobject
and OpenMP and I must say the results are exciting. Recently,
somebody sent in a patch to improve the complearn tree search from
O(n^5) to O(n^3) and this really made my week last week, speeding up
tree search by thousands of times. I am very excited to start
applying the new technology but this whole ruby extension thing openmp
problem has rained on my parade! nice to hear from you again, -r.

On 10/3/07, ara.t.howard <ara.t.howard@gmail.com> wrote:
>
> On Oct 3, 2007, at 3:53 PM, Rudi Cilibrasi wrote:
> > I adjusted the extconf.rb to include this line:
> >
> > have_library('gomp', 'main')
> >
> > And rebuilding works but running a simple test program (just trying
> > to require
> > complearn) fails in a different way than before:
> >
>
> this sounds like it gets the link right (LD_LIBRARY_PATH) but fails
> at runtime. try setting LD_RUN_PATH to the location of the gomp lib
> *before* compiling
>
> for example
>
> export LD_RUN_PATH=/full/path/to/directory/with/gomp/lib
> ruby extconf.rb && make && sudo make install
>
> just a hunch...
>
> cheers.
>
> ps. long time time no hear!
>
> a @ http://draw...
> --
> we can deny everything, except that we have the possibility of being
> better. simply reflect on that.
> h.h. the 14th dalai lama
>
>
>
>
>

--
"We can try to do it by breaking free of the mental prison of
separation and exclusion and see the world in its interconnectedness
and non-separability, allowing new alternatives to emerge." -- after
Vandana Shiva

ara.t.howard

10/6/2007 3:34:00 PM

On Oct 4, 5:22 am, "Rudi Cilibrasi" <cilib...@gmail.com> wrote:
> Hi Ara,
>
> Good to hear from you and I am glad to hear you are still thinking about dynamic
> linking issues. Unfortunately I tried your suggestion and so far I
> cannot detect any measurable difference in behavior. Same error
> messages in both cases, with and without the #have_library(gomp...)
> line.
>
> Yes, it has taken years but I have rewritten complearn to use gobject
> and OpenMP and I must say the results are exciting. Recently,
> somebody sent in a patch to improve the complearn tree search from
> O(n^5) to O(n^3) and this really made my week last week, speeding up
> tree search by thousands of times. I am very excited to start
> applying the new technology but this whole ruby extension thing openmp
> problem has rained on my parade! nice to hear from you again, -r.

try setting LD_LIBRARY_PATH to the location of libgimp (directory
containing lib) and see if that helps

for example

export LD_LIBRARY_PATH=/usr/local/lib/
ruby a.rb

and see if you still get the error. if dlopen fails then there is
some other issue. make sure the lib is actually there - if it this
works then massaging LD_RUN_PATH *should* work.

regards.

Rudi Cilibrasi

10/6/2007 7:09:00 PM

Hmm, I tried the LD_LIBRARY_PATH thing and still no luck. But, gga gave the
suggestion that perhaps it is a bad idea (or even now crashing) for the other
reason that Ruby has its own threads that are probably grossly incompatible
with OpenMP threading. So, I instead rewrote the entire Ruby binding and also
made a new standalone application called uncd: the Universal NCD connectort.
It uses stdin and stdout with a handmade protocol to communicate with
a server process to allow easy language binding in Ruby and Erlang and
eventually Perl, Java, Python etc. Here is the source code if you are
interested I think it is a major improvement over the old code and
will be doing this for future bindings too I think:

http://hg.cilibrar.com/compl...

I learned this approach from Erlang books. I think I like the idea of
coarse-grained libraries providing functionality via separate
process-space as we move into ubiquitous multicore land.

Thanks for taking the time to try some answers I always have more to
learn about the LD_* variables and dynamic linking.

best regards,

-r.

On 10/6/07, -a <ara.t.howard@gmail.com> wrote:
>
>
> On Oct 4, 5:22 am, "Rudi Cilibrasi" <cilib...@gmail.com> wrote:
> > Hi Ara,
> >
> > Good to hear from you and I am glad to hear you are still thinking about dynamic
> > linking issues. Unfortunately I tried your suggestion and so far I
> > cannot detect any measurable difference in behavior. Same error
> > messages in both cases, with and without the #have_library(gomp...)
> > line.
> >
> > Yes, it has taken years but I have rewritten complearn to use gobject
> > and OpenMP and I must say the results are exciting. Recently,
> > somebody sent in a patch to improve the complearn tree search from
> > O(n^5) to O(n^3) and this really made my week last week, speeding up
> > tree search by thousands of times. I am very excited to start
> > applying the new technology but this whole ruby extension thing openmp
> > problem has rained on my parade! nice to hear from you again, -r.
>
>
> try setting LD_LIBRARY_PATH to the location of libgimp (directory
> containing lib) and see if that helps
>
> for example
>
> export LD_LIBRARY_PATH=/usr/local/lib/
> ruby a.rb
>
> and see if you still get the error. if dlopen fails then there is
> some other issue. make sure the lib is actually there - if it this
> works then massaging LD_RUN_PATH *should* work.
>
> regards.
>
>
>

--
"We can try to do it by breaking free of the mental prison of
separation and exclusion and see the world in its interconnectedness
and non-separability, allowing new alternatives to emerge." -- after
Vandana Shiva

comp.lang.ruby

problem with OpenMP in Ruby extension

Rudi Cilibrasi

ara.t.howard

Rudi Cilibrasi

ara.t.howard

Rudi Cilibrasi

x Login to ForumsZone