Asp Forum - [ANN] rocaml 0.6.0: fast, easy Ruby extensions in Objective Caml

Mauricio Fernández

10/17/2007 3:32:00 PM

rocaml allows you to write Ruby extensions in Objective Caml.

http://eige.../h...

Developing Ruby extensions with rocaml is easier and more convenient than
writing a plain old C extension because rocaml performs Ruby <-> OCaml
conversions for a wide range of types, including abstract types and arrays,
tuples, variants and records of values of any supported type (e.g. arrays of
arrays of variants of tuples of ...).

Moreover, exceptions raised in the OCaml code are captured by the generated
extension and raised inside Ruby.

Making an extension with rocaml involves two steps:
* implementing the desired functionality in Objective Caml, and registering
the functions to be exported (using Callback.register : string -> 'a -> unit
or the included camlp4 extension)
* creating the extconf.rb file (just modify the sample extconf.rb distributed
with rocaml) defining the interface of your Objective Caml code.

** At no point is there any need to write a single line of C code. **

The mandatory trivial example
=============================

This example doesn't do justice to the usefulness of rocaml because the
extension is beyond trivial and you could as well have written it in C using
RubyInline. The advantages of rocaml (and of Objective Caml) usually become
visible when the extension takes more than two lines (take a look at the
3-line Marshal replacement that is 3 times faster than Ruby's, though...).
Here follows a minimal example however, merely to show how easily rocaml
extensions can be made.

Here's the OCaml code placed in fib.ml:
let rec fib n = if n < 2 then 1 else fib (n-1) + fib (n-2)
export fib

Here's the interface declaration in your extconf.rb:
Interface.generate("fib") do
def_module("Fib") do
fun "fib", INT => INT
end
end

That's it. The extension can be built like any run-of-the-mill C extension
with
ruby extconf.rb
make

The resulting Ruby extension that can be used as usual:
require 'fib'
p Fib.fib 10

Download
========
You can get rocaml at http://eige.../h...

License
=======
rocaml is distributed under the same terms as Ruby.

rocaml copyright (c) 2007 Mauricio Fernandez <mfp@acm.org>

--
Mauricio Fernandez - http://eige...

15 Answers

Daniel Berger

10/17/2007 7:11:00 PM

On Oct 17, 9:31 am, Mauricio Fernandez <m...@acm.org> wrote:
> rocaml allows you to write Ruby extensions in Objective Caml.

<snip>

I grabbed the latest and greatest Ocaml tarball and installed it.
However, when I try to install rocaml I get this:

sh: camlp5: not found
*** extconf.rb failed ***

I have no camlp5 in /usr/local/bin. I have camlp4, however, and the
comments in rocaml_extconf.rb suggest that either should suffice.

What should I do?

Thanks,

Dan

Daniel Berger

10/17/2007 7:48:00 PM

On Oct 17, 1:10 pm, Daniel Berger <djber...@gmail.com> wrote:
> On Oct 17, 9:31 am, Mauricio Fernandez <m...@acm.org> wrote:
>
> > rocaml allows you to write Ruby extensions in Objective Caml.
>
> <snip>
>
> I grabbed the latest and greatest Ocaml tarball and installed it.
> However, when I try to install rocaml I get this:
>
> sh: camlp5: not found
> *** extconf.rb failed ***
>
> I have no camlp5 in /usr/local/bin. I have camlp4, however, and the
> comments in rocaml_extconf.rb suggest that either should suffice.
>
> What should I do?

I decided to grab camlp5 (http://pauillac.inria.fr/~d...) and
try again. The extconf.rb script succeeds now. However, now the make
fails:

ocamldep.opt rubyOCamlUtil.ml > .depend
sh: ocamldep.opt: not found
*** Error code 1
make: Fatal error: Command failed for target `.depend'

I have ocamldep, but no ocamldep.opt. Where do I go from here?

Thanks,

Dan

Mauricio Fernández

10/17/2007 8:04:00 PM

On Thu, Oct 18, 2007 at 04:10:55AM +0900, Daniel Berger wrote:
>
>
> On Oct 17, 9:31 am, Mauricio Fernandez <m...@acm.org> wrote:
> > rocaml allows you to write Ruby extensions in Objective Caml.
>
> <snip>
>
> I grabbed the latest and greatest Ocaml tarball and installed it.
> However, when I try to install rocaml I get this:
>
> sh: camlp5: not found
> *** extconf.rb failed ***
>
> I have no camlp5 in /usr/local/bin. I have camlp4, however, and the
> comments in rocaml_extconf.rb suggest that either should suffice.
>
> What should I do?

I see you solved this by installing camlp5, but I'd still like to fix the
camlp* selection code. What does camlp4 -v return for you?

$ camlp4 -v
Camlp4 version 3.10.0

camlp4 in 3.10.0 is incompatible with 3.09.2's (OTOH camlp5 *is* backwards
compatible).

The code that detects camlp5 is

have_camlp5 = ! `camlp5 -v 2>&1`["version"].empty?

For some reason it didn't work properly but I can't see why. Any idea?

--
Mauricio Fernandez - http://eige... - singular Ruby

Daniel Berger

10/17/2007 8:17:00 PM

On Oct 17, 2:04 pm, Mauricio Fernandez <m...@acm.org> wrote:
> On Thu, Oct 18, 2007 at 04:10:55AM +0900, Daniel Berger wrote:
>
> > On Oct 17, 9:31 am, Mauricio Fernandez <m...@acm.org> wrote:
> > > rocaml allows you to write Ruby extensions in Objective Caml.
>
> > <snip>
>
> > I grabbed the latest and greatest Ocaml tarball and installed it.
> > However, when I try to install rocaml I get this:
>
> > sh: camlp5: not found
> > *** extconf.rb failed ***
>
> > I have no camlp5 in /usr/local/bin. I have camlp4, however, and the
> > comments in rocaml_extconf.rb suggest that either should suffice.
>
> > What should I do?
>
> I see you solved this by installing camlp5, but I'd still like to fix the
> camlp* selection code. What does camlp4 -v return for you?
>
> $ camlp4 -v
> Camlp4 version 3.10.0

Same for me.

> camlp4 in 3.10.0 is incompatible with 3.09.2's (OTOH camlp5 *is* backwards
> compatible).
>
> The code that detects camlp5 is
>
> have_camlp5 = ! `camlp5 -v 2>&1`["version"].empty?
>
> For some reason it didn't work properly but I can't see why. Any idea?

If camlp5 isn't found, you'll end up with nil. Changing 'empty?' to
'nil?' should do the trick, although people will still see a "sh:
camlp5: not found" echoed to stdout, which may be confusing.

Any ideas on the make failure?

Regards,

Dan

Mauricio Fernández

10/17/2007 8:21:00 PM

On Thu, Oct 18, 2007 at 04:48:13AM +0900, Daniel Berger wrote:
> On Oct 17, 1:10 pm, Daniel Berger <djber...@gmail.com> wrote:
> > On Oct 17, 9:31 am, Mauricio Fernandez <m...@acm.org> wrote:
> >
> > > rocaml allows you to write Ruby extensions in Objective Caml.
[...]
> > I grabbed the latest and greatest Ocaml tarball and installed it.
> > However, when I try to install rocaml I get this:
> >
> > sh: camlp5: not found
> > *** extconf.rb failed ***
> >
> > I have no camlp5 in /usr/local/bin. I have camlp4, however, and the
> > comments in rocaml_extconf.rb suggest that either should suffice.
[...]
> I decided to grab camlp5 (http://pauillac.inria.fr/~d...) and
> try again. The extconf.rb script succeeds now. However, now the make
> fails:
>
> ocamldep.opt rubyOCamlUtil.ml > .depend
> sh: ocamldep.opt: not found
> *** Error code 1
> make: Fatal error: Command failed for target `.depend'
>
> I have ocamldep, but no ocamldep.opt. Where do I go from here?

You didn't
make opt.opt
(or make world.opt) when you built ocaml-3.10.0, did you?
(BTW, on Debian the .opt binaries are in ocaml-native-compilers, in case
somebody is reading this)

For the time being, you can just install the native-code compilers or
hand-edit the generated Makefile; scroll down to

...
#############################################################################
# #
# Objective Caml #
# #
#############################################################################

OCAMLC = ocamlc.opt
OCAMLOPT = ocamlopt.opt
OCAMLDEP = ocamldep.opt
OFLAGS = -pp 'camlp5o -I . pa_rocaml.cmo'
OCAML_INCLUDES =
OCAML_LIBS = nums.cmxa
...

and remove the .opt extensions. I will add some code to detect whether the
opt binaries are available.

One last thing; what's your platform? rocaml extensions might or might not
work on AMD64 and OSX because their linkers aren't very smart (they can't
handle non-PIC code, it seems). According to OCaml's 3.10.0 changelog,
- Intel/AMD 64 bits: generate position-independent code by default.
but I don't know if that's enough or something special must be done at compile
time.

The next Objective Caml release will feature dynamic loading of native code,
so this won't remain a problem for long.
Meanwhile, static extensions can be used.

--
Mauricio Fernandez - http://eige...

Mauricio Fernández

10/17/2007 9:35:00 PM

On Thu, Oct 18, 2007 at 05:17:02AM +0900, Daniel Berger wrote:
> On Oct 17, 2:04 pm, Mauricio Fernandez <m...@acm.org> wrote:
> > On Thu, Oct 18, 2007 at 04:10:55AM +0900, Daniel Berger wrote:
[...]
> > > I grabbed the latest and greatest Ocaml tarball and installed it.
> > > However, when I try to install rocaml I get this:
> >
> > > sh: camlp5: not found
> > > *** extconf.rb failed ***
> >
> > > I have no camlp5 in /usr/local/bin. I have camlp4, however, and the
> > > comments in rocaml_extconf.rb suggest that either should suffice.
> >
> > > What should I do?
> >
> > I see you solved this by installing camlp5, but I'd still like to fix the
> > camlp* selection code. What does camlp4 -v return for you?
[...]
> > The code that detects camlp5 is
> >
> > have_camlp5 = ! `camlp5 -v 2>&1`["version"].empty?
> >
> > For some reason it didn't work properly but I can't see why. Any idea?
>
> If camlp5 isn't found, you'll end up with nil. Changing 'empty?' to
> 'nil?' should do the trick, although people will still see a "sh:
> camlp5: not found" echoed to stdout, which may be confusing.

That's what 2>&1 is meant to prevent, but you're very right, it should be
nil?. Now I wonder why you didn't get a NoMethodError.

> Any ideas on the make failure?

Yes, the native code compilers are missing, see my other message for a couple
solutions. There was another bug in the code that detects whether they are
present; I have pushed the fix to the repository at

http://eige.../repos/ro...

and it will be in the next tarball, 0.6.1, to follow shortly.

Wed Oct 17 23:29:10 CEST 2007 Mauricio Fernandez <mfp@acm.org>
* rocaml_extconf.rb: fixed native compiler detection.
Wed Oct 17 23:21:08 CEST 2007 Mauricio Fernandez <mfp@acm.org>
* rocaml_extconf.rb: fix camlp[45] detection.
diff -rN -u old-rocaml/rocaml_extconf.rb new-rocaml/rocaml_extconf.rb
--- old-rocaml/rocaml_extconf.rb 2007-10-17 23:29:42.000000000 +0200
+++ new-rocaml/rocaml_extconf.rb 2007-10-17 23:29:42.000000000 +0200
@@ -43,7 +43,7 @@
exit
end

-maybe_opt = lambda{|x| opt = "#{x}.opt"; system(x) ? opt : x }
+maybe_opt = lambda{|x| opt = "#{x}.opt"; system(opt) ? opt : x }

if OCAML_PACKAGES.empty? then
OCAMLC = maybe_opt["ocamlc"]
@@ -91,9 +91,9 @@

# determine whether camlp4 (or camlp5) can be used:

-have_camlp5 = ! `camlp5 -v 2>&1`["version"].empty?
+have_camlp5 = ! `camlp5 -v 2>&1`["version"].nil?
camlp4version = `camlp4 -v 2>&1`[/version\s+(\d.*)/, 1]
-have_camlp4 = ! camlp4version.empty?
+have_camlp4 = ! camlp4version.nil?

pa_rocaml_revdeps = Dir["*.ml"].map do |f|
"#{f.sub(/\.ml$/, ".cmx")}: pa_rocaml.cmo"

--
Mauricio Fernandez - http://eige... - singular Ruby

Daniel Berger

10/18/2007 3:26:00 PM

On Oct 17, 2:21 pm, Mauricio Fernandez <m...@acm.org> wrote:
> On Thu, Oct 18, 2007 at 04:48:13AM +0900, Daniel Berger wrote:
> > On Oct 17, 1:10 pm, Daniel Berger <djber...@gmail.com> wrote:
> > > On Oct 17, 9:31 am, Mauricio Fernandez <m...@acm.org> wrote:
>
> > > > rocaml allows you to write Ruby extensions in Objective Caml.
> [...]
> > > I grabbed the latest and greatest Ocaml tarball and installed it.
> > > However, when I try to install rocaml I get this:
>
> > > sh: camlp5: not found
> > > *** extconf.rb failed ***
>
> > > I have no camlp5 in /usr/local/bin. I have camlp4, however, and the
> > > comments in rocaml_extconf.rb suggest that either should suffice.
> [...]
> > I decided to grab camlp5 (http://pauillac.inria.fr/~d...) and
> > try again. The extconf.rb script succeeds now. However, now the make
> > fails:
>
> > ocamldep.opt rubyOCamlUtil.ml > .depend
> > sh: ocamldep.opt: not found
> > *** Error code 1
> > make: Fatal error: Command failed for target `.depend'
>
> > I have ocamldep, but no ocamldep.opt. Where do I go from here?
>
> You didn't
> make opt.opt
> (or make world.opt) when you built ocaml-3.10.0, did you?

Ok, I did make world.opt and reinstalled. Then I rebuilt camlp5 with
"transitional". Now it seems I'm missing some header files:

>make
ocamldep.opt rubyOCamlUtil.ml > .depend
cc -I. -I/usr/local/lib/ruby/1.8/sparc-solaris2.10 -I/usr/local/lib/
ruby/1.8/sparc-solaris2.10 -I. -KPIC -g -c foo_rocaml_wrapper.c
"foo_rocaml_wrapper.c", line 14: cannot find include file: <caml/
mlvalues.h>
"foo_rocaml_wrapper.c", line 15: cannot find include file: <caml/
callback.h>
"foo_rocaml_wrapper.c", line 16: cannot find include file: <caml/
memory.h>
"foo_rocaml_wrapper.c", line 17: cannot find include file: <caml/
alloc.h>
"foo_rocaml_wrapper.c", line 18: cannot find include file: <caml/
fail.h>
...

I don't think OCaml's installation instructions could be any more
confusing. What option did I miss?

Regards,

Dan

Mauricio Fernández

10/18/2007 6:10:00 PM

On Fri, Oct 19, 2007 at 12:26:04AM +0900, Daniel Berger wrote:
> > You didn't
> > make opt.opt
> > (or make world.opt) when you built ocaml-3.10.0, did you?
>
> Ok, I did make world.opt and reinstalled. Then I rebuilt camlp5 with
> "transitional". Now it seems I'm missing some header files:
>
> >make
> ocamldep.opt rubyOCamlUtil.ml > .depend
> cc -I. -I/usr/local/lib/ruby/1.8/sparc-solaris2.10 -I/usr/local/lib/
> ruby/1.8/sparc-solaris2.10 -I. -KPIC -g -c foo_rocaml_wrapper.c
> "foo_rocaml_wrapper.c", line 14: cannot find include file: <caml/
> mlvalues.h>
[...]
>
> I don't think OCaml's installation instructions could be any more
> confusing. What option did I miss?

I'm sorry for all the issues and appreciate your patience. The basic problem
is that I'm using Debian and everything Just Works(TM) with its OCaml
packages, so it is harder for me to anticipate problems like those you ran
into. I see that you're using Solaris, which adds further uncertainty because
I'm not sure its linker can build shared objects with non-PIC code [1]. Some
magic incantations in the form of $LDFLAGS might be required. You're stepping
on new ground :)

The missing headers should be in /usr/local/lib/ocaml/3.10.0/caml (or maybe
/usr/local/lib/ocaml/caml; in my system I have a symlink from
/usr/include/caml to /usr/lib/ocaml/3.10.0/caml, which is why this never
happened to me). The extension should build correctly after symlinking or
applying this one-line patch:

Thu Oct 18 19:14:58 CEST 2007 Mauricio Fernandez <mfp@acm.org>
* rocaml_extconf.rb: add ocaml_native_lib_path to the INCFLAGS (-Idir).
diff -rN -u -w old-rocaml/rocaml_extconf.rb new-rocaml/rocaml_extconf.rb
--- old-rocaml/rocaml_extconf.rb 2007-10-18 19:25:49.000000000 +0200
+++ new-rocaml/rocaml_extconf.rb 2007-10-18 19:25:49.000000000 +0200
@@ -43,6 +43,8 @@
exit
end

+$INCFLAGS << " -I#{ocaml_native_lib_path}"
+
maybe_opt = lambda{|x| opt = "#{x}.opt"; system(opt) ? opt : x }

if OCAML_PACKAGES.empty? then

Note that the extension you're building ("foo") isn't complete. It includes
the wrappers but not the OCaml implementations of the corresponding functions;
in fact, it is only meant to serve as the extconf.rb template for new
extensions.

You can find the actual examples under examples/:
* marshal: 3-line specialized marshallers that can be over 3 times faster than
Ruby's Marshal (the largest speedup is achieved with float arrays)
* tree: a 30-line RB tree with 3X faster lookup than RBTree
(you'll need rbtree if you want to run the benchmarks in test_tree.rb)
* oo, records, variants: show how abstract types, records and variants are
converted between Ruby and OCaml. Abstract types become objects, records
turn into hashes with symbol keys and variants are mapped to symbols or
arrays.

Thank you,

[1] According to http://gcc.gnu.org/ml/gcc/1999-05n/msg...

SunOS and Solaris will quite happily build .so from non-PIC objects - they
just don't "share" very well as the relocations cause copy-on-write and hence
private pages.

So there's a good chance it will work.

--
Mauricio Fernandez - http://eige...

Daniel Berger

10/18/2007 6:30:00 PM

On Oct 18, 12:09 pm, Mauricio Fernandez <m...@acm.org> wrote:
> On Fri, Oct 19, 2007 at 12:26:04AM +0900, Daniel Berger wrote:
> > > You didn't
> > > make opt.opt
> > > (or make world.opt) when you built ocaml-3.10.0, did you?
>
> > Ok, I did make world.opt and reinstalled. Then I rebuilt camlp5 with
> > "transitional". Now it seems I'm missing some header files:
>
> > >make
> > ocamldep.opt rubyOCamlUtil.ml > .depend
> > cc -I. -I/usr/local/lib/ruby/1.8/sparc-solaris2.10 -I/usr/local/lib/
> > ruby/1.8/sparc-solaris2.10 -I. -KPIC -g -c foo_rocaml_wrapper.c
> > "foo_rocaml_wrapper.c", line 14: cannot find include file: <caml/
> > mlvalues.h>
> [...]
>
> > I don't think OCaml's installation instructions could be any more
> > confusing. What option did I miss?
>
> I'm sorry for all the issues and appreciate your patience. The basic problem
> is that I'm using Debian and everything Just Works(TM) with its OCaml
> packages, so it is harder for me to anticipate problems like those you ran
> into. I see that you're using Solaris, which adds further uncertainty because
> I'm not sure its linker can build shared objects with non-PIC code [1]. Some
> magic incantations in the form of $LDFLAGS might be required. You're stepping
> on new ground :)

It would seem so. Your one line patch to rocaml_extconf.rb worked
(thanks!), but now I get this:

irb(main):002:0> require 'foo'
LoadError: ld.so.1: ruby: fatal: relocation error: file /export/home/
djberge/src/ruby/rocaml-0.6.1/foo.so: symbol __muldi3: referenced
symbol not found - /export/home/djberge/src/ruby/rocaml-0.6.1/foo.so
from /export/home/djberge/src/ruby/rocaml-0.6.1/foo.so
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/
custom_require.rb:27:in `require'
from (irb):2

This is Sun Studio 12 on Sparc/Solaris 10, btw.

I already tried installing a gcc add-on found here: http://cooltools.sunsourc...

No joy. Any ideas?

Oh, and these warnings showed up during the build. Dunno if you're
interested:

"foo_rocaml_wrapper.c", line 61: warning: statement not reached
"foo_rocaml_wrapper.c", line 68: warning: statement not reached
"foo_rocaml_wrapper.c", line 338: warning: statement not reached
"foo_rocaml_wrapper.c", line 388: warning: statement not reached
"foo_rocaml_wrapper.c", line 438: warning: statement not reached
"foo_rocaml_wrapper.c", line 485: warning: statement not reached
"foo_rocaml_wrapper.c", line 532: warning: statement not reached
"foo_rocaml_wrapper.c", line 582: warning: statement not reached
"foo_rocaml_wrapper.c", line 632: warning: statement not reached
"foo_rocaml_wrapper.c", line 681: warning: statement not reached
"foo_rocaml_wrapper.c", line 730: warning: statement not reached
"foo_rocaml_wrapper.c", line 779: warning: statement not reached
"foo_rocaml_wrapper.c", line 829: warning: statement not reached

Regards,

Dan

Daniel Berger

10/18/2007 6:36:00 PM

On Oct 18, 12:30 pm, Daniel Berger <djber...@gmail.com> wrote:

<snip>

Some more information that may or may not be useful:

jberge-/export/home/djberge/src/ruby/rocaml-0.6.1-697>ldd foo.so
librt.so.1 => /lib/librt.so.1
libpthread.so.1 => /lib/libpthread.so.1
libdl.so.1 => /lib/libdl.so.1
libcrypt_i.so.1 => /usr/lib/libcrypt_i.so.1
libm.so.2 => /lib/libm.so.2
libc.so.1 => /lib/libc.so.1
libaio.so.1 => /lib/libaio.so.1
libmd5.so.1 => /lib/libmd5.so.1
libgen.so.1 => /lib/libgen.so.1
/platform/SUNW,Sun-Blade-100/lib/libc_psr.so.1
/platform/SUNW,Sun-Blade-100/lib/libmd5_psr.so.1

Regards,

Dan

comp.lang.ruby

[ANN] rocaml 0.6.0: fast, easy Ruby extensions in Objective Caml

Mauricio Fernández

Daniel Berger

Daniel Berger

Mauricio Fernández

Daniel Berger

Mauricio Fernández

Mauricio Fernández

Daniel Berger

Mauricio Fernández

Daniel Berger

Daniel Berger

x Login to ForumsZone