[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.lisp

understanding the cffi search path.

Jim Newton

6/16/2015 12:12:00 PM

I'm trying to figure out why dlopen() cannot find my .dylib file.
I looked in the documentation of cffi, but I didn't see the explanation of how the
search path is treated as far as recursive search is concerned.

I get the following error when trying to load my system. It would be great if someone
could give me some insight into how this is supposed to work.

debugger invoked on a CFFI:LOAD-FOREIGN-LIBRARY-ERROR:
Unable to load foreign library (LIB-MAGICK-WAND).
Error opening shared object "libMagickWand.dylib":
dlopen(libMagickWand.dylib, 10): image not found.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.


Indeed libMagickWand.dylib does exist, but is in a strange place.
csh> ls -l /usr/local/Cellar/imagemagick/6.9.1-4/lib/*.dylib
-r--r--r-- 1 jnewton staff 500428 Jun 16 12:01 /usr/local/Cellar/imagemagick/6.9.1-4/lib/libMagick++-6.Q16.6.dylib
lrwxr-xr-x 1 jnewton staff 25 Jun 3 13:38 /usr/local/Cellar/imagemagick/6.9.1-4/lib/libMagick++-6.Q16.dylib -> libMagick++-6.Q16.6.dylib
lrwxr-xr-x 1 jnewton staff 25 Jun 16 13:33 /usr/local/Cellar/imagemagick/6.9.1-4/lib/libMagick++.dylib -> libMagick++-6.Q16.6.dylib
-r--r--r-- 1 jnewton staff 1679688 Jun 16 12:01 /usr/local/Cellar/imagemagick/6.9.1-4/lib/libMagickCore-6.Q16.2.dylib
lrwxr-xr-x 1 jnewton staff 27 Jun 3 13:38 /usr/local/Cellar/imagemagick/6.9.1-4/lib/libMagickCore-6.Q16.dylib -> libMagickCore-6.Q16.2.dylib
lrwxr-xr-x 1 jnewton staff 27 Jun 16 13:34 /usr/local/Cellar/imagemagick/6.9.1-4/lib/libMagickCore.dylib -> libMagickCore-6.Q16.2.dylib
-r--r--r-- 1 jnewton staff 1070632 Jun 16 12:01 /usr/local/Cellar/imagemagick/6.9.1-4/lib/libMagickWand-6.Q16.2.dylib
lrwxr-xr-x 1 jnewton staff 27 Jun 3 13:38 /usr/local/Cellar/imagemagick/6.9.1-4/lib/libMagickWand-6.Q16.dylib -> libMagickWand-6.Q16.2.dylib
lrwxr-xr-x 1 jnewton staff 27 Jun 16 13:34 /usr/local/Cellar/imagemagick/6.9.1-4/lib/libMagickWand.dylib -> libMagickWand-6.Q16.2.dylib
csh>

The default value of cffi:*foreign-library-directories* inside my lisp image is the following:


( (CFFI::EXPLODE-PATH-ENVIRONMENT-VARIABLE "LD_LIBRARY_PATH")
(CFFI::EXPLODE-PATH-ENVIRONMENT-VARIABLE "DYLD_LIBRARY_PATH") (UIOP/OS:GETCWD)
(CFFI::DARWIN-FALLBACK-LIBRARY-PATH))

But even if I push /usr/local/Cellar/ onto the list, I still get the same error from dlopen().

(#P"/usr/local/Cellar/"
(CFFI::EXPLODE-PATH-ENVIRONMENT-VARIABLE "LD_LIBRARY_PATH")
(CFFI::EXPLODE-PATH-ENVIRONMENT-VARIABLE "DYLD_LIBRARY_PATH") (UIOP/OS:GETCWD)
(CFFI::DARWIN-FALLBACK-LIBRARY-PATH))
7 Answers

Didier Verna

6/16/2015 1:00:00 PM

0

Jim Newton wrote:

> I'm trying to figure out why dlopen() cannot find my .dylib file. I
> looked in the documentation of cffi, but I didn't see the explanation
> of how the search path is treated as far as recursive search is
> concerned.
>
> I get the following error when trying to load my system. It would be
> great if someone could give me some insight into how this is supposed
> to work.
>
> debugger invoked on a CFFI:LOAD-FOREIGN-LIBRARY-ERROR:
> Unable to load foreign library (LIB-MAGICK-WAND).
> Error opening shared object "libMagickWand.dylib":
> dlopen(libMagickWand.dylib, 10): image not found.

It is possible that the problem is not in loading your dylib, but
another one on which it depends. CFFI just won't give you a different
error message in this case, which is confusing.

On Mac, try using 'otool -L' (the equivalent of Unix's ldd) on your
library, and see if it has all its dependencies properly installed.

--
My new Jazz CD entitled "Roots and Leaves" is out!
Check it out: http://didierverna.com/records/roots-and-...

Lisp, Jazz, Aïkido: http://www.didier...

Jim Newton

6/16/2015 1:36:00 PM

0

At first glance it seems the dependences are there, and indeed all files (not broken links)

csh> otool -L libMagickWand.dylib
libMagickWand.dylib:
/usr/local/lib/libMagickWand-6.Q16.2.dylib (compatibility version 3.0.0, current version 3.0.0)
/usr/local/Cellar/imagemagick/6.9.1-4/lib/libMagickCore-6.Q16.2.dylib (compatibility version 3.0.0, current version 3.0.0)
/usr/local/lib/libfreetype.6.dylib (compatibility version 18.0.0, current version 18.4.0)
/usr/lib/liblzma.5.dylib (compatibility version 6.0.0, current version 6.3..0)
/usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.5)
/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
/usr/local/lib/libltdl.7.dylib (compatibility version 11.0.0, current version 11.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)


On Tuesday, June 16, 2015 at 2:59:55 PM UTC+2, Didier Verna wrote:
>
> It is possible that the problem is not in loading your dylib, but
> another one on which it depends. CFFI just won't give you a different
> error message in this case, which is confusing.
>
> On Mac, try using 'otool -L' (the equivalent of Unix's ldd) on your
> library, and see if it has all its dependencies properly installed.
>
> --
> My new Jazz CD entitled "Roots and Leaves" is out!
> Check it out: http://didierverna.com/records/roots-and-...
>
> Lisp, Jazz, Aïkido: http://www.didier...

Jim Newton

6/17/2015 8:01:00 AM

0

I found the issue.
I don't think it is a lisp issue at all, I think it is just the way UNIX handles loading dynamic libraries. Although I might be mistaken.
It turns out the directories in cffi:*foreign-library-directories* do NOT get searched recursively. I have to put EVERY directory name in the list which contains a .dylib file which I'm interested in.
This can be done either in lisp or the shell.
In lisp:
CL> (pushnew #p"/usr/local/Cellar/imagemagick/6.9.1-4/lib/" cffi:*foreign-library-directories*)

or in the shell before starting lisp.
sh> export LD_LIBRARY_PATH="/usr/local/Cellar/imagemagick/6.9.1-4/lib"

Jim Newton

6/17/2015 8:07:00 AM

0

In my opinion the error message I got back could be a little more enlightening.
I didn't know whether the message meant that the file was found but did not contain an image, (or image could not be loaded). That seems to be Didier's interpretations as well, i.e., that the image failed to load from some reason such as missing dependencies. Or whether the file itself was not found, which was the case.

> debugger invoked on a CFFI:LOAD-FOREIGN-LIBRARY-ERROR:
> Unable to load foreign library (LIB-MAGICK-WAND).
> Error opening shared object "libMagickWand.dylib":
> dlopen(libMagickWand.dylib, 10): image not found.

Would it be possible to improve this error message in CFFI? Or is it really the case that the underlying OS system call didn't give any better useful information to report? If CFFI could have told me that libMagickWand.dylib was not found in any of the following directories ... That would have helped.

Jim Newton

5/26/2016 7:50:00 AM

0

I neglected to explain my solution to this in the original thread. Now it's happening again.

> debugger invoked on a CFFI:LOAD-FOREIGN-LIBRARY-ERROR:
> Unable to load foreign library (LIB-MAGICK-WAND).
> Error opening shared object "libMagickWand.dylib":
> dlopen(libMagickWand.dylib, 10): image not found.

The solution in general is to set the environment variable LD_LIBRARY_PATH to the full
path of the directory which contains libMagickWand.dylib. Unfortunately MAC-OS (or at
least csh and sh on the mac) refuse to LD_LIBRARY_PATH. It is indeed frustrating. Try it!
If anyone can figure out how to set those environment variables from the shell on the
mac, I'd love to hear how you did it.

I found several articles of people saying you shouldn't depend on LD_LIBRARY_PATH.
Nevertheless cffi does.

The way I'm doing this now, is in my .emacs file, before loading slime.


;; ;; Setup load-path, autoloads and your lisp system
;; ;; Not needed if you install SLIME via MELPA
(setenv "LD_LIBRARY_PATH" "/usr/local/Cellar/imagemagick/6.9.1-4/lib")
(require 'slime-autoloads)
(load "~/emacs.d/slime-rc.el")

Pascal J. Bourguignon

5/26/2016 5:36:00 PM

0

Jim Newton <jimka.issy@gmail.com> writes:

> I neglected to explain my solution to this in the original thread. Now it's happening again.
>
>> debugger invoked on a CFFI:LOAD-FOREIGN-LIBRARY-ERROR:
>> Unable to load foreign library (LIB-MAGICK-WAND).
>> Error opening shared object "libMagickWand.dylib":
>> dlopen(libMagickWand.dylib, 10): image not found.
>
> The solution in general is to set the environment variable LD_LIBRARY_PATH to the full
> path of the directory which contains libMagickWand.dylib. Unfortunately MAC-OS (or at
> least csh and sh on the mac) refuse to LD_LIBRARY_PATH. It is indeed frustrating. Try it!
> If anyone can figure out how to set those environment variables from the shell on the
> mac, I'd love to hear how you did it.

See:
https://developer.apple.com/library/mac/documentation/MacOSX/Conceptual/BPRuntimeConfig/Articles/Environmen...

> I found several articles of people saying you shouldn't depend on LD_LIBRARY_PATH.
> Nevertheless cffi does.

You can also sometimes set the environment variable inside the process
(depending on the implementation). You'd have to do that before
initializing cffi. Instead, you can set directly the cffi special
variables pointing to the library directories.


> The way I'm doing this now, is in my .emacs file, before loading slime.

You could do it in ~/.swank.lisp


> ;; ;; Setup load-path, autoloads and your lisp system
> ;; ;; Not needed if you install SLIME via MELPA
> (setenv "LD_LIBRARY_PATH" "/usr/local/Cellar/imagemagick/6.9.1-4/lib")
> (require 'slime-autoloads)
> (load "~/emacs.d/slime-rc.el")


--
__Pascal Bourguignon__ http://www.informat...
â??The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.� -- Carl Bass CEO Autodesk

Jim Newton

5/27/2016 9:38:00 AM

0

For completeness and future reference, I've updated my .sbclrc file to include.
This is where I discovered that setenv is not setf-able. :-(

(sb-posix:setenv "LD_LIBRARY_PATH"
(let ((ld-path (sb-posix:getenv "LD_LIBRARY_PATH")))
(if (not ld-path)
"/usr/local/Cellar/imagemagick/6.9.1-4/lib"
(concatenate 'string "/usr/local/Cellar/imagemagick/6.9.1-4/lib:"
ld-path)))
1)