[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.lisp

can I use the asdf :around-compile to work around the externalized objects limitation of compile-file

Jim Newton

2/29/2016 9:48:00 AM

Can I use the :around-compile feature of asdf to work around this limitation of compile-file?

I have a deftype in my program which has a side effect, for example

(deftype rte (pattern)
`(and sequence
(satisfies ,(calc-match-function pattern))))

The call to calc-match-function returns a dependably named symbol which has thus
had its symbol-function set to a generated and compiled function. The code for the
function is created as a function of the pattern, and the resulting lambda expression has
been passed to eval to create a function object. If calc-match-function is called a second
time with the same pattern, it simply returns the calculated name, but doesn't re-generate nor
re-comiple the function.

This works fine if the code is loaded from source, but not if loaded from fasl. Why,
because the derived function exists in the lisp where the function is compiled, but not necessarily
in the lisp which loads the fasl.

This is a known limitation of compile-file:
http://www.lispworks.com/documentation/HyperSpec/Body/f_...
Programs to be compiled by the file compiler must only contain externalizable objects; for details on such objects, see Section 3.2.4 (Literal Objects in Compiled Files). For information on how to extend the set of externalizable objects, see the function make-load-form and Section 3.2.4.4 (Additional Constraints on Externalizable Objects).


What I wonder is whether I can use the :around-compile asdf hook to notice which
such named function got created by compiling the file, and serialize code "somewhere"
which will define these functions at fasl load time?
13 Answers

Kaz Kylheku

2/29/2016 4:01:00 PM

0

On 2016-02-29, Jim Newton <jimka.issy@gmail.com> wrote:
> Can I use the :around-compile feature of asdf to work around this limitation of compile-file?
>
> I have a deftype in my program which has a side effect, for example
>
> (deftype rte (pattern)
> `(and sequence
> (satisfies ,(calc-match-function pattern))))
>
> The call to calc-match-function returns a dependably named symbol which has thus
> had its symbol-function set to a generated and compiled function. The code for the
> function is created as a function of the pattern, and the resulting lambda expression has
> been passed to eval to create a function object. If calc-match-function is called a second
> time with the same pattern, it simply returns the calculated name, but doesn't re-generate nor
> re-comiple the function.
>
> This works fine if the code is loaded from source, but not if loaded from fasl. Why,
> because the derived function exists in the lisp where the function is compiled, but not necessarily
> in the lisp which loads the fasl.

Common Lisp provides a compile-and-load friendly mechanism for
calculating a value once and then returning the calculated value.
Look at LOAD-TIME-VALUE.

LOAD-TIME-VALUE can calculate objects which are not externalizable
in the compiled format. This is because the construction of these
objects is triggered at load time.

If you absolutely must calculate this value at compile time, because
(for instance) it depends on some data in the build environment, then
LOAD-TIME-VALUE won't work. It's possible that some of the inputs could
be split? Identify the minimum that you need from compile time, and
encode that in your object file as simple types that are externalizable.
Then use LOAD-TIME-VALUE to construct the non-externalizable object
(function or whatever) out of those compile-time-captured pieces.

Jim Newton

3/1/2016 9:25:00 AM

0


> Common Lisp provides a compile-and-load friendly mechanism for
> calculating a value once and then returning the calculated value.
> Look at LOAD-TIME-VALUE.
>


Thanks, this looks interesting. I'll investigate whether it can be used inside a deftype....

Jim Newton

3/2/2016 4:11:00 PM

0

Hi Kaz, it is not clear to me how to use LOAD-TIME-VALUE. Can you or someone help me
with this? I've uploaded a pretty simplified example of what I'm trying to do.

https://www.dropbox.com/s/h44vsl54yyo4ya7/great...

The tar file contains a greater.asd file.
When I load the asdf system the first time, and call the function
CL-USER> (greater:test)
It returns (nil nil 1 1 17)

But when I restart lisp, and now load the system, (i.e., loading from fasl files)
and call (greater:test)
I get the following:

The function GREATER-TYPES::|greater-10| is undefined.
[Condition of type UNDEFINED-FUNCTION]

Restarts:
0: [RETRY] Retry SLIME REPL evaluation request.
1: [*ABORT] Return to SLIME's top level.
2: [ABORT] abort thread (#<THREAD "repl-thread" RUNNING {1012EB8003}>)

Backtrace:
0: ("undefined function")
1: (GREATER::F 11 12) [tl,external]
2: (GREATER:TEST)
3: (SB-INT:SIMPLE-EVAL-IN-LEXENV (GREATER:TEST) #<NULL-LEXENV>)
4: (EVAL (GREATER:TEST))
--more--




On Monday, February 29, 2016 at 5:01:37 PM UTC+1, Kaz Kylheku wrote:
>
> Common Lisp provides a compile-and-load friendly mechanism for
> calculating a value once and then returning the calculated value.
> Look at LOAD-TIME-VALUE.
>
> LOAD-TIME-VALUE can calculate objects which are not externalizable
> in the compiled format. This is because the construction of these
> objects is triggered at load time.
>

Kaz Kylheku

3/2/2016 4:48:00 PM

0

On 2016-03-02, Jim Newton <jimka.issy@gmail.com> wrote:
> Hi Kaz, it is not clear to me how to use LOAD-TIME-VALUE. Can you or someone help me
> with this? I've uploaded a pretty simplified example of what I'm trying to do.
>
> https://www.dropbox.com/s/h44vsl54yyo4ya7/great...
>
> The tar file contains a greater.asd file.
> When I load the asdf system the first time, and call the function
> CL-USER> (greater:test)
> It returns (nil nil 1 1 17)
>
> But when I restart lisp, and now load the system, (i.e., loading from fasl files)
> and call (greater:test)
> I get the following:
>
> The function GREATER-TYPES::|greater-10| is undefined.
> [Condition of type UNDEFINED-FUNCTION]

This is almost certainly because you're interning your own symbols
and defining them as functions right inside the type expander.

The compiler will not see these dynamically created functions,
and so it won't propagate compiled versions of them into the FASL.

A quick and easy solution that will work isn't popping into my head,
though.

Jim Newton

3/3/2016 9:11:00 AM

0

Yes, I understand that the creation of the functions in this dynamic way
presents problems for the compiler.

My original question was whether I can use the :around-compile option in asdf to solve the problem.
E.g., I could write a function which would serialize all these functions, and print them to a file.
That would be a file which :around-compile creates, but it would also need to be listed
as a :component of the system which asdf is trying to load. :around-compile would have to
figure out which such functions are created by loading system-1, and then dump a file which
which would be loaded by system-2.

I don't know how to ask inside the :around-compile function, "What is the full path of
the the file xyzzy.lisp which you are going to load in the next system?"

Another problem with this approach is a meta-circularity issue. The name of the :around-compile function
will be something which is defined inside the package i'm trying to load. Thus the function is not
yet defined at the time the .asd file gets read.

>
> This is almost certainly because you're interning your own symbols
> and defining them as functions right inside the type expander.
>
> The compiler will not see these dynamically created functions,
> and so it won't propagate compiled versions of them into the FASL.
>
> A quick and easy solution that will work isn't popping into my head,
> though.

Madhu

3/3/2016 1:35:00 PM

0

time for a :jim-newton keyword option for asdf? fare will add it
in the next release. tip: don't use asdf

BTW do you know how to solve this WITHOUT asdf?

see the thread "Re: passing values from compile-time to load-time" from
2006

Kaz Kylheku

3/3/2016 8:19:00 PM

0

On 2016-03-03, Jim Newton <jimka.issy@gmail.com> wrote:
> Yes, I understand that the creation of the functions in this dynamic way
> presents problems for the compiler.

One possible solution is to drop the requirement that these functions
must go into FASL image as if they were file-compiled.

Lisp gives you a compiler that you can access at any time, as late as
you want.

You can stage the compilation of these functions to load time.

(That is perhaps where LOAD-TIME-VALUE could come in handy.)

Jim Newton

3/4/2016 1:13:00 PM

0

Yes, I wonder whether this is possible. But to use load-time-value, there must be a place
in the code where one usage of LOAD-TIME-VALUE occurs per function needing to be defined,
or else one LOAD-TIME-VALUE which contains several serialized definitions.

If the type is being expanded as a result of a declaration within a function, or with an invocation of THE,
it isn't clear where the LOAD-TIME-VALUE could go.

Is there a way to tell ASDF, that loading a given system has or might have created a file
which now needs to be loaded from source but doesn't need to be compiled?
I think the answer is NO.


> Lisp gives you a compiler that you can access at any time, as late as
> you want.
>
> You can stage the compilation of these functions to load time.
>
> (That is perhaps where LOAD-TIME-VALUE could come in handy.)

taruss

3/4/2016 5:12:00 PM

0

On Friday, March 4, 2016 at 5:12:55 AM UTC-8, Jim Newton wrote:

> Is there a way to tell ASDF, that loading a given system has or might have created a file
> which now needs to be loaded from source but doesn't need to be compiled?
> I think the answer is NO.

I'm pretty sure the answer is yes, but I'm not an ASDF guru.
But briefly looking at the manual, it seems like it should be flexible enough to
do what you want, although maybe not the optional file part. Perhaps
:static-file will help with that?

https://common-lisp.net/project/asdf/asdf/A-more-involved-example.html#A-more-involv...

Kaz Kylheku

3/4/2016 9:21:00 PM

0

On 2016-03-04, Jim Newton <jimka.issy@gmail.com> wrote:
> Yes, I wonder whether this is possible. But to use load-time-value, there must be a place
> in the code where one usage of LOAD-TIME-VALUE occurs per function needing to be defined,
> or else one LOAD-TIME-VALUE which contains several serialized definitions.

One possiblity might be to simply keep the functions as anonymous. The
load-time-value form is inserted at the call site.

(funcall (load-time-value (compile (lambda (...)...) nil))
args ...)

This load-time-value is evaluated once at load time, so subsequent
evaluations of the funcall form just see the resulting function.

With some additional hacking, multiple such call sites using a common
function can be made to share an actual function.

(funcall (load-time-value (produce-the-right-function 'symbol-123))
args ...)

The task is then to somehow gather up and stage the materials from
compile time so that they are accessible at load time to this
produce-the-right-function dispatcher.