Asp Forum - sbcl) high memory needs during compilation

Antsan

12/27/2015 3:13:00 PM

I've been working on my own programming language for a while now and I am really
happy with how it is turning out, but unfortunately I've now been forced to start
SBCL with a dynamic space size of 2GB for the compilation not to fail due to
running out of dynamic space. That seems kind of excessive to me.
The source is here: https://bitbucket.org/thomas_bartscher/...

I think the problem lies within POSLIN-INSTALL-PRIMS which basically has
definitions for every last primary operation inlined into it.
https://bitbucket.org/thomas_bartscher/.../b6f0c887e85afc6e0fd3cfb533ec598f3e0fb871/source/interaction/new-poslin.lisp
(defmacro poslin-install-prims (&rest standards)
`(progn
(setf (op-env path)
(with (op-env path)
:. (binding <noop> "no op")))
,@(loop for standard in standards
append
(mapcar #`(progn
(setf (op-env path)
(with (op-env path)
',(intern (first a1)
:keyword)
(binding (<prim> (lambda ()
,(fourth a1))
,(first a1))
,(third a1))))
,@(when (second a1)
`((setf (imm-env path)
(with (imm-env path)
',(intern (first a1)
:keyword))))))
(symbol-value standard)))))

Currently all the defined operations can be found here:
https://bitbucket.org/thomas_bartscher/.../b6f0c887e85afc6e0fd3cfb533ec598f3e0fb871/source/prims/prim.lisp
There's currently 85 primary operations.

Does anybody have any idea how I can reduce the space needed for compiling this?

3 Answers

Pascal Costanza

12/27/2015 3:43:00 PM

On 27/12/2015 16:13, Antsan wrote:
> I've been working on my own programming language for a while now and I am really
> happy with how it is turning out, but unfortunately I've now been forced to start
> SBCL with a dynamic space size of 2GB for the compilation not to fail due to
> running out of dynamic space. That seems kind of excessive to me.
> The source is here: https://bitbucket.org/thomas_bartscher/...
>
> I think the problem lies within POSLIN-INSTALL-PRIMS which basically has
> definitions for every last primary operation inlined into it.
> https://bitbucket.org/thomas_bartscher/.../b6f0c887e85afc6e0fd3cfb533ec598f3e0fb871/source/interaction/new-poslin.lisp
> (defmacro poslin-install-prims (&rest standards)
> `(progn
> (setf (op-env path)
> (with (op-env path)
> :. (binding <noop> "no op")))
> ,@(loop for standard in standards
> append
> (mapcar #`(progn
> (setf (op-env path)
> (with (op-env path)
> ',(intern (first a1)
> :keyword)
> (binding (<prim> (lambda ()
> ,(fourth a1))
> ,(first a1))
> ,(third a1))))
> ,@(when (second a1)
> `((setf (imm-env path)
> (with (imm-env path)
> ',(intern (first a1)
> :keyword))))))
> (symbol-value standard)))))
>
> Currently all the defined operations can be found here:
> https://bitbucket.org/thomas_bartscher/.../b6f0c887e85afc6e0fd3cfb533ec598f3e0fb871/source/prims/prim.lisp
> There's currently 85 primary operations.
>
> Does anybody have any idea how I can reduce the space needed for compiling this?

I don't know what your code does, so the following may be irrelevant.

However, one thing to try is this: The loop form appends partial results
returned from a mapcar invocation. Append here means that each partial
result will be copied, except the last, just like in the regular append
function. Since the result of mapcar is guaranteed to be a fresh list,
this is wasteful. You might as well use nconc and avoid the copying:
(loop for standard in standards nconc (mapcar ...))

If it's possible to modify the lists passed to mapcar in place, it may
be better to drop the mapcar invocation and instead loop over these
lists and indeed modify them in place. But this is more work, and may
not be safe.

Pascal

--
My website: http:/...
Common Lisp Document Repository: http://cdr.eu...
Closer to MOP & ContextL: http://common-lisp.net/proje...
The views expressed are my own, and not those of my employer.

Pascal J. Bourguignon

12/27/2015 5:12:00 PM

Pascal Costanza <pc@p-cos.net> writes:

> On 27/12/2015 16:13, Antsan wrote:
>> I've been working on my own programming language for a while now and I am really
>> happy with how it is turning out, but unfortunately I've now been forced to start
>> SBCL with a dynamic space size of 2GB for the compilation not to fail due to
>> running out of dynamic space. That seems kind of excessive to me.
>> The source is here: https://bitbucket.org/thomas_bartscher/...
>>
>> I think the problem lies within POSLIN-INSTALL-PRIMS which basically has
>> definitions for every last primary operation inlined into it.
>> https://bitbucket.org/thomas_bartscher/.../b6f0c887e85afc6e0fd3cfb533ec598f3e0fb871/source/interaction/new-poslin.lisp
>> (defmacro poslin-install-prims (&rest standards)
>> `(progn
>> (setf (op-env path)
>> (with (op-env path)
>> :. (binding <noop> "no op")))
>> ,@(loop for standard in standards
>> append
>> (mapcar #`(progn
>> (setf (op-env path)
>> (with (op-env path)
>> ',(intern (first a1)
>> :keyword)
>> (binding (<prim> (lambda ()
>> ,(fourth a1))
>> ,(first a1))
>> ,(third a1))))
>> ,@(when (second a1)
>> `((setf (imm-env path)
>> (with (imm-env path)
>> ',(intern (first a1)
>> :keyword))))))
>> (symbol-value standard)))))
>>
>> Currently all the defined operations can be found here:
>> https://bitbucket.org/thomas_bartscher/.../b6f0c887e85afc6e0fd3cfb533ec598f3e0fb871/source/prims/prim.lisp
>> There's currently 85 primary operations.
>>
>> Does anybody have any idea how I can reduce the space needed for compiling this?
>
> I don't know what your code does, so the following may be irrelevant.
>
> However, one thing to try is this: The loop form appends partial
> results returned from a mapcar invocation. Append here means that each
> partial result will be copied, except the last, just like in the
> regular append function. Since the result of mapcar is guaranteed to
> be a fresh list, this is wasteful. You might as well use nconc and
> avoid the copying: (loop for standard in standards nconc (mapcar ...))
>
> If it's possible to modify the lists passed to mapcar in place, it may
> be better to drop the mapcar invocation and instead loop over these
> lists and indeed modify them in place. But this is more work, and may
> not be safe.

Or using (loop â?¦ :nconc (mapcar â?¦)) this will always be safe, since
mapcar creates a fresh list.

But in anycase, I doubt that's where the need for memory comes from,
since from one iteration to another, the garbage collector can reclaim
the list returned by the previous mapcar.

--
__Pascal Bourguignon__ http://www.informat...
â??The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.â? -- Carl Bass CEO Autodesk

Antsan

12/27/2015 10:05:00 PM

Am Sonntag, 27. Dezember 2015 18:12:19 UTC+1 schrieb informatimago:
> Pascal Costanza <pc@p-cos.net> writes:
>
> > On 27/12/2015 16:13, Antsan wrote:
> >> I've been working on my own programming language for a while now and I am really
> >> happy with how it is turning out, but unfortunately I've now been forced to start
> >> SBCL with a dynamic space size of 2GB for the compilation not to fail due to
> >> running out of dynamic space. That seems kind of excessive to me.
> >> The source is here: https://bitbucket.org/thomas_bartscher/...
> >>
> >> I think the problem lies within POSLIN-INSTALL-PRIMS which basically has
> >> definitions for every last primary operation inlined into it.
> >> https://bitbucket.org/thomas_bartscher/.../b6f0c887e85afc6e0fd3cfb533ec598f3e0fb871/source/interaction/new-poslin.lisp
> >> (defmacro poslin-install-prims (&rest standards)
> >> `(progn
> >> (setf (op-env path)
> >> (with (op-env path)
> >> :. (binding <noop> "no op")))
> >> ,@(loop for standard in standards
> >> append
> >> (mapcar #`(progn
> >> (setf (op-env path)
> >> (with (op-env path)
> >> ',(intern (first a1)
> >> :keyword)
> >> (binding (<prim> (lambda ()
> >> ,(fourth a1))
> >> ,(first a1))
> >> ,(third a1))))
> >> ,@(when (second a1)
> >> `((setf (imm-env path)
> >> (with (imm-env path)
> >> ',(intern (first a1)
> >> :keyword))))))
> >> (symbol-value standard)))))
> >>
> >> Currently all the defined operations can be found here:
> >> https://bitbucket.org/thomas_bartscher/.../b6f0c887e85afc6e0fd3cfb533ec598f3e0fb871/source/prims/prim.lisp
> >> There's currently 85 primary operations.
> >>
> >> Does anybody have any idea how I can reduce the space needed for compiling this?
> >
> > I don't know what your code does, so the following may be irrelevant.
> >
> > However, one thing to try is this: The loop form appends partial
> > results returned from a mapcar invocation. Append here means that each
> > partial result will be copied, except the last, just like in the
> > regular append function. Since the result of mapcar is guaranteed to
> > be a fresh list, this is wasteful. You might as well use nconc and
> > avoid the copying: (loop for standard in standards nconc (mapcar ...))
> >
> > If it's possible to modify the lists passed to mapcar in place, it may
> > be better to drop the mapcar invocation and instead loop over these
> > lists and indeed modify them in place. But this is more work, and may
> > not be safe.
>
>
> Or using (loop ... :nconc (mapcar ...)) this will always be safe, since
> mapcar creates a fresh list.
>
> But in anycase, I doubt that's where the need for memory comes from,
> since from one iteration to another, the garbage collector can reclaim
> the list returned by the previous mapcar.
>
> --
> __Pascal Bourguignon__ http://www.informat...
> "The factory of the future will have only two employees, a man and a
> dog. The man will be there to feed the dog. The dog will be there to
> keep the man from touching the equipment." -- Carl Bass CEO Autodesk

Yeah, I exchanged APPEND for NCONC and compiling the interpreter still failed
the same way as ever.

Compilation actually fails with this file:
https://bitbucket.org/thomas_bartscher/.../b6f0c887e85afc6e0fd3cfb533ec598f3e0fb871/source/repl/repl.lisp
I think with REPL-DYN. As that one expands NEW-POSLIN which includes
POSLIN-INSTALL-PRIMS I think that's no surprise.

I compiled poslin with enough heap space and then loaded it into an sbcl with
the standard settings. Executing
> (new-poslin *prim*)
there does lead to the crash while
> (new-poslin)
alone doesn't. So I am pretty confident that the problem lies withing
POSLIN-INSTALL-PRIMS. The expansion of (poslin-install-prims *prim*) takes up
1167 lines. 86 of those are just "(progn", but still.

I think I'd need to compile the primary operations of poslin separately, but I
don't quite see how I would do that. It's not like I can build up the map at
compile time - as far as I know Common Lisp doesn't support expanding into
anything but lists or literals.

comp.lang.lisp

sbcl) high memory needs during compilation

Antsan

Pascal Costanza

Pascal J. Bourguignon

Antsan

x Login to ForumsZone