Asp Forum - Reading extended ANSI Characters with Clisp in Windows

P. Taine

6/6/2015 7:38:00 PM

I thought I had asked this before, but I can't find an aswer.

I am trying to use characters above 128 in strings, such as the Yen symbol (¥),
but READ chokes on some of them. I need these in various places, including im
strings in a "case" macro.

In the read-eval-print loop NOTHING above 128 is accepted, which includes all
accented characters!

I note that READ-LINE is (somewhat) happy with these, but in the DOS window
displays them incorrectly, though PRINT-LINE seems to give the correct results.

How do I make these safe for READ, and how do I specify the code-page to be used
in the DOS window?

What am I missing in the documentation?

P. Taine

13 Answers

Pascal J. Bourguignon

6/6/2015 10:42:00 PM

P. Taine <user@domaine.invalid> writes:

> I thought I had asked this before, but I can't find an aswer.
>
> I am trying to use characters above 128 in strings, such as the Yen symbol (Â¥),
> but READ chokes on some of them. I need these in various places, including im
> strings in a "case" macro.
>
> In the read-eval-print loop NOTHING above 128 is accepted, which includes all
> accented characters!
>
> I note that READ-LINE is (somewhat) happy with these, but in the DOS window
> displays them incorrectly, though PRINT-LINE seems to give the correct results.
>
> How do I make these safe for READ, and how do I specify the code-page to be used
> in the DOS window?
>
> What am I missing in the documentation?

You are missing the :EXTERNAL-FORMAT parameter of OPEN, WITH-OPEN-FILE,
etc.

Also, check the customization variables:
CUSTOM:*PATHNAME-ENCODING*
CUSTOM:*DEFAULT-FILE-ENCODING*
CUSTOM:*TERMINAL-ENCODING*
CUSTOM:*MISC-ENCODING*
#+FFI CUSTOM:*FOREIGN-ENCODING*

Notice that clisp doesn't use keywords to specify external-formats, but
objects built with EXT:MAKE-ENCODING. Some of them are predefined in
the CHARSET package.

--
__Pascal Bourguignon__ http://www.informat...
â??The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.â? -- Carl Bass CEO Autodesk

P. Taine

6/7/2015 12:14:00 AM

On Sun, 07 Jun 2015 00:41:35 +0200, "Pascal J. Bourguignon"
<pjb@informatimago.com> wrote:

>You are missing the :EXTERNAL-FORMAT parameter of OPEN, WITH-OPEN-FILE,
>etc.
>
>Also, check the customization variables:
>CUSTOM:*PATHNAME-ENCODING*
>CUSTOM:*DEFAULT-FILE-ENCODING*
>CUSTOM:*TERMINAL-ENCODING*
>CUSTOM:*MISC-ENCODING*
>#+FFI CUSTOM:*FOREIGN-ENCODING*
>
>Notice that clisp doesn't use keywords to specify external-formats, but
>objects built with EXT:MAKE-ENCODING. Some of them are predefined in
>the CHARSET package.

I will explore those. I stumbled on the EXT:MAKE_ENCODING function, but was
unable to determine what to do with the object it created. I'll read up on
EXTERNAL-FORMAT and the CUSTOM:etc variable and hope that understanding dawns.

Thank you,

P. Taine

Pascal J. Bourguignon

6/7/2015 12:39:00 AM

P. Taine <user@domaine.invalid> writes:

> On Sun, 07 Jun 2015 00:41:35 +0200, "Pascal J. Bourguignon"
> <pjb@informatimago.com> wrote:
>
>>You are missing the :EXTERNAL-FORMAT parameter of OPEN, WITH-OPEN-FILE,
>>etc.
>>
>>Also, check the customization variables:
>>CUSTOM:*PATHNAME-ENCODING*
>>CUSTOM:*DEFAULT-FILE-ENCODING*
>>CUSTOM:*TERMINAL-ENCODING*
>>CUSTOM:*MISC-ENCODING*
>>#+FFI CUSTOM:*FOREIGN-ENCODING*
>>
>>Notice that clisp doesn't use keywords to specify external-formats, but
>>objects built with EXT:MAKE-ENCODING. Some of them are predefined in
>>the CHARSET package.
>
> I will explore those. I stumbled on the EXT:MAKE_ENCODING function, but was
> unable to determine what to do with the object it created. I'll read up on
> EXTERNAL-FORMAT and the CUSTOM:etc variable and hope that understanding dawns.

Basically, you put something like:

(setf custom:*pathname-encoding* (ext:make-encoding :charset charset:iso-8859-1
:line-terminator :unix))
(setf custom:*default-file-encoding* (ext:make-encoding :charset charset:utf-8
:line-terminator :unix))
(setf custom:*terminal-encoding* (ext:make-encoding :charset charset:utf-8
:line-terminator :unix))
(setf custom:*misc-encoding* (ext:make-encoding :charset charset:utf-8
:line-terminator :unix))
#+ffi (setf custom:*foreign-encoding* (ext:make-encoding :charset charset:iso-8859-1
:line-terminator :unix))
in your ~/.clisprc.lisp

You can also use the -E option, eg.: clisp -ansi -E utf-8

--
__Pascal Bourguignon__ http://www.informat...
â??The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.â? -- Carl Bass CEO Autodesk

P. Taine

6/7/2015 4:03:00 PM

On Sun, 07 Jun 2015 02:38:58 +0200, "Pascal J. Bourguignon"
<pjb@informatimago.com> wrote:

>P. Taine <user@domaine.invalid> writes:
>
>> On Sun, 07 Jun 2015 00:41:35 +0200, "Pascal J. Bourguignon"
>> <pjb@informatimago.com> wrote:
>>
>>>You are missing the :EXTERNAL-FORMAT parameter of OPEN, WITH-OPEN-FILE,
>>>etc.
>>>
>>>Also, check the customization variables:
>>>CUSTOM:*PATHNAME-ENCODING*
>>>CUSTOM:*DEFAULT-FILE-ENCODING*
>>>CUSTOM:*TERMINAL-ENCODING*
>>>CUSTOM:*MISC-ENCODING*
>>>#+FFI CUSTOM:*FOREIGN-ENCODING*
>>>
>>>Notice that clisp doesn't use keywords to specify external-formats, but
>>>objects built with EXT:MAKE-ENCODING. Some of them are predefined in
>>>the CHARSET package.
>>
>> I will explore those. I stumbled on the EXT:MAKE_ENCODING function, but was
>> unable to determine what to do with the object it created. I'll read up on
>> EXTERNAL-FORMAT and the CUSTOM:etc variable and hope that understanding dawns.
>
>Basically, you put something like:
>
> (setf custom:*pathname-encoding* (ext:make-encoding :charset charset:iso-8859-1
> :line-terminator :unix))
> (setf custom:*default-file-encoding* (ext:make-encoding :charset charset:utf-8
> :line-terminator :unix))
> (setf custom:*terminal-encoding* (ext:make-encoding :charset charset:utf-8
> :line-terminator :unix))
> (setf custom:*misc-encoding* (ext:make-encoding :charset charset:utf-8
> :line-terminator :unix))
> #+ffi (setf custom:*foreign-encoding* (ext:make-encoding :charset charset:iso-8859-1
> :line-terminator :unix))
>in your ~/.clisprc.lisp
>
>You can also use the -E option, eg.: clisp -ansi -E utf-8

I think I'm going to have to punt on this and change some of my parameters!

Note that I have zero UNIX experience and have never used C or any of its
descendants. I'm an old (24 years retired) IBM main-frame programmer, mostly in
assembler language, so much of the documentation goes over my head. Sorry.
For example, I'm not sure what "in you ~/clisprc.lisp" means.

My windows short-cut invoking clisp has been:
"C:\Program Files\clisp-2.49\clisp.exe" -K full -I -M MyLisp.mem -E ISO-8859-1
where "MyLisp.mem" is a saved image (is this the correct term?) after I have
loaded a bunch of macros and function which replicate some features of LISP/VM
which I am used to.

I am using the United States-International, so Ctrl-Alt-s should give me ß
(German sharp-s). When I type that into clisp I get an error "beep", and
nothing is echoed (or saved).

What I have tried.

I modified the invocation, first simply adding the -ansi, than changing
ISO-8859-1 to utf-8. Didn't help.

I ran the five setf expressions which you provided, but this also failed to
change the behavior.

I then fell further down the rabbit hole, trying to see which code-page the DOS
window was using. My attempt failed as follows

[6]> (ext:run-program "C:/windows/system32/chcp.com")
Parameter format not correct - /windows
1

Trying to set the code page had the same failure:

[5]> (ext:run-program "C:/windows/system32/chcp.com" :arguments '("1252"))
Parameter format not correct - /windows
1

So it seems that I'm missing some important point here. Since (as far as I can
tell) READ-LINE and WRITE-LINE seem to work (even if the characters appear
garbled internally, they look the same written as read) I'll just have to change
some of the constants in my programs to avoid them.

Thanks of the attempt to help me, I guess I'm just too dense.

P. Taine

Pascal J. Bourguignon

6/7/2015 8:02:00 PM

P. Taine <user@domaine.invalid> writes:

>>You can also use the -E option, eg.: clisp -ansi -E utf-8
>
> I think I'm going to have to punt on this and change some of my parameters!
>
> Note that I have zero UNIX experience and have never used C or any of its
> descendants. I'm an old (24 years retired) IBM main-frame programmer, mostly in
> assembler language, so much of the documentation goes over my head. Sorry.
> For example, I'm not sure what "in you ~/clisprc.lisp" means.

Sorry for the shortcuts I took.

http://www.clisp.org...enc...
http://www.clisp.org...enc...#path-enc
http://www.clisp.org...clisp.ht...
http://www.clisp.org...

I'm not much of a MS-Windows user myself. I don't know exactly where
the clisprc.lisp file is stored. One way to find out would be to evaluate:

(user-homedir-pathname)

Notice: the result may depend on how you launch clisp. Eg. I get
different user homedir pathnames when I double-click on the clisp icon
than when I launch it from a *shell* buffer in GNU emacs.

(user-homedir-pathname)

The file that is loaded by clisp, when it exists, on launch (unless you
run clisp with the -norc option), is:

(make-pathname :name ".clisprc" :type "lisp"
:defaults (user-homedir-pathname))

also, #P"~/.clisprc.lisp" should work equivalently (but it seems it
doesn't always do).

> My windows short-cut invoking clisp has been:
> "C:\Program Files\clisp-2.49\clisp.exe" -K full -I -M MyLisp.mem -E ISO-8859-1
> where "MyLisp.mem" is a saved image (is this the correct term?)

(yes).

> after I have
> loaded a bunch of macros and function which replicate some features of LISP/VM
> which I am used to.
>
> I am using the United States-International, so Ctrl-Alt-s should give me Ã?
> (German sharp-s). When I type that into clisp I get an error "beep", and
> nothing is echoed (or saved).
>
> What I have tried.
>
> I modified the invocation, first simply adding the -ansi, than changing
> ISO-8859-1 to utf-8. Didn't help.

-ansi is unrelated to encodings. I just advise you to always use -ansi,
since that prevents some deviations from the ANSI Common Lisp standard,
and therefore let you write more conforming code, that you may more
easily use with other CL implementations if you would even want to, or
if you share code with other programmers (eg. if you want to use CL
libraries available on the Internet).

> I ran the five setf expressions which you provided, but this also failed to
> change the behavior.
>
> I then fell further down the rabbit hole, trying to see which code-page the DOS
> window was using.

Good idea.

> My attempt failed as follows
>
> [6]> (ext:run-program "C:/windows/system32/chcp.com")
> Parameter format not correct - /windows
> 1
>
> Trying to set the code page had the same failure:
>
> [5]> (ext:run-program "C:/windows/system32/chcp.com" :arguments '("1252"))
> Parameter format not correct - /windows
> 1

I don't know much about MS-Windows, so tell me if I'm wrong, but I heard
that it uses \ as pathname component separator, and / as parameter
separator. Like:

c:\windows\system32\dir.exe/p

Therefore I would try:

(ext:run-program "C:\\windows\\system32\\chcp.com" :arguments '("1252"))

(Of course, some on MS-Windows programs translate / into \ in pathnames,
so much confusion ensue. Your only fault is to choose to use MS-Windows
instead of a sane OS like GNU/Linux (by the way, did you hear that nowadays,
even IBM uses MacOSX systems? (which are unix-based and include a lot of
GNU software too)
http://uk.businessinsider.com/ibm-is-buying-thousands-of-employees-macs-2...
).

Now the important point, is that the terminal encoding depends on the
_terminal_ (or terminal emulator) you use to communicate with clisp.
You should set the custom:*terminal-encoding* variable according to the
terminal you are currently using, and the current setting of encoding of
this terminal too, when it is parameterizable. Also, I guess that by
default clisp is able to detect the encoding configured for the
terminal, perhaps using the chcp program or some environment variable;
at least on unix systems, it definitely use environment variables when
present, unless the -E option is given, if you don't set it in the
..clisprc.lisp file.

Also, you will want to set different default encoding for the data
files and for the pathnames (and possibly for foreign data and
miscellaneous other situations).

-Edomain encoding

Specifies the encoding used for the given domain, overriding the
default which depends on the environment variables LC_ALL, LC_CTYPE,
LANG. -Edomain can be:

-Efile affecting CUSTOM:*DEFAULT-FILE-ENCODING*
-Epathname affecting CUSTOM:*PATHNAME-ENCODING*
-Eterminal affecting CUSTOM:*TERMINAL-ENCODING*
-Eforeign affecting CUSTOM:*FOREIGN-ENCODING*
-Emisc affecting CUSTOM:*MISC-ENCODING*
-E affecting all of the above.

(there's a little problem with -Epathname, in that it assumes that a
single encoding is used for all the file systems, but in general this is
not the case. For example on unix you can mount a MS-DOS file system
using eg. CP1250 encoding onto a root unix file system using iso-8859-1,
and then you can mount a MacOSX file system using utf-8 onto a directory
in the MS-DOS file system, and therefore obtain a pathname such as:

/mnt/unixâ?¢file-system/MS-DOS-mount-point/Â©MS/Mac-MP/photo-Ã©tÃ©.jpg
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ---------- .............
iso-8859-1 CP1250 utf-8

where the different components are encoded with different encodings.
Unfortunately, there's no known system that deals with this situation
properly (hence the advice to restrict yourself to ascii in pathnames)).

Anyways, the first thing to do is to determine or decide what encoding
you will use.

- determine what encoding is used by your terminal,
- determine what encoding is used by your file system for pathnames,
- decide what encoding you will use for your files (by default, this can
be changed file-by-file when you open them).

Then you can pass the right -E options when you launch clisp:

"C:\Program Files\clisp-2.49\clisp.exe" -K full -I -M MyLisp.mem -E ISO-8859-1 -Efile UTF-8 -Epathname WINDOWS-1250 -Eterminal CP850

Then, you can override those encoding settings in ~/.clisprc.lisp, or
at least those that are not dependent on the terminal. You can edit the
..clisprc.lisp file by evaluating:

(ed (make-pathname :name ".clisprc" :type "lisp"
:defaults (user-homedir-pathname)))

by default this opens a NotePad editor window on that file.
You can put something like:

(setf custom:*pathname-encoding* (ext:make-encoding :charset charset:windows-1250
:line-terminator :dos))
(setf custom:*default-file-encoding* (ext:make-encoding :charset charset:utf-8
:line-terminator :dos))
(setf custom:*misc-encoding* (ext:make-encoding :charset charset:iso-8859-1
:line-terminator :dos))
#+ffi (setf custom:*foreign-encoding* (ext:make-encoding :charset charset:iso-8859-1
:line-terminator :dos))

and possibly also set *terminal-encoding*, if you almost always use
clisp from the same terminal:

(setf custom:*terminal-encoding* (ext:make-encoding :charset charset:cp850
:line-terminator :dos))

of course, use the right charset, as previously determined or decided.

At this point, I should also ask about your lisp image. Who did build
it and how is it designed? It is possible in a lisp image, to set an
init-function that is called before entering the REPL, and in this
init-function, the lisp image could reset the CUSTOM:*TERMINAL-ENCODING*
variable. The lisp image could also contain other functions in
CUSTOM:*INIT-HOOKS*, but the rc file (.clisprc.lisp) is loaded after
*INIT-HOOKS*, but the init-function is called last.

You may also set other customization variables in this .clisprc.lisp
file, notably the custom:*ansi* flag, so you don't have to remember to
type -ansi everytime you launch clisp. Here is my setting:

(let ((counter 0))
(defun prompt-body ()
(format nil "~A[~D]"
(if (packagep *package*)
(first (sort (cons (package-name *package*) (package-nicknames *package*))
(function <=) :key (function length)))
"#<INVALID *PACKAGE*>")
(incf counter))))

(setf
custom:*ansi* t
custom:*coerce-fixnum-char-ansi* t
custom:*floating-point-contagion-ansi* t
custom:*merge-pathnames-ansi* t
custom:*parse-namestring-ansi* t
custom:*parse-namestring-dot-file* :name
custom:*print-pathnames-ansi* t
custom:*sequence-count-ansi* t

custom:*default-float-format* 'single-float
custom:*warn-on-floating-point-contagion* nil
custom:*suppress-check-redefinition* nil
custom:*deftype-depth-limit* nil
custom:*package-tasks-treat-specially* nil

custom:*pprint-first-newline* t
custom:*print-closure* nil
custom:*print-indent-lists* 1
custom:*print-pretty-fill* nil
custom:*print-rpars* nil
custom:*prompt-start* "CLISP/"
custom:*prompt-body* (function prompt-body)
custom:*prompt-finish* "> "

custom:*load-compiling* nil
custom:*load-paths* '(#p"")
custom:*load-obsolete-action* nil
custom:*compile-warnings* t

custom:*source-file-types* '("lisp" "lsp" "cl")
custom:*compiled-file-types* '("fas")

custom:*applyhook* nil
custom:*evalhook* nil
custom:*trace-indent* t
custom:*apropos-do-more* nil
custom:*apropos-matcher* nil

custom:*error-handler* nil
custom:*break-on-warnings* nil

custom:*inspect-browser* nil
custom:*inspect-frontend* :tty
custom:*inspect-length* 100000
custom:*inspect-print-length* 100000
custom:*inspect-print-level* 5
custom:*inspect-print-lines* 5

custom:*user-mail-address* "put@your.email.address.here"
custom:*current-language* 'i18n:english)

You may also configure a browser and an editor. The browser is called
up when you use the DESCRIBE and DOCUMENTATION functions, and the editor
when you call ED.

custom:*ed*
custom:*browser*
custom:*browsers*
custom:*clhs-root-default*
custom:*with-html-output-doctype*

You may get GNU emacs 24.4 from
https://ftp.gnu.org/gnu/emac...
and then you would set the custom:*editor* to emacsclient to send
editing tasks to emacs, but for this you will have to start the server
in emacs (eg. putting (server-start) in ~/.emacs); you may have to
customize in emacs the server-auth-dir emacs variable.

(setf custom:*editor*
"c:\\gnu\\ntemacs24\\bin\\emacsclient -server-file c:\\temp\\emacs-server")

Ask again about emacs configuration and this when you'll be ready about it.

> So it seems that I'm missing some important point here. Since (as far as I can
> tell) READ-LINE and WRITE-LINE seem to work (even if the characters appear
> garbled internally, they look the same written as read) I'll just have to change
> some of the constants in my programs to avoid them.

Yes, this is the trap when dealing with encoding problems, that
end-to-end it may seem to work, but this is just by chance, when using
1-1 encodings, and indeed, internally you have wrong data.

> Thanks of the attempt to help me, I guess I'm just too dense.

Not at all, it's just that the MS-Windows situation is complex.

--
__Pascal Bourguignon__ http://www.informat...
â??The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.â? -- Carl Bass CEO Autodesk

P. Taine

6/8/2015 1:48:00 AM

On Sun, 07 Jun 2015 22:02:10 +0200, "Pascal J. Bourguignon"
<pjb@informatimago.com> wrote:

>P. Taine <user@domaine.invalid> writes:
>
>>>You can also use the -E option, eg.: clisp -ansi -E utf-8
>>
>> I think I'm going to have to punt on this and change some of my parameters!
>>
>> Note that I have zero UNIX experience and have never used C or any of its
>> descendants. I'm an old (24 years retired) IBM main-frame programmer, mostly in
>> assembler language, so much of the documentation goes over my head. Sorry.
>> For example, I'm not sure what "in you ~/clisprc.lisp" means.
>
>Sorry for the shortcuts I took.
>
>http://www.clisp.org...enc...
>http://www.clisp.org...enc...#path-enc
>http://www.clisp.org...clisp.ht...
>http://www.clisp.org...
>
>
>I'm not much of a MS-Windows user myself. I don't know exactly where
>the clisprc.lisp file is stored. One way to find out would be to evaluate:
>
> (user-homedir-pathname)
>
>Notice: the result may depend on how you launch clisp. Eg. I get
>different user homedir pathnames when I double-click on the clisp icon
>than when I launch it from a *shell* buffer in GNU emacs.
>
> (user-homedir-pathname)
>
>The file that is loaded by clisp, when it exists, on launch (unless you
>run clisp with the -norc option), is:
>
> (make-pathname :name ".clisprc" :type "lisp"
> :defaults (user-homedir-pathname))
>
>also, #P"~/.clisprc.lisp" should work equivalently (but it seems it
>doesn't always do).
>
>
>> My windows short-cut invoking clisp has been:
>> "C:\Program Files\clisp-2.49\clisp.exe" -K full -I -M MyLisp.mem -E ISO-8859-1
>> where "MyLisp.mem" is a saved image (is this the correct term?)
>
>(yes).
>
>> after I have
>> loaded a bunch of macros and function which replicate some features of LISP/VM
>> which I am used to.
>>
>> I am using the United States-International, so Ctrl-Alt-s should give me ß
>> (German sharp-s). When I type that into clisp I get an error "beep", and
>> nothing is echoed (or saved).
>>
>> What I have tried.
>>
>> I modified the invocation, first simply adding the -ansi, than changing
>> ISO-8859-1 to utf-8. Didn't help.
>
>-ansi is unrelated to encodings. I just advise you to always use -ansi,
>since that prevents some deviations from the ANSI Common Lisp standard,
>and therefore let you write more conforming code, that you may more
>easily use with other CL implementations if you would even want to, or
>if you share code with other programmers (eg. if you want to use CL
>libraries available on the Internet).
>
>
>> I ran the five setf expressions which you provided, but this also failed to
>> change the behavior.
>>
>> I then fell further down the rabbit hole, trying to see which code-page the DOS
>> window was using.
>
>Good idea.
>
>
>> My attempt failed as follows
>>
>> [6]> (ext:run-program "C:/windows/system32/chcp.com")
>> Parameter format not correct - /windows
>> 1
>>
>> Trying to set the code page had the same failure:
>>
>> [5]> (ext:run-program "C:/windows/system32/chcp.com" :arguments '("1252"))
>> Parameter format not correct - /windows
>> 1
>
>I don't know much about MS-Windows, so tell me if I'm wrong, but I heard
>that it uses \ as pathname component separator, and / as parameter
>separator. Like:
>
> c:\windows\system32\dir.exe/p
>
>Therefore I would try:
>
> (ext:run-program "C:\\windows\\system32\\chcp.com" :arguments '("1252"))
>
>(Of course, some on MS-Windows programs translate / into \ in pathnames,
>so much confusion ensue. Your only fault is to choose to use MS-Windows
>instead of a sane OS like GNU/Linux (by the way, did you hear that nowadays,
>even IBM uses MacOSX systems? (which are unix-based and include a lot of
>GNU software too)
>http://uk.businessinsider.com/ibm-is-buying-thousands-of-employees-macs-2...
>).
>
>
>
>Now the important point, is that the terminal encoding depends on the
>_terminal_ (or terminal emulator) you use to communicate with clisp.
>You should set the custom:*terminal-encoding* variable according to the
>terminal you are currently using, and the current setting of encoding of
>this terminal too, when it is parameterizable. Also, I guess that by
>default clisp is able to detect the encoding configured for the
>terminal, perhaps using the chcp program or some environment variable;
>at least on unix systems, it definitely use environment variables when
>present, unless the -E option is given, if you don't set it in the
>.clisprc.lisp file.
>
>Also, you will want to set different default encoding for the data
>files and for the pathnames (and possibly for foreign data and
>miscellaneous other situations).
>
> -Edomain encoding
>
> Specifies the encoding used for the given domain, overriding the
> default which depends on the environment variables LC_ALL, LC_CTYPE,
> LANG. -Edomain can be:
>
> -Efile affecting CUSTOM:*DEFAULT-FILE-ENCODING*
> -Epathname affecting CUSTOM:*PATHNAME-ENCODING*
> -Eterminal affecting CUSTOM:*TERMINAL-ENCODING*
> -Eforeign affecting CUSTOM:*FOREIGN-ENCODING*
> -Emisc affecting CUSTOM:*MISC-ENCODING*
> -E affecting all of the above.
>
>
>
>(there's a little problem with -Epathname, in that it assumes that a
>single encoding is used for all the file systems, but in general this is
>not the case. For example on unix you can mount a MS-DOS file system
>using eg. CP1250 encoding onto a root unix file system using iso-8859-1,
>and then you can mount a MacOSX file system using utf-8 onto a directory
>in the MS-DOS file system, and therefore obtain a pathname such as:
>
> /mnt/unix?file-system/MS-DOS-mount-point/©MS/Mac-MP/photo-été.jpg
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ---------- .............
> iso-8859-1 CP1250 utf-8
>
>where the different components are encoded with different encodings.
>Unfortunately, there's no known system that deals with this situation
>properly (hence the advice to restrict yourself to ascii in pathnames)).
>
>
>
>Anyways, the first thing to do is to determine or decide what encoding
>you will use.
>
>- determine what encoding is used by your terminal,
>- determine what encoding is used by your file system for pathnames,
>- decide what encoding you will use for your files (by default, this can
> be changed file-by-file when you open them).
>
>Then you can pass the right -E options when you launch clisp:
>
> "C:\Program Files\clisp-2.49\clisp.exe" -K full -I -M MyLisp.mem > -E ISO-8859-1 -Efile UTF-8 -Epathname WINDOWS-1250 -Eterminal CP850
>
>
>
>Then, you can override those encoding settings in ~/.clisprc.lisp, or
>at least those that are not dependent on the terminal. You can edit the
>.clisprc.lisp file by evaluating:
>
> (ed (make-pathname :name ".clisprc" :type "lisp"
> :defaults (user-homedir-pathname)))
>
>by default this opens a NotePad editor window on that file.
>You can put something like:
>
> (setf custom:*pathname-encoding* (ext:make-encoding :charset charset:windows-1250
> :line-terminator :dos))
> (setf custom:*default-file-encoding* (ext:make-encoding :charset charset:utf-8
> :line-terminator :dos))
> (setf custom:*misc-encoding* (ext:make-encoding :charset charset:iso-8859-1
> :line-terminator :dos))
> #+ffi (setf custom:*foreign-encoding* (ext:make-encoding :charset charset:iso-8859-1
> :line-terminator :dos))
>
>and possibly also set *terminal-encoding*, if you almost always use
>clisp from the same terminal:
>
> (setf custom:*terminal-encoding* (ext:make-encoding :charset charset:cp850
> :line-terminator :dos))
>
>of course, use the right charset, as previously determined or decided.
>
>
>
>
>At this point, I should also ask about your lisp image. Who did build
>it and how is it designed? It is possible in a lisp image, to set an
>init-function that is called before entering the REPL, and in this
>init-function, the lisp image could reset the CUSTOM:*TERMINAL-ENCODING*
>variable. The lisp image could also contain other functions in
>CUSTOM:*INIT-HOOKS*, but the rc file (.clisprc.lisp) is loaded after
>*INIT-HOOKS*, but the init-function is called last.
>
>
>
>
>You may also set other customization variables in this .clisprc.lisp
>file, notably the custom:*ansi* flag, so you don't have to remember to
>type -ansi everytime you launch clisp. Here is my setting:
>
>(let ((counter 0))
> (defun prompt-body ()
> (format nil "~A[~D]"
> (if (packagep *package*)
> (first (sort (cons (package-name *package*) (package-nicknames *package*))
> (function <=) :key (function length)))
> "#<INVALID *PACKAGE*>")
> (incf counter))))
>
>
>(setf
> custom:*ansi* t
> custom:*coerce-fixnum-char-ansi* t
> custom:*floating-point-contagion-ansi* t
> custom:*merge-pathnames-ansi* t
> custom:*parse-namestring-ansi* t
> custom:*parse-namestring-dot-file* :name
> custom:*print-pathnames-ansi* t
> custom:*sequence-count-ansi* t
>
> custom:*default-float-format* 'single-float
> custom:*warn-on-floating-point-contagion* nil
> custom:*suppress-check-redefinition* nil
> custom:*deftype-depth-limit* nil
> custom:*package-tasks-treat-specially* nil
>
> custom:*pprint-first-newline* t
> custom:*print-closure* nil
> custom:*print-indent-lists* 1
> custom:*print-pretty-fill* nil
> custom:*print-rpars* nil
> custom:*prompt-start* "CLISP/"
> custom:*prompt-body* (function prompt-body)
> custom:*prompt-finish* "> "
>
> custom:*load-compiling* nil
> custom:*load-paths* '(#p"")
> custom:*load-obsolete-action* nil
> custom:*compile-warnings* t
>
> custom:*source-file-types* '("lisp" "lsp" "cl")
> custom:*compiled-file-types* '("fas")
>
> custom:*applyhook* nil
> custom:*evalhook* nil
> custom:*trace-indent* t
> custom:*apropos-do-more* nil
> custom:*apropos-matcher* nil
>
> custom:*error-handler* nil
> custom:*break-on-warnings* nil
>
> custom:*inspect-browser* nil
> custom:*inspect-frontend* :tty
> custom:*inspect-length* 100000
> custom:*inspect-print-length* 100000
> custom:*inspect-print-level* 5
> custom:*inspect-print-lines* 5
>
> custom:*user-mail-address* "put@your.email.address.here"
> custom:*current-language* 'i18n:english)
>
>
>You may also configure a browser and an editor. The browser is called
>up when you use the DESCRIBE and DOCUMENTATION functions, and the editor
>when you call ED.
>
> custom:*ed*
> custom:*browser*
> custom:*browsers*
> custom:*clhs-root-default*
> custom:*with-html-output-doctype*
>
>
>You may get GNU emacs 24.4 from
>https://ftp.gnu.org/gnu/emac...
>and then you would set the custom:*editor* to emacsclient to send
>editing tasks to emacs, but for this you will have to start the server
>in emacs (eg. putting (server-start) in ~/.emacs); you may have to
>customize in emacs the server-auth-dir emacs variable.
>
> (setf custom:*editor*
> "c:\\gnu\\ntemacs24\\bin\\emacsclient -server-file c:\\temp\\emacs-server")
>
>Ask again about emacs configuration and this when you'll be ready about it.
>
>
>> So it seems that I'm missing some important point here. Since (as far as I can
>> tell) READ-LINE and WRITE-LINE seem to work (even if the characters appear
>> garbled internally, they look the same written as read) I'll just have to change
>> some of the constants in my programs to avoid them.
>
>Yes, this is the trap when dealing with encoding problems, that
>end-to-end it may seem to work, but this is just by chance, when using
>1-1 encodings, and indeed, internally you have wrong data.
>
>
>> Thanks of the attempt to help me, I guess I'm just too dense.
>
>Not at all, it's just that the MS-Windows situation is complex.

Wow -- I've got a fair bit of reading ahead of me. Note that I'm not happy with
Windows, I "grew up" on CP/CMS and VM/370. Before that on machines where you
walk up and put the punched cards into the reader and debugged on the control
panel -- (first computer, IBM 650, 2000 signed decimal numbers of storage on a
drum, and all vacuum tubes).

Never used UNIX or any of its clones. Don't get me started on paper-tape style
input/output!

As to the "MyLisp" image. I don't have any system building system. I compiled
my various macro and functions. Then I restarted the base system, exported all
the names from :utilities package, loaded the compiled stuff and saved the
image.

I think that is all I did, though I find traces of trying defsys, but not for
the exact set of macros and functions I have in MyLisp.

I thank you again for you patience, and I will bury my self in the references
you have so kindly provided.

Yours,

P. Taine

Pascal J. Bourguignon

6/8/2015 2:43:00 AM

P. Taine <user@domaine.invalid> writes:

> As to the "MyLisp" image. I don't have any system building system. I compiled
> my various macro and functions. Then I restarted the base system, exported all
> the names from :utilities package, loaded the compiled stuff and saved the
> image.

Good, so you if you didn't pass to saveinitmem a :init-function
parameter with a function resetting the encodings, you should be good
setting them in ~/.clisprc.lisp

> I think that is all I did, though I find traces of trying defsys, but not for
> the exact set of macros and functions I have in MyLisp.

If defsys is an utility to load and compile multiple-source file
systems, then nowadays we would rather use ASDF for this purpose, since
it is also well integrated with quicklisp (http://www.qui...),
which allows you to easily download, install and load asdf systems
(libraries).
cf. http://www.cliki.net/Getti...
http://www.mohiji.org/2011/01/31/modern-common-lisp-o...

Also, given the speed of nowaday computers, loading asdf systems upon
demand, even everytime your launch clisp, is fast enough that it's often
not necessary to save an image. Notably, for your own sources, since
you may modify them more often than the libraries you got with
quicklisp, you would want to reload them with asdf (or let quicklisp do
it), rather than saved in an image (since you would have to remember
saving the image again after having modified the external source files
and reloaded them).

That said, I find image-based development intriguing and interesting,
but we would need more tools and utilities for it to be safe and nice.
Once upon a time, I wrote a few tools to help:
http://www.informat...develop/lisp/com/informatimago/small-cl-...
but a lot more would be needed (notably, and in-image editor, instead of
using an external one working on external files;
perhaps something like
http://www.informat...develop/lisp/com/informatimago/small-cl-p...
).

One good example would be the Interlisp Programming Environment
http://www.ics.uci.edu/~andre/ics228s2006/teitelmanma...

--
__Pascal Bourguignon__ http://www.informat...
â??The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.â? -- Carl Bass CEO Autodesk

P. Taine

6/8/2015 2:48:00 PM

On Mon, 08 Jun 2015 04:42:40 +0200, "Pascal J. Bourguignon"
<pjb@informatimago.com> wrote:

>P. Taine <user@domaine.invalid> writes:
>
>> As to the "MyLisp" image. I don't have any system building system. I compiled
>> my various macro and functions. Then I restarted the base system, exported all
>> the names from :utilities package, loaded the compiled stuff and saved the
>> image.
>
>Good, so you if you didn't pass to saveinitmem a :init-function
>parameter with a function resetting the encodings, you should be good
>setting them in ~/.clisprc.lisp
>
>
>> I think that is all I did, though I find traces of trying defsys, but not for
>> the exact set of macros and functions I have in MyLisp.
>
>If defsys is an utility to load and compile multiple-source file
>systems, then nowadays we would rather use ASDF for this purpose, since
>it is also well integrated with quicklisp (http://www.qui...),
>which allows you to easily download, install and load asdf systems
>(libraries).
>cf. http://www.cliki.net/Getti...
>http://www.mohiji.org/2011/01/31/modern-common-lisp-o...
>
>Also, given the speed of nowaday computers, loading asdf systems upon
>demand, even everytime your launch clisp, is fast enough that it's often
>not necessary to save an image. Notably, for your own sources, since
>you may modify them more often than the libraries you got with
>quicklisp, you would want to reload them with asdf (or let quicklisp do
>it), rather than saved in an image (since you would have to remember
>saving the image again after having modified the external source files
>and reloaded them).
>
>That said, I find image-based development intriguing and interesting,
>but we would need more tools and utilities for it to be safe and nice.
>Once upon a time, I wrote a few tools to help:
>http://www.informatimago.com/develop/lisp/com/informatimago/small-cl-...
>but a lot more would be needed (notably, and in-image editor, instead of
>using an external one working on external files;
>perhaps something like
>http://www.informatimago.com/develop/lisp/com/informatimago/small-cl-p...
>).
>
>One good example would be the Interlisp Programming Environment
>http://www.ics.uci.edu/~andre/ics228s2006/teitelmanma...

I will look at Quicklisp and ASDF and hope not to drown in recent jargon.

The last major project I worked on before retiring (in 1991) was LISP-VM,
running on the IBM 370 time-sharing machines. I wrote a lot of the assembler
language parts, designed and implemented an intermediate language, very much
like assembler, between LISP and binary, and took over the compiler. A lot of
the basic functions were written directly in "LISP-assembler". For example, it
was a two data space system with stop-and-copy garbage collection, so half the
memory was un-used at any one time. All functions were well defined on
structures with cycles and/or multiple pointers to the same object. I "cheated"
on the copy and comparison functions by building suspending garbage collection
and creating shadow images in the supposedly unused area. This made it easy to
keep track of objects that had multiple references. We had two flavors of
EQUAL: 1) objects for which any access function applied to the two objects
yielded values which were also EQUAL in this sense (see below), and 2) UEQUAL
where the same update operation (rplaca, setelt, etc.) for the same value to
each object left the two objects UEQUAL.

We had a editor and library system within LISP-VM which maintained collections
of programs and provided a very nice (we thought) editor. Unfortunately I
didn't work on that part and porting it to different terminal hardware on a
system that I didn't have much experience with (PCs and Windows in this case)
seemed like too big a job. I looked at e-macs but somehow it has never appealed
to me (to quite Guy Steele, I like to feel the bits between my toes). So I just
use a very simple text editor (EditPad, which has no support, at least for the
"classic" version). I have to count my own parentheses marks!

Oh well, I guess that's what retirement is for.

P. Taine

Pascal J. Bourguignon

6/8/2015 7:48:00 PM

P. Taine <user@domaine.invalid> writes:

> The last major project I worked on before retiring (in 1991) was LISP-VM,
> running on the IBM 370 time-sharing machines.

If you could recover the sources of LISP-VM, perhaps you could package
them and compile them on Hercules? http://www.hercule...

--
__Pascal Bourguignon__ http://www.informat...
â??The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.â? -- Carl Bass CEO Autodesk

P. Taine

6/9/2015 2:56:00 PM

On Mon, 08 Jun 2015 21:47:44 +0200, "Pascal J. Bourguignon"
<pjb@informatimago.com> wrote:

>P. Taine <user@domaine.invalid> writes:
>
>> The last major project I worked on before retiring (in 1991) was LISP-VM,
>> running on the IBM 370 time-sharing machines.
>
>If you could recover the sources of LISP-VM, perhaps you could package
>them and compile them on Hercules? http://www.hercule...

Thank you for the pointer.

Interesting. I see one of the "recommendations" from an old friend, Michel Hack
(what a great name for a super-programmer!). I would still have to face
converting code which supported IBM 3270 to work on a PC! Probably beyond my
current capabilities. Also, everything expects SVC to VM operation system.

Since I'm not doing any heavy duty coding, I'll probably just totter along as I
have been.

P. Taine

comp.lang.lisp

Reading extended ANSI Characters with Clisp in Windows

P. Taine

Pascal J. Bourguignon

P. Taine

Pascal J. Bourguignon

P. Taine

Pascal J. Bourguignon

P. Taine

Pascal J. Bourguignon

P. Taine

Pascal J. Bourguignon

P. Taine

x Login to ForumsZone