[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.python

Unicode characters in btye-strings

Steven D'Aprano

3/12/2010 12:36:00 PM

I know this is wrong, but I'm not sure just how wrong it is, or why.
Using Python 2.x:

>>> s = "éâ�"
>>> print s
éâ�
>>> len(s)
6
>>> list(s)
['\xc3', '\xa9', '\xc3', '\xa2', '\xc3', '\x84']

Can somebody explain what happens when I put non-ASCII characters into a
non-unicode string? My guess is that the result will depend on the
current encoding of my terminal.

In this case, my terminal is set to UTF-8. If I change it to ISO 8859-1,
and repeat the above, I get this:

>>> list("éâ�")
['\xe9', '\xe2', '\xc4']

If I do this:

>>> s = u"éâ�"
>>> s.encode('utf-8')
'\xc3\xa9\xc3\xa2\xc3\x84'
>>> s.encode('iso8859-1')
'\xe9\xe2\xc4'

which at least explains why the bytes have the values which they do.


Thank you,



--
Steven
5 Answers

Robert Kern

3/12/2010 3:38:00 PM

0

On 2010-03-12 06:35 AM, Steven D'Aprano wrote:
> I know this is wrong, but I'm not sure just how wrong it is, or why.
> Using Python 2.x:
>
>>>> s = "éâ�"
>>>> print s
> éâ�
>>>> len(s)
> 6
>>>> list(s)
> ['\xc3', '\xa9', '\xc3', '\xa2', '\xc3', '\x84']
>
> Can somebody explain what happens when I put non-ASCII characters into a
> non-unicode string? My guess is that the result will depend on the
> current encoding of my terminal.

Exactly right.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

Martin v. Loewis

3/12/2010 8:57:00 PM

0

>> Can somebody explain what happens when I put non-ASCII characters into a
>> non-unicode string? My guess is that the result will depend on the
>> current encoding of my terminal.
>
> Exactly right.

To elaborate on the "what happens" part: the string that gets entered is
typically passed as a byte sequence, from the terminal (application) to
the OS kernel, from the OS kernel to Python's stdin, and from there to
the parser. Python recognizes the string delimiters, but (practically)
leaves the bytes between the delimiters as-is (*), creating a byte
string object with the very same bytes.

The more interesting question is what happens when you do

py> s = u"éâ�"

Here, Python needs to decode the bytes, according to some encoding.
Usually, it would want to use the source encoding (as given through
-*- Emacs -*- markers), but there are none. Various Python versions then
try different things; what they should do is to determine the terminal
encoding, and decode the bytes according to that one.

Regards,
Martin

(*) If a source encoding was given, the source is actually recoded to
UTF-8, parsed, and then re-encoded back into the original encoding.

Michael Rudolf

3/12/2010 10:56:00 PM

0

Am 12.03.2010 21:56, schrieb Martin v. Loewis:
> (*) If a source encoding was given, the source is actually recoded to
> UTF-8, parsed, and then re-encoded back into the original encoding.

Why is that? So "unicode"-strings (as in u"string") are not really
unicode-, but utf8-strings?

Need citation plz.

Thx,
Michael

John Bokma

3/12/2010 11:38:00 PM

0

Michael Rudolf <spamfresser@ch3ka.de> writes:

> Am 12.03.2010 21:56, schrieb Martin v. Loewis:
>> (*) If a source encoding was given, the source is actually recoded to
>> UTF-8, parsed, and then re-encoded back into the original encoding.
>
> Why is that? So "unicode"-strings (as in u"string") are not really
> unicode-, but utf8-strings?

utf8 is a Unicode *encoding*.

--
John Bokma j3b

Hacking & Hiking in Mexico - http://john...
http://castle... - Perl & Python Development

Martin v. Loewis

3/12/2010 11:52:00 PM

0

Michael Rudolf wrote:
> Am 12.03.2010 21:56, schrieb Martin v. Loewis:
>> (*) If a source encoding was given, the source is actually recoded to
>> UTF-8, parsed, and then re-encoded back into the original encoding.
>
> Why is that?

Why is what? That string literals get reencoded into the source encoding?

> So "unicode"-strings (as in u"string") are not really
> unicode-, but utf8-strings?

No. String literals, in 2.x, are not written with u"", and are stored in
the source encoding. Above procedure applies to regular strings (see
where the "*" goes in my original article).

> Need citation plz.

You really want a link to the source code implementing that?

Regards,
Martin