[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

converting UTF-8 to entities like 剛

Jian Lin

5/9/2009 12:05:00 PM


I was trying to convert UTF-8 content into a series of entities like
剛 so that whatever the page encoding is, the characters would
show...

so I used something like this:
<%
begin
t = ''
s = Iconv.conv("UTF-32", "UTF-8", some_utf8_string)

s.scan(/(.)(.)(.)(.)/) do |b1, b2, b3, b4|
t += ("&#x" + "%02X" % b3.ord) + ("%02X" % b4.ord) + ";"
end
rescue => details
t = "exception " + details
end
%>

<%= t %>

but some characters get converted, and some don't. Is it true that
(.)(.)(.)(.) will not necessarily match 4 bytes at a time?

At first, I was going to use

s = Iconv.conv("UTF-16", "UTF-8", some_utf8_string)

but then i found that utf-16 is also variable length... so I used UTF-32
instead which is fixed length. The UTF-8 string I have is just the
Basic Plane... so should be all in the 0x0000 to 0xFFFF range in
unicode.
--
Posted via http://www.ruby-....
17 Answers

Robert Dober

5/9/2009 12:14:00 PM

0

On Sat, May 9, 2009 at 2:04 PM, Jian Lin <winterheat@gmail.com> wrote:
sorry for a quite superficial answer, but can you use the unicode
switch for regexen in your Ruby Version. This seems to be the problem.

Robert


--=20
Si tu veux construire un bateau ...
Ne rassemble pas des hommes pour aller chercher du bois, pr=E9parer des
outils, r=E9partir les t=E2ches, all=E9ger le travail=85 mais enseigne aux
gens la nostalgie de l=92infini de la mer.

If you want to build a ship, don=92t herd people together to collect
wood and don=92t assign them tasks and work, but rather teach them to
long for the endless immensity of the sea.

--
Antoine de Saint-Exup=E9ry

Jian Lin

5/9/2009 12:28:00 PM

0

Robert Dober wrote:
> On Sat, May 9, 2009 at 2:04 PM, Jian Lin <winterheat@gmail.com> wrote:
> sorry for a quite superficial answer, but can you use the unicode
> switch for regexen in your Ruby Version. This seems to be the problem.
>
> Robert


it really might be the 0 that is choking the regular expression match...
if i use

s.scan(/(.)(.)(.)(.)/s)

then it works better but still not all characters are converted...

but the way i have a solution using the byte processing ... in next post

--
Posted via http://www.ruby-....

Jian Lin

5/9/2009 12:29:00 PM

0

this works:

but i am sure there are more elegant solutions.

<%
begin
t = ''
s = Iconv.conv("UTF-32", "UTF-8", some_utf8_string)

(s.length / 4).times do |i|
b3 = s[i*4 + 2]
b4 = s[i*4 + 3]
t += ("&#x" + "%02X" % b3) + ("%02X" % b4) + ";"
end
rescue => details
t = "exception " + details
end
%>

<%= t %>
--
Posted via http://www.ruby-....

Jian Lin

5/9/2009 12:40:00 PM

0

Robert Dober wrote:
> On Sat, May 9, 2009 at 2:04 PM, Jian Lin <winterheat@gmail.com> wrote:
> sorry for a quite superficial answer, but can you use the unicode
> switch for regexen in your Ruby Version. This seems to be the problem.
>
> Robert


by the way... Robert... what is the regexen? is it the regular
expression modifier? I'd like it to match absolutely anything
(newline, 0, etc)... but seems like there is no match
--
Posted via http://www.ruby-....

Rick DeNatale

5/9/2009 1:30:00 PM

0

On Sat, May 9, 2009 at 8:40 AM, Jian Lin <winterheat@gmail.com> wrote:
> Robert Dober wrote:
>> On Sat, May 9, 2009 at 2:04 PM, Jian Lin <winterheat@gmail.com> wrote:
>> sorry for a quite superficial answer, but can you use the unicode
>> switch for regexen in your Ruby Version. This seems to be the problem.
>>
>> Robert
>
>
> by the way... Robert... what is the regexen? =A0is it the regular
> expression modifier? =A0 I'd like it to match absolutely anything
> (newline, 0, etc)... but seems like there is no match

I'm pretty sure that Robert used regexen as the geeky way of pluralizing re=
gex.

The unicode switch (a u regular expression option) forces the use of
unicode to interpret the string being matched, otherwise it uses
whatever the encoding of the source file containing the regular
expression.

e.g. /./u

If you want . to match newlines you want the m (multi-line) option.
Normally . will match anything BUT a new line, m changes this.

rb(main):001:0> "a\nb".match(/a.b/)
=3D> nil
irb(main):002:0> "a\nb".match(/a.b/m)
=3D> #<MatchData:0x6a248>

--=20
Rick DeNatale

Blog: http://talklikeaduck.denh...
Twitter: http://twitter.com/Ri...
WWR: http://www.workingwithrails.com/person/9021-ric...
LinkedIn: http://www.linkedin.com/in/ri...

Jian Lin

5/9/2009 1:42:00 PM

0

Rick Denatale wrote:
> On Sat, May 9, 2009 at 8:40 AM, Jian Lin <winterheat@gmail.com> wrote:
>> (newline, 0, etc)... but seems like there is no match
> I'm pretty sure that Robert used regexen as the geeky way of pluralizing
> regex.
>
> The unicode switch (a u regular expression option) forces the use of
> unicode to interpret the string being matched, otherwise it uses
> whatever the encoding of the source file containing the regular
> expression.
>
> e.g. /./u
>
> If you want . to match newlines you want the m (multi-line) option.
> Normally . will match anything BUT a new line, m changes this.
>
> rb(main):001:0> "a\nb".match(/a.b/)
> => nil
> irb(main):002:0> "a\nb".match(/a.b/m)
> => #<MatchData:0x6a248>

aha... here i just want to match 4 bytes at a time, no matter what the
bytes are. Using "m" won't do it... the "u" would be helpful if i
match one UTF-8 character at a time and then process it... right now i
actually convert it all at once to UTF-32 and then process it... so I
wonder if there is a way to match 4 bytes at a time.

--
Posted via http://www.ruby-....

7stud --

5/9/2009 8:10:00 PM

0

Jian Lin wrote:
> Rick Denatale wrote:
>> On Sat, May 9, 2009 at 8:40 AM, Jian Lin <winterheat@gmail.com> wrote:
>>> (newline, 0, etc)... but seems like there is no match
>> I'm pretty sure that Robert used regexen as the geeky way of pluralizing
>> regex.
>>
>> The unicode switch (a u regular expression option) forces the use of
>> unicode to interpret the string being matched, otherwise it uses
>> whatever the encoding of the source file containing the regular
>> expression.
>>
>> e.g. /./u
>>
>> If you want . to match newlines you want the m (multi-line) option.
>> Normally . will match anything BUT a new line, m changes this.
>>
>> rb(main):001:0> "a\nb".match(/a.b/)
>> => nil
>> irb(main):002:0> "a\nb".match(/a.b/m)
>> => #<MatchData:0x6a248>
>
> aha... here i just want to match 4 bytes at a time, no matter what the
> bytes are. Using "m" won't do it... the "u" would be helpful if i
> match one UTF-8 character at a time and then process it... right now i
> actually convert it all at once to UTF-32 and then process it... so I
> wonder if there is a way to match 4 bytes at a time.

So what's the problem? A dot matches any byte (with the 'm' switch).
Make a regex with four dots:

/..../

or

/.{4}/
--
Posted via http://www.ruby-....

7stud --

5/9/2009 8:16:00 PM

0

7stud -- wrote:
Whoops. With the 'm' switch:

/..../m

or

/.{4}/m

--
Posted via http://www.ruby-....

Jian Lin

5/9/2009 8:36:00 PM

0

7stud -- wrote:
> 7stud -- wrote:
> Whoops. With the 'm' switch:
>
> /..../m
>
> or
>
> /.{4}/m


the problem is that some characters are converted to the correct
&#x9ABC; etc, but some characters are not... you can try if you want...
just go to Google News and get a China, taiwan, or hk news headline.

--
Posted via http://www.ruby-....

7stud --

5/9/2009 11:30:00 PM

0

Jian Lin wrote:
> 7stud -- wrote:
>> 7stud -- wrote:
>> Whoops. With the 'm' switch:
>>
>> /..../m
>>
>> or
>>
>> /.{4}/m
>
>
> the problem is that some characters are converted to the correct
> &#x9ABC; etc, but some characters are not... you can try if you want...
> just go to Google News and get a China, taiwan, or hk news headline.

Then why do you insist that you are trying to match any 4 bytes?
--
Posted via http://www.ruby-....