Asp Forum - Elementary string-parsing

Odysseus

2/4/2008 3:21:00 AM

I'm writing my first 'real' program, i.e. that has a purpose aside from
serving as a learning exercise. I'm posting to solicit comments about my
efforts at translating strings from an external source into useful data,
regarding efficiency and 'pythonicity' both. My only significant
programming experience is in PostScript, and I feel that I haven't yet
'found my feet' concerning the object-oriented aspects of Python, so I'd
be especially interested to know where I may be neglecting to take
advantage of them.

My input is in the form of correlated lists of strings, which I want to
merge (while ignoring some extraneous items). I populate a dictionary
called "found" with these data, still in string form. It contains
sub-dictionaries of various items keyed to strings extracted from the
list "names"; these sub-dictionaries in turn contain the associated
items I want from "cells". After loading in the strings (I have omitted
the statements that pick up strings that require no further processing,
some of them coming from a third list), I convert selected items in
place. Here's the function I wrote:

def extract_data():
i = 0
while i < len(names):
name = names[i][6:] # strip off "Name: "
found[name] = {'epoch1': cells[10 * i + na],
'epoch2': cells[10 * i + na + 1],
'time': cells[10 * i + na + 5],
'score1': cells[10 * i + na + 6],
'score2': cells[10 * i + na + 7]}
###
Following is my first parsing step, for those data that represent real
numbers. The two obstacles I'm contending with here are that the figures
have commas grouping the digits in threes, and that sometimes the data
are non-numeric -- I'll deal with those later. Is there a more elegant
way of removing the commas than the split-and-rejoin below?
###
for k in ('time', 'score1', 'score2'):
v = found[name][k]
if v != "---" and v != "n/a": # skip non-numeric data
v = ''.join(v.split(",")) # remove commas between 000s
found[name][k] = float(v)
###
The next one is much messier. A couple of the strings represent times,
which I think will be most useful in 'native' form, but the input is in
the format "DD Mth YYYY HH:MM:SS UTC". Near the beginning of my program
I have "from calendar import timegm". Before I can feed the data to this
function, though, I have to convert the month abbreviation to a number.
I couldn't come up with anything more elegant than look-up from a list:
the relevant part of my initialization is
'''
m_abbrevs = ("Jan", "Feb", "Mar", "Apr", "May", "Jun",
"Jul", "Aug", "Sep", "Oct", "Nov", "Dec")
'''
I'm also rather unhappy with the way I kluged the seventh and eighth
values in the tuple passed to timegm, the order of the date in the week
and in the year respectively. (I would hate to have to calculate them.)
The function doesn't seem to care what values I give it for these -- as
long as I don't omit them -- so I guess they're only there for the sake
of matching the output of the inverse function. Is there a version of
timegm that takes a tuple of only six (or seven) elements, or any better
way to handle this situation?
###
for k in ('epoch1', 'epoch2'):
dlist = found[name][k].split(" ")
m = 0
while m < 12:
if m_abbrevs[m] == dlist[1]:
dlist[1] = m + 1
break
m += 1
tlist = dlist[3].split(":")
found[name][k] = timegm((int(dlist[2]), int(dlist[1]),
int(dlist[0]), int(tlist[0]),
int(tlist[1]), int(tlist[2]),
-1, -1, 0))
i += 1

The function appears to be working OK as is, but I would welcome any &
all suggestions for improving it or making it more idiomatic.

--
Odysseus

16 Answers

Dennis Lee Bieber

2/4/2008 4:51:00 AM

On Mon, 04 Feb 2008 03:21:18 GMT, Odysseus
<odysseus1479-at@yahoo-dot.ca> declaimed the following in
comp.lang.python:

> My input is in the form of correlated lists of strings, which I want to
> merge (while ignoring some extraneous items). I populate a dictionary
> called "found" with these data, still in string form. It contains
> sub-dictionaries of various items keyed to strings extracted from the
> list "names"; these sub-dictionaries in turn contain the associated
> items I want from "cells". After loading in the strings (I have omitted
> the statements that pick up strings that require no further processing,
> some of them coming from a third list), I convert selected items in
> place. Here's the function I wrote:
>
Rather complicated description... A sample of the real/actual input
/file/ would be useful.

> def extract_data():
> i = 0
> while i < len(names):

for i in range(len(names)):
> name = names[i][6:] # strip off "Name: "

cellRoot = 10 * i + na #where did na come from?
#heck, where do names and cells
#come from? Globals? Not recommended..

use

def extract_data(names, na, cells):

and

return <something>

> found[name] = {'epoch1': cells[10 * i + na],
{ "epoch1" : cells[cellRoot],
> 'epoch2': cells[10 * i + na + 1],
"epoch2" : cells[cellRoot + 1],
> 'time': cells[10 * i + na + 5],
> 'score1': cells[10 * i + na + 6],
> 'score2': cells[10 * i + na + 7]}
#same modification on rest -- avoid recomputing the 10*i+na

> ###
> Following is my first parsing step, for those data that represent real
> numbers. The two obstacles I'm contending with here are that the figures
> have commas grouping the digits in threes, and that sometimes the data
> are non-numeric -- I'll deal with those later. Is there a more elegant
> way of removing the commas than the split-and-rejoin below?
> ###
> for k in ('time', 'score1', 'score2'):
> v = found[name][k]
> if v != "---" and v != "n/a": # skip non-numeric data
> v = ''.join(v.split(",")) # remove commas between 000s
> found[name][k] = float(v)
> ###

I'd suggest splitting this into a short function, and invoking it in
the preceding... say it is called "parsed"

"time" : parsed(cells[cellRoot + 5]),

> The next one is much messier. A couple of the strings represent times,
> which I think will be most useful in 'native' form, but the input is in
> the format "DD Mth YYYY HH:MM:SS UTC". Near the beginning of my program
> I have "from calendar import timegm". Before I can feed the data to this
> function, though, I have to convert the month abbreviation to a number.
> I couldn't come up with anything more elegant than look-up from a list:
> the relevant part of my initialization is

Did you check the library for time/date parsing/formatting
operations?

>>> import time
>>> aTime = "03 Feb 2008 20:35:46 UTC" #DD Mth YYYY HH:MM:SS UTC
>>> time.strptime(aTime, "%d %b %Y %H:%M:%S %Z")
(2008, 2, 3, 20, 35, 46, 6, 34, 0)
>>>

Note that the %Z is a problematic entry... Per the help file
"""
Support for the %Z directive is based on the values contained in tzname
and whether daylight is true. Because of this, it is platform-specific
except for recognizing UTC and GMT which are always known (and are
considered to be non-daylight savings timezones).
"""

>>> aTime = "03 Feb 2008 20:35:46 PST" #DD Mth YYYY HH:MM:SS
>>> time.strptime(aTime, "%d %b %Y %H:%M:%S %Z")
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
File "E:\Python24\lib\_strptime.py", line 293, in strptime
raise ValueError("time data did not match format: data=%s fmt=%s"
%
ValueError: time data did not match format: data=03 Feb 2008 20:35:46
PST fmt=%d %b %Y %H:%M:%S %Z
>>>

--
Wulfraed Dennis Lee Bieber KD6MOG
wlfraed@ix.netcom.com wulfraed@bestiaria.com
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: web-asst@bestiaria.com)
HTTP://www.bestiaria.com/

Marc 'BlackJack' Rintsch

2/4/2008 7:56:00 AM

On Mon, 04 Feb 2008 03:21:18 +0000, Odysseus wrote:

> def extract_data():
> i = 0
> while i < len(names):
> name = names[i][6:] # strip off "Name: "
> found[name] = {'epoch1': cells[10 * i + na],
> 'epoch2': cells[10 * i + na + 1],
> 'time': cells[10 * i + na + 5],
> 'score1': cells[10 * i + na + 6],
> 'score2': cells[10 * i + na + 7]}

Here and in later code you use a ``while`` loop although it is known at
loop start how many times the loop body will be executed. That's a job
for a ``for`` loop. If possible not over an integer that is used later
just as index into list, but the list itself. Here you need both, index
and objects from `names`. There's the `enumerate()` function for creating
an iterable of (index, name) from `names`.

I'd put all the relevant information that describes a field of the
dictionary that is put into `found` into tuples and loop over it. There
is the cell name, the index of the cell and function that converts the
string from that cell into an object that is stored in the dictionary.
This leads to (untestet):

def extract_data(names, na, cells):
found = dict()
for i, name in enumerate(names):
data = dict()
cells_index = 10 * i + na
for cell_name, index, parse in (('epoch1', 0, parse_date),
('epoch2', 1, parse_date),
('time', 5, parse_number),
('score1', 6, parse_number),
('score2', 7, parse_number)):
data[cell_name] = parse(cells[cells_index + index])
assert name.startswith('Name: ')
found[name[6:]] = data
return found

The `parse_number()` function could look like this:

def parse_number(string):
try:
return float(string.replace(',', ''))
except ValueError:
return string

Indeed the commas can be replaced a bit more elegant. :-)

`parse_date()` is left as an exercise for the reader.

> for k in ('epoch1', 'epoch2'):
> dlist = found[name][k].split(" ")
> m = 0
> while m < 12:
> if m_abbrevs[m] == dlist[1]:
> dlist[1] = m + 1
> break
> m += 1
> tlist = dlist[3].split(":")
> found[name][k] = timegm((int(dlist[2]), int(dlist[1]),
> int(dlist[0]), int(tlist[0]),
> int(tlist[1]), int(tlist[2]),
> -1, -1, 0))
> i += 1
>
> The function appears to be working OK as is, but I would welcome any &
> all suggestions for improving it or making it more idiomatic.

As already said, that ``while`` loop should be a ``for`` loop. But if you
put `m_abbrevs` into a `list` you can replace the loop with a single call
to its `index()` method: ``dlist[1] = m_abbrevs.index(dlist[1]) + 1``.

Ciao,
Marc 'BlackJack' Rintsch

Odysseus

2/4/2008 9:43:00 AM

In article <60nunqF1ro06iU4@mid.uni-berlin.de>,
Marc 'BlackJack' Rintsch <bj_666@gmx.net> wrote:

<snip>

> Here and in later code you use a ``while`` loop although it is known at
> loop start how many times the loop body will be executed. That's a job
> for a ``for`` loop. If possible not over an integer that is used later
> just as index into list, but the list itself. Here you need both, index
> and objects from `names`. There's the `enumerate()` function for creating
> an iterable of (index, name) from `names`.

Thanks, that will be very useful. I was casting about for a replacement
for PostScript's "for" loop, and the "while" loop (which PS lacks -- and
which I've never missed there) was all I could come up with.

> I'd put all the relevant information that describes a field of the
> dictionary that is put into `found` into tuples and loop over it. There
> is the cell name, the index of the cell and function that converts the
> string from that cell into an object that is stored in the dictionary.
> This leads to (untestet):
>
> def extract_data(names, na, cells):
> found = dict()

The problem with initializing the 'super-dictionary' within this
function is that I want to be able to add to it in further passes, with
a new set of "names" & "cells" each time.

BTW what's the difference between the above and "found = {}"?

> for i, name in enumerate(names):
> data = dict()
> cells_index = 10 * i + na
> for cell_name, index, parse in (('epoch1', 0, parse_date),
> ('epoch2', 1, parse_date),
> ('time', 5, parse_number),
> ('score1', 6, parse_number),
> ('score2', 7, parse_number)):
> data[cell_name] = parse(cells[cells_index + index])

This looks a lot more efficient than my version, but what about the
strings that don't need parsing? Would it be better to define a
'pass-through' function that just returns its input, so they can be
handled by the same loop, or to handle them separately with another loop?

> assert name.startswith('Name: ')

I looked up "assert", but all I could find relates to debugging. Not
that I think debugging is something I can do without ;) but I don't
understand what this line does.

> found[name[6:]] = data
> return found
>
> The `parse_number()` function could look like this:
>
> def parse_number(string):
> try:
> return float(string.replace(',', ''))
> except ValueError:
> return string
>
> Indeed the commas can be replaced a bit more elegant. :-)

Nice, but I'm somewhat intimidated by the whole concept of
exception-handling (among others). How do you know to expect a
"ValueError" if the string isn't a representation of a number? Is there
a list of common exceptions somewhere? (Searching for "ValueError"
turned up hundreds of passing mentions, but I couldn't find a definition
or explanation.)

<snip>
>
> As already said, that ``while`` loop should be a ``for`` loop. But if you
> put `m_abbrevs` into a `list` you can replace the loop with a single call
> to its `index()` method: ``dlist[1] = m_abbrevs.index(dlist[1]) + 1``.

I had gathered that lists shouldn't be used for storing constants. Is
that more of a suggestion than a rule? I take it tuples don't have an
"index()" method.

Thanks for the detailed advice. I'll post back if I have any trouble
implementing your suggestions.

--
Odysseus

Paul Hankin

2/4/2008 9:57:00 AM

On Feb 4, 3:21 am, Odysseus <odysseus1479...@yahoo-dot.ca> wrote:
> The next one is much messier. A couple of the strings represent times,
> which I think will be most useful in 'native' form, but the input is in
> the format "DD Mth YYYY HH:MM:SS UTC".

time.strptime will do this!

You can find the documentation at http://docs.python.org/lib/module...

Untested:
time.strptime(my_date, '%d %b %y %H:%M:%S %Z')

--
Paul Hankin

John Machin

2/4/2008 11:05:00 AM

On Feb 4, 8:43 pm, Odysseus <odysseus1479...@yahoo-dot.ca> wrote:
> In article <60nunqF1ro06...@mid.uni-berlin.de>,
> Marc 'BlackJack' Rintsch <bj_...@gmx.net> wrote:
> > found = dict()
> BTW what's the difference between the above and "found = {}"?

{} takes 4 fewer keystrokes, doesn't have the overhead of a function
call, and works with Pythons at least as far back as 1.5.2 -- apart
from that, it's got absolutely nothing going for it ;-)

Odysseus

2/4/2008 12:25:00 PM

In article <13qd6ec9vv1qv9a@corp.supernews.com>,
Dennis Lee Bieber <wlfraed@ix.netcom.com> wrote:

<snip>

> Rather complicated description... A sample of the real/actual input
> /file/ would be useful.

Sorry, I didn't want to go on too long about the background, but I guess
more context would have helped. The data actually come from a web page;
I use a class based on SGMLParser to do the initial collection. The
items in the "names" list were originally "title" attributes of anchor
tags and are obtained with a "start_a" method, while "cells" holds the
contents of the <td> tags, obtained by a "handle_data" method according
to the state of a flag that's set to True by a "start_td" method and to
False by an "end_td". I don't care about anything else on the page, so I
didn't define most of the tag-specific methods available.

<snip>

> cellRoot = 10 * i + na #where did na come from?
> #heck, where do names and cells
> #come from? Globals? Not recommended..

The variable "na" is the number of 'not applicable' items (headings and
whatnot) preceding the data I'm interested in.

I'm not clear on what makes an object global, other than appearing as an
operand of a "global" statement, which I don't use anywhere. But "na" is
assigned its value in the program body, not within any function: does
that make it global? Why is this not recommended? If I wrap the
assignment in a function, making "na" a local variable, how can
"extract_data" then access it?

The lists of data are attributes (?) of my SGMLParser class; in my
misguided attempt to pare irrelevant details from "extract_data" I
obfuscated this aspect. I have a "parse_page(url)" function that returns
an instance of the class, as "captured", and the lists in question are
actually called "captured.names" and "captured.cells". The
"parse_page(url)" function is called in the program body; does that make
its output global as well?

> use
>
> def extract_data(names, na, cells):
>
> and
>
> return <something>

What should it return? A Boolean indicating success or failure? All the
data I want should all have been stored in the "found" dictionary by the
time the function finishes traversing the list of names.

> > for k in ('time', 'score1', 'score2'):
> > v = found[name][k]
> > if v != "---" and v != "n/a": # skip non-numeric data
> > v = ''.join(v.split(",")) # remove commas between 000s
> > found[name][k] = float(v)
>
> I'd suggest splitting this into a short function, and invoking it in
> the preceding... say it is called "parsed"
>
> "time" : parsed(cells[cellRoot + 5]),

Will do. I guess part of my problem is that being unsure of myself I'm
reluctant to attempt too much in a single complex statement, finding it
easier to take small and simple (but inefficient) steps. I'll have to
learn to consolidate things as I go.

> Did you check the library for time/date parsing/formatting
> operations?
>
> >>> import time
> >>> aTime = "03 Feb 2008 20:35:46 UTC" #DD Mth YYYY HH:MM:SS UTC
> >>> time.strptime(aTime, "%d %b %Y %H:%M:%S %Z")
> (2008, 2, 3, 20, 35, 46, 6, 34, 0)

I looked at the documentation for the "time" module, including
"strptime", but I didn't realize the "%b" directive would match the
month abbreviations I'm dealing with. It's described as "Locale's
abbreviated month name"; if someone were to run my program on a French
system e.g., wouldn't it try to find a match among "jan", "fév", ...,
"déc" (or whatever) and fail? Is there a way to declare a "locale" that
will override the user's settings? Are the locale-specific strings
documented anywhere? Can one assume them to be identical in all
English-speaking countries, at least? Now it's pretty unlikely in this
case that such an 'international situation' will arise, but I didn't
want to burn any bridges ...

I was also somewhat put off "strptime" on reading the caveat "Note: This
function relies entirely on the underlying platform's C library for the
date parsing, and some of these libraries are buggy. There's nothing to
be done about this short of a new, portable implementation of
strptime()." If it works, however, it'll be a lot tidier than what I was
doing. I'll make a point of testing it on its own, with a variety of
inputs.

> Note that the %Z is a problematic entry...

> ValueError: time data did not match format: data=03 Feb 2008
> 20:35:46 PST fmt=%d %b %Y %H:%M:%S %Z

All the times are UTC, so fortunately this is a non-issue for my
purposes of the moment. May I assume that leaving the zone out will
cause the time to be treated as UTC?

Thanks for your help, and for bearing with my elementary questions and
my fumbling about.

--
Odysseus

Marc 'BlackJack' Rintsch

2/4/2008 4:30:00 PM

On Mon, 04 Feb 2008 12:25:24 +0000, Odysseus wrote:

> I'm not clear on what makes an object global, other than appearing as an
> operand of a "global" statement, which I don't use anywhere. But "na" is
> assigned its value in the program body, not within any function: does
> that make it global?

Yes. The term "global" usually means "module global" in Python.

> Why is this not recommended?

Because the functions depend on some magic data coming from "nowhere" and
it's much harder to follow the data flow in a program. If you work with
globals you can't be sure what the following will print:

def spam():
global x
x = 42
beep()
print x

`beep()` might change `x` or any function called by `beep()` and so on.

Another issue is testing. If you rely on global names it's harder to test
individual functions. If I want to test your `extract_data()` I first have
to look through the whole function body and search all the global
references and bind those names to values before I can call the function.
This might not be enough, any function called by `extract_data()` might
need some global assignments too. This way you'll get quite soon to a
point where the single parts of a program can't be tested in isolation and
are not reusable for other programs.

In programs without such global names you see quite clearly in the
``def`` line what the function expects as input.

> If I wrap the assignment in a function, making "na" a local variable, how
> can "extract_data" then access it?

Give it as an argument. As a rule of thumb values should enter a function
as arguments and leave it as return values.

It's easy to "enforce" if you have minimal code on the module level. The
usual idiom is:

def main():
# Main program comes here.

if __name__ == '__main__':
main()

Then main is called when the script is called as program, but not called if
you just import the script as module. For example to test functions or to
reuse the code from other scripts.

>> def extract_data(names, na, cells):
>>
>> and
>>
>> return <something>
>
> What should it return? A Boolean indicating success or failure? All the
> data I want should all have been stored in the "found" dictionary by the
> time the function finishes traversing the list of names.

Then create the `found` dictionary in that function and return it at the
end.

Ciao,
Marc 'BlackJack' Rintsch

Marc 'BlackJack' Rintsch

2/4/2008 5:02:00 PM

On Mon, 04 Feb 2008 09:43:04 +0000, Odysseus wrote:

> In article <60nunqF1ro06iU4@mid.uni-berlin.de>,
> Marc 'BlackJack' Rintsch <bj_666@gmx.net> wrote:
>
>> def extract_data(names, na, cells):
>> found = dict()
>
> The problem with initializing the 'super-dictionary' within this
> function is that I want to be able to add to it in further passes, with
> a new set of "names" & "cells" each time.

Then you can either pass in `found` as argument instead of creating it
here, or you collect the passes in the calling code with the `update()`
method of `dict`. Something like this:

found = dict()
for pass in passes:
# ...
found.update(extract_data(names, na, cells))

> BTW what's the difference between the above and "found = {}"?

I find it more "explicit". ``dict`` and ``list`` are easier to
distinguish than ``{}`` and ``[]`` after a loooong coding session or when
printed/displayed in a small font. It's just a matter of taste.

>> for i, name in enumerate(names):
>> data = dict()
>> cells_index = 10 * i + na
>> for cell_name, index, parse in (('epoch1', 0, parse_date),
>> ('epoch2', 1, parse_date),
>> ('time', 5, parse_number),
>> ('score1', 6, parse_number),
>> ('score2', 7, parse_number)):
>> data[cell_name] = parse(cells[cells_index + index])
>
> This looks a lot more efficient than my version, but what about the
> strings that don't need parsing? Would it be better to define a
> 'pass-through' function that just returns its input, so they can be
> handled by the same loop, or to handle them separately with another loop?

I'd handle them in the same loop. A "pass-through" function for strings
already exists:

In [255]: str('hello')
Out[255]: 'hello'

>> assert name.startswith('Name: ')
>
> I looked up "assert", but all I could find relates to debugging. Not
> that I think debugging is something I can do without ;) but I don't
> understand what this line does.

It checks if `name` really starts with 'Name: '. This way I turned the
comment into code that checks the assertion in the comment.

>> The `parse_number()` function could look like this:
>>
>> def parse_number(string):
>> try:
>> return float(string.replace(',', ''))
>> except ValueError:
>> return string
>>
>> Indeed the commas can be replaced a bit more elegant. :-)
>
> Nice, but I'm somewhat intimidated by the whole concept of
> exception-handling (among others). How do you know to expect a
> "ValueError" if the string isn't a representation of a number?

Experience. I just tried what happens if I feed `float()` with a string
that is no number:

In [256]: float('abc')
---------------------------------------------------------------------------
<type 'exceptions.ValueError'> Traceback (most recent call last)

/home/bj/<ipython console> in <module>()

<type 'exceptions.ValueError'>: invalid literal for float(): abc

> Is there a list of common exceptions somewhere? (Searching for
> "ValueError" turned up hundreds of passing mentions, but I couldn't find
> a definition or explanation.)

The definition is quite vague. The type of an argument is correct, but
there's something wrong with the value.

See http://docs.python.org/lib/module-excep... for an overview of
the built in exceptions.

>> As already said, that ``while`` loop should be a ``for`` loop. But if
>> you put `m_abbrevs` into a `list` you can replace the loop with a
>> single call to its `index()` method: ``dlist[1] =
>> m_abbrevs.index(dlist[1]) + 1``.
>
> I had gathered that lists shouldn't be used for storing constants. Is
> that more of a suggestion than a rule?

Some suggest this. Others say tuples are for data where the position of
an element has a "meaning" and lists are for elements that all have the
same "meaning" for some definition of meaning. As an example ('John',
'Doe', 'Dr.') vs. ['Peter', 'Paul', 'Mary']. In the first example we have
name, surname, title and in the second example all elements are just
names. Unless the second example models a relation like child, father,
mother, or something like that. Anyway, if you can make the source simpler
and easier to understand by using the `index()` method, use a list. :-)

Ciao,
Marc 'BlackJack' Rintsch

Dennis Lee Bieber

2/4/2008 7:01:00 PM

On Mon, 04 Feb 2008 09:43:04 GMT, Odysseus
<odysseus1479-at@yahoo-dot.ca> declaimed the following in
comp.lang.python:

>
> Thanks, that will be very useful. I was casting about for a replacement
> for PostScript's "for" loop, and the "while" loop (which PS lacks -- and
> which I've never missed there) was all I could come up with.
>
Have you read the language reference manual yet? It is a rather
short document given that the language syntactic elements are not that
complex -- but would have exposed you to the "for" statement (along with
"return" and passing arguments). If your only other programming
experience is base PostScript you wouldn't really be familiar with
passing arguments or returning values -- as an RPN stack-based language,
argument passing is just listing the arguments before a function call
(putting a copy of them on the stack), and returns are whatever the
function left on the stack at the end; hence they appear sort of global.

After the language reference manual, the library reference manual
chapter on built-ins and data types would be next for study -- the rest
can usually be handled via search functions (working with time
conversions, look for modules with date or time <G>).

>
> The problem with initializing the 'super-dictionary' within this
> function is that I want to be able to add to it in further passes, with
> a new set of "names" & "cells" each time.
>
Well, you could initialize it in the "pass" loop, and pass it in
along with everything else...

Though (since I'm responding halfway in on a dialog, and don't have
the full prior level quoted)...

It looked a bit like you were using a SAX-style parser to collect
"names" and "cells" -- and then passing the "bunch" to another function
to trim out and convert data... It would take me a bit to restudy the
SAX parsing scheme (I did it once, back in the days of v1.5 or so) but
the way I'd /try/ to do it is to have the stream handler keep track of
which cell (<td> tag) is currently being parsed, and convert the string
data at that level. You'd initialize the record dictionary to {} (and
cell position to 0) on the <tr> tag, and return the populated record on
the </tr> tag.

Might want to check into making a class/instance of the parser so
you can make the record dictionary and column (cell) position instance
attributes (avoiding globals).

> Nice, but I'm somewhat intimidated by the whole concept of
> exception-handling (among others). How do you know to expect a
> "ValueError" if the string isn't a representation of a number? Is there

Read the library reference for the function in question? Though it
appears the reference doesn't list the error raised for an invalid
string representation -- in which case just try one in the interactive
shell...

> a list of common exceptions somewhere? (Searching for "ValueError"
> turned up hundreds of passing mentions, but I couldn't find a definition
> or explanation.)

Library reference -- under "Exception" (note the cap). {Section 2.4
Built-in Exceptions in my version of Python -- same chapter I mentioned
above about data types}

> I had gathered that lists shouldn't be used for storing constants. Is
> that more of a suggestion than a rule? I take it tuples don't have an
> "index()" method.
>
Tuples are "read-only" whereas a list can be modified in-place...
But many look on tuples as a data "record" where each position has a
different meaning; lists are collections where each position has a
different value but the same meaning...

Tuple: (name, street, city, state, zip)
List: [name1, name2, name3, name...]

for name in List:
#makes sense as you would do the same process on each name

for field in Tuple:
#does NOT make sense; why would you do the same
#process on street and zip, for example

--
Wulfraed Dennis Lee Bieber KD6MOG
wlfraed@ix.netcom.com wulfraed@bestiaria.com
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: web-asst@bestiaria.com)
HTTP://www.bestiaria.com/

Odysseus

2/5/2008 4:03:00 AM

In article <13qeo77n2p7nv26@corp.supernews.com>,
Dennis Lee Bieber <wlfraed@ix.netcom.com> wrote:

> On Mon, 04 Feb 2008 09:43:04 GMT, Odysseus
> <odysseus1479-at@yahoo-dot.ca> declaimed the following in
> comp.lang.python:
>
> >
> > Thanks, that will be very useful. I was casting about for a replacement
> > for PostScript's "for" loop, and the "while" loop (which PS lacks -- and
> > which I've never missed there) was all I could come up with.
> >
> Have you read the language reference manual yet? It is a rather
> short document given that the language syntactic elements are not that
> complex -- but would have exposed you to the "for" statement (along with
> "return" and passing arguments).

Sorry, translation problem: I am acquainted with Python's "for" -- if
far from fluent with it, so to speak -- but the PS operator that's most
similar (traversing a compound object, element by element, without any
explicit indexing or counting) is called "forall". PS's "for" loop is
similar to BASIC's (and ISTR Fortran's):

start_value increment end_value {procedure} for

I don't know the proper generic term -- "indexed loop"? -- but at any
rate it provides a counter, unlike Python's command of the same name.

> If your only other programming experience is base PostScript you
> wouldn't really be familiar with passing arguments or returning
> values -- as an RPN stack-based language, argument passing is just
> listing the arguments before a function call (putting a copy of them
> on the stack), and returns are whatever the function left on the
> stack at the end; hence they appear sort of global.

Working directly in the operand stack is efficient, but can make
interpretation by humans -- and debugging -- very difficult. So for the
sake of coder-friendliness it's generally advisable to use variables
(i.e. assign values to keys in a dictionary) in most cases instead of
passing values 'silently' via the stack. I'm beginning to realize that
for Python the situation is just about the opposite ...

Anyway, I have been reading the documentation on the website, but much
of the terminology is unfamiliar to me. When looking things up I seem to
get an inordinate number of 404 errors from links returned by the search
function, and often the language-reference or tutorial entries (if any)
are buried several pages down. In general I'm finding the docs rather
frustrating to navigate.

> After the language reference manual, the library reference manual
> chapter on built-ins and data types would be next for study -- the rest
> can usually be handled via search functions (working with time
> conversions, look for modules with date or time <G>).

As I mentioned elsethread, I did look at the "time" documentation; it
was there that I found a reference to the "calendar.timegm" function I
used in my first attempt.

> It looked a bit like you were using a SAX-style parser to collect
> "names" and "cells" -- and then passing the "bunch" to another function
> to trim out and convert data... It would take me a bit to restudy the
> SAX parsing scheme (I did it once, back in the days of v1.5 or so) but
> the way I'd /try/ to do it is to have the stream handler keep track of
> which cell (<td> tag) is currently being parsed, and convert the string
> data at that level. You'd initialize the record dictionary to {} (and
> cell position to 0) on the <tr> tag, and return the populated record on
> the </tr> tag.

This is what my setup looks like -- mostly cribbed from _Dive Into
Python_ -- where "PageParser" is a class based on "SGMLParser":

from sgmllib import SGMLParser
from urllib import urlopen

# ...

def parse_page(url):
usock = urlopen(url)
parser = PageParser()
parser.feed(usock.read())
parser.close()
usock.close()
return parser

# ...

captured = parse_page(base_url + suffix)

I only use "parse_page" the once at this stage, but my plan was to call
it repeatedly while varying "suffix" (depending on the data found by the
previous pass). On each pass the class will initialize itself, which is
why I was collecting the data into a 'standing' (global) dictionary. Are
you suggesting essentially that I'd do better to make the text-parsing
function into a method of "PageParser"? Can one add, to such a derived
class, methods that don't have protoypes in the parent?

> Might want to check into making a class/instance of the parser so
> you can make the record dictionary and column (cell) position instance
> attributes (avoiding globals).

AFAICT my "captured" is an instance of "PageParser", but I'm unclear on
how I would add attributes to it -- and as things stand it will get
rebuilt from scratch each time a page is read in.

> > [...] I'm somewhat intimidated by the whole concept of
> > exception-handling (among others). How do you know to expect a
> > "ValueError" if the string isn't a representation of a number?
>
> Read the library reference for the function in question? Though it
> appears the reference doesn't list the error raised for an invalid
> string representation -- in which case just try one in the interactive
> shell...

Under "2.1 Built-in Functions"

<http://docs.python.org/lib/built-in-func...

"""float([x])
Convert a string or a number to floating point. If the argument is a
string, it must contain a possibly signed decimal or floating point
number, possibly embedded in whitespace. Otherwise, the argument may be
a plain or long integer or a floating point number, and a floating point
number with the same value (within Python's floating point precision) is
returned. If no argument is given, returns 0.0.
Note: When passing in a string, values for NaN and Infinity may be
returned, depending on the underlying C library. The specific set of
strings accepted which cause these values to be returned depends
entirely on the C library and is known to vary.
"""

Not a word about errors. If I understand the note correctly,
"float('---')" might cause an error or might happily return "NaN", so it
appears experimentation is the only way to go. (On my system I get the
error, but if I wanted to run the program elsewhere, or share it, I
suppose the code would have to be tested in each environment.)

> > Is there a list of common exceptions somewhere? (Searching for
> > "ValueError" turned up hundreds of passing mentions, but I couldn't
> > find a definition or explanation.)
>
> Library reference -- under "Exception" (note the cap). {Section 2.4
> Built-in Exceptions in my version of Python -- same chapter I mentioned
> above about data types}

Thanks -- one would think my search should have directed me there ...

> Tuples are "read-only" whereas a list can be modified in-place...
> But many look on tuples as a data "record" where each position has a
> different meaning; lists are collections where each position has a
> different value but the same meaning...
>
> Tuple: (name, street, city, state, zip)

Why wouldn't one use a dictionary for that?

> List: [name1, name2, name3, name...]
>
> for name in List:
> #makes sense as you would do the same process on each name
>
> for field in Tuple:
> #does NOT make sense; why would you do the same
> #process on street and zip, for example

I see the distinction. Thanks again ...

--
Odysseus

comp.lang.python

Elementary string-parsing

Odysseus

Dennis Lee Bieber

Marc 'BlackJack' Rintsch

Odysseus

Paul Hankin

John Machin

Odysseus

Marc 'BlackJack' Rintsch

Marc 'BlackJack' Rintsch

Dennis Lee Bieber

Odysseus

x Login to ForumsZone