[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.python

Data mapper - need to map an dictionary of values to a model

Luke

1/15/2008 12:56:00 AM

I am writing an order management console. I need to create an import
system that is easy to extend. For now, I want to accept an dictionary
of values and map them to my data model. The thing is, I need to do
things to certain columns:

- I need to filter some of the values (data comes in as YYYY-MM-
DDTHH:MM:SS-(TIMEZONE-OFFSET) and it needs to map to Order.date as a
YYYY-MM-DD field)
- I need to map parts of an input column to more than one model param
(for instance if I get a full name for input--like "John Smith"--I
need a function to break it apart and map it to
Order.shipping_first_name and Order.shipping_last_name)
- Sometimes I need to do it the other way too... I need to map
multiple input columns to one model param (If I get a shipping fee, a
shipping tax, and a shipping discount, I need them added together and
mapped to Order.shipping_fee)

I have begun this process, but I'm finding it difficult to come up
with a good system that is extensible and easy to understand. I won't
always be the one writing the importers, so I'd like it to be pretty
straight-forward. Any ideas?

Oh, I should also mention that many times the data will map to several
different models. For instance, the importer I'm writing first would
map to 3 different models (Order, OrderItem, and OrderCharge)

I am not looking for anybody to write any code for me. I'm simply
asking for inspiration. What design patterns would you use here? Why?
8 Answers

Bearophile

1/15/2008 9:53:00 AM

0

Luke:
>What design patterns would you use here?<

What about "generator (scanner) with parameters"? :-)

Bye,
bearophile

Luke

1/15/2008 7:11:00 PM

0

On Jan 15, 1:53 am, bearophileH...@lycos.com wrote:
> Luke:
>
> >What design patterns would you use here?<
>
> What about "generator (scanner) with parameters"? :-)
>
> Bye,
> bearophile

I'm not familiar with this pattern. I will search around, but if you
have any links or you would like to elaborate, that would be
wonderful. :)

Bearophile

1/15/2008 7:27:00 PM

0

Luke:
> I'm not familiar with this pattern. I will search around, but if you
> have any links or you would like to elaborate, that would be
> wonderful. :)

It's not a pattern, it's a little thing:

def line_filter(filein, params):
for line in filein:
if good(line, params):
yield extract(line, params)

That equals to this too:

def line_filter(filein, params):
return (extract(line, params) for line in filein if good(line,
params))

But probably that's not enough to solve your problem, so other people
can give you a better answer.

Bye,
bearophile

George Sakkis

1/15/2008 11:54:00 PM

0

On Jan 14, 7:56 pm, Luke <Luke.Visin...@gmail.com> wrote:

> I am writing an order management console. I need to create an import
> system that is easy to extend. For now, I want to accept an dictionary
> of values and map them to my data model. The thing is, I need to do
> things to certain columns:
>
> - I need to filter some of the values (data comes in as YYYY-MM-
> DDTHH:MM:SS-(TIMEZONE-OFFSET) and it needs to map to Order.date as a
> YYYY-MM-DD field)
> - I need to map parts of an input column to more than one model param
> (for instance if I get a full name for input--like "John Smith"--I
> need a function to break it apart and map it to
> Order.shipping_first_name and Order.shipping_last_name)
> - Sometimes I need to do it the other way too... I need to map
> multiple input columns to one model param (If I get a shipping fee, a
> shipping tax, and a shipping discount, I need them added together and
> mapped to Order.shipping_fee)
>
> I have begun this process, but I'm finding it difficult to come up
> with a good system that is extensible and easy to understand. I won't
> always be the one writing the importers, so I'd like it to be pretty
> straight-forward. Any ideas?
>
> Oh, I should also mention that many times the data will map to several
> different models. For instance, the importer I'm writing first would
> map to 3 different models (Order, OrderItem, and OrderCharge)
>
> I am not looking for anybody to write any code for me. I'm simply
> asking for inspiration. What design patterns would you use here? Why?

The specific transformations you describe are simple to be coded
directly but unless you constrain the set of possible transformations
that can take place, I don't see how can this be generalized in any
useful way. It just seems too open-ended.

The only pattern I can see here is breaking down the overall
transformation to independent steps, just like the three you
described. Given some way to specify each separate transformation,
their combination can be factored out. To illustrate, here's a trivial
example (with dicts for both input and output):

class MultiTransformer(object):
def __init__(self, *tranformers):
self._tranformers = tranformers

def __call__(self, input):
output = {}
for t in self._tranformers:
output.update(t(input))
return output

date_tranformer = lambda input: {'date' : input['date'][:10]}
name_tranformer = lambda input: dict(
zip(('first_name', 'last_name'),
input['name']))
fee_tranformer = lambda input: {'fee' : sum([input['fee'],
input['tax'],
input['discount']])}
tranformer = MultiTransformer(date_tranformer,
name_tranformer,
fee_tranformer)
print tranformer(dict(date='2007-12-22 03:18:99-EST',
name='John Smith',
fee=30450.99,
tax=459.15,
discount=985))
# output
#{'date': '2007-12-22', 'fee': 31895.140000000003,
'first_name': #'J', 'last_name': 'o'}


You can see that the MultiTransformer doesn't buy you much by itself;
it just allows dividing the overall task to smaller bits that can be
documented, tested and reused separately. For anything more
sophisticated, you have to constrain what are the possible
transformations that can happen. I did something similar for
transforming CSV input rows (http://pypi.python.org/pypi...) so
that it's easy to specify 1-to-{0,1} transformations but not 1-to-many
or many-to-1.

HTH,
George

George Sakkis

1/16/2008

0

On Jan 15, 6:53 pm, George Sakkis <george.sak...@gmail.com> wrote:

> name_tranformer = lambda input: dict(
> zip(('first_name', 'last_name'),
> input['name']))

Of course that should write:

name_tranformer = lambda input: dict(
zip(('first_name', 'last_name'),
input['name'].split()))

George

Luke

1/16/2008 12:03:00 AM

0

On Jan 15, 3:53 pm, George Sakkis <george.sak...@gmail.com> wrote:
> On Jan 14, 7:56 pm, Luke <Luke.Visin...@gmail.com> wrote:
>
>
>
> > I am writing an order management console. I need to create an import
> > system that is easy to extend. For now, I want to accept an dictionary
> > of values and map them to my data model. The thing is, I need to do
> > things to certain columns:
>
> > - I need to filter some of the values (data comes in as YYYY-MM-
> > DDTHH:MM:SS-(TIMEZONE-OFFSET) and it needs to map to Order.date as a
> > YYYY-MM-DD field)
> > - I need to map parts of an input column to more than one model param
> > (for instance if I get a full name for input--like "John Smith"--I
> > need a function to break it apart and map it to
> > Order.shipping_first_name and Order.shipping_last_name)
> > - Sometimes I need to do it the other way too... I need to map
> > multiple input columns to one model param (If I get a shipping fee, a
> > shipping tax, and a shipping discount, I need them added together and
> > mapped to Order.shipping_fee)
>
> > I have begun this process, but I'm finding it difficult to come up
> > with a good system that is extensible and easy to understand. I won't
> > always be the one writing the importers, so I'd like it to be pretty
> > straight-forward. Any ideas?
>
> > Oh, I should also mention that many times the data will map to several
> > different models. For instance, the importer I'm writing first would
> > map to 3 different models (Order, OrderItem, and OrderCharge)
>
> > I am not looking for anybody to write any code for me. I'm simply
> > asking for inspiration. What design patterns would you use here? Why?
>
> The specific transformations you describe are simple to be coded
> directly but unless you constrain the set of possible transformations
> that can take place, I don't see how can this be generalized in any
> useful way. It just seems too open-ended.
>
> The only pattern I can see here is breaking down the overall
> transformation to independent steps, just like the three you
> described. Given some way to specify each separate transformation,
> their combination can be factored out. To illustrate, here's a trivial
> example (with dicts for both input and output):
>
> class MultiTransformer(object):
> def __init__(self, *tranformers):
> self._tranformers = tranformers
>
> def __call__(self, input):
> output = {}
> for t in self._tranformers:
> output.update(t(input))
> return output
>
> date_tranformer = lambda input: {'date' : input['date'][:10]}
> name_tranformer = lambda input: dict(
> zip(('first_name', 'last_name'),
> input['name']))
> fee_tranformer = lambda input: {'fee' : sum([input['fee'],
> input['tax'],
> input['discount']])}
> tranformer = MultiTransformer(date_tranformer,
> name_tranformer,
> fee_tranformer)
> print tranformer(dict(date='2007-12-22 03:18:99-EST',
> name='John Smith',
> fee=30450.99,
> tax=459.15,
> discount=985))
> # output
> #{'date': '2007-12-22', 'fee': 31895.140000000003,
> 'first_name': #'J', 'last_name': 'o'}
>
> You can see that the MultiTransformer doesn't buy you much by itself;
> it just allows dividing the overall task to smaller bits that can be
> documented, tested and reused separately. For anything more
> sophisticated, you have to constrain what are the possible
> transformations that can happen. I did something similar for
> transforming CSV input rows (http://pypi.python.org/pypi...) so
> that it's easy to specify 1-to-{0,1} transformations but not 1-to-many
> or many-to-1.
>
> HTH,
> George

thank you that is very helpful. I will ponder that for a while :)

FerretBill

3/16/2009 3:21:00 PM

0

On Mon, 16 Mar 2009 11:10:28 -0400, "A to Z"
<REMOVETHESEaddietzCAPS@verizonPLEASE.net> wrote:

>
><donnieb78@hotmail.com> wrote in message
>news:841305a2-f7e6-46d3-849d-46a2d850cef3@q11g2000yqh.googlegroups.com...
>> Allegedly you can make red lights turn to green by duct-taping a
>> magnet to the bottom of your car. Sounds like bullshit to me, but
>> hopefully somebody who's knowledgeable about the science can tell us
>> for sure.
>>
>>
>> http://www.wonderhowto.com/how-to/video/how-to-trigger-traffic-lights-to-change-from-red-to-gr...
>
>so if an eastbound car has a magnet, and a northbound car has one too, who
>wins?
>
The wheelchair company?

Donnieb78

3/16/2009 4:35:00 PM

0

Given some of the other how-to videos on this site -- "Feel High
Without Taking Drugs,"Make People Naked Using Photoshop and "Is She a
Hooker or a Cop?" -- I'm not sure if the whole magnet-and-red light
thing passed a rigorous scientific peer-review process.