[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.python

Re: plugin / intra process communication system

Florian Ludwig

2/14/2010 12:05:00 PM

On Sun, 2010-02-14 at 10:16 +0100, Paul Kölle wrote:
> Am 13.02.2010 10:50, schrieb Florian Ludwig:
> > Hi,
> >
> > I'm looking for a module/plugin/intra-process-communication/hook system
> > for python. Maybe someone here could point me to some project I missed
> > or might have some good ideas if I end up implementing it myself.
> >
> > [...]
> >
> > The plugins you wrote probably import them from the wiki module directly
> > which means your blog would depend on the wiki. (?!) So where to put the
> > interface/[...] definition that is imported by those plugins? Create a
> > package/module just containing the interface? This really get
> > troublesome if different people wrote the wiki, the blog and another
> > third one a plugin.

> If you are talking about code sharing you can move the common code out
> of your applications in a seperate namespace. If you follow the model
> trac is using you would install a module/package/egg with the basic
> functionality of the pluginsystem (i.e. what's in core.py and env.py +
> logging and whatever you think is necessary).

> All shared code like your auth-plugins would go in a common plugin
> directory to which you can refer via PYTHONPATH.

You're right, its about code sharing/reusing - should have said it more
clearly. What I am looking for is the pluginsystem that makes this easy.
Here there problem with the trac (and other plugin systems I've seen)
approach:

You need to define something like:
|
| class IAuthPlugin(Interface): [...]
|
in your blog software.

Now within the open_id_login_plugin:
| from blog import IAuthPlugin
|
| class OpenIdPlugin(object):
| implements(IAuthPlugin)
|

So the OpenIdPlugin is specific for the blog and hardly sharable.
Actually it is but it depends on the blog as it gets imported.

This might be thought of as an implementation detail but its seems
pretty common.


> Another common technique is chaining of WSGI middleware..., check out
> pythonpaste.org.

WSGI middleware just doesn't do it in some situations. Also it only
works out for web-based applications (my example was one but my question
was more general).


Thanks for your answer,
Florian



--
Florian Ludwig <dino@phidev.org>
7 Answers

Diez B. Roggisch

2/14/2010 5:48:00 PM

0

Am 14.02.10 13:05, schrieb Florian Ludwig:
> On Sun, 2010-02-14 at 10:16 +0100, Paul Kölle wrote:
>> Am 13.02.2010 10:50, schrieb Florian Ludwig:
>>> Hi,
>>>
>>> I'm looking for a module/plugin/intra-process-communication/hook system
>>> for python. Maybe someone here could point me to some project I missed
>>> or might have some good ideas if I end up implementing it myself.
>>>
>>> [...]
>>>
>>> The plugins you wrote probably import them from the wiki module directly
>>> which means your blog would depend on the wiki. (?!) So where to put the
>>> interface/[...] definition that is imported by those plugins? Create a
>>> package/module just containing the interface? This really get
>>> troublesome if different people wrote the wiki, the blog and another
>>> third one a plugin.
>
>> If you are talking about code sharing you can move the common code out
>> of your applications in a seperate namespace. If you follow the model
>> trac is using you would install a module/package/egg with the basic
>> functionality of the pluginsystem (i.e. what's in core.py and env.py +
>> logging and whatever you think is necessary).
>
>> All shared code like your auth-plugins would go in a common plugin
>> directory to which you can refer via PYTHONPATH.
>
> You're right, its about code sharing/reusing - should have said it more
> clearly. What I am looking for is the pluginsystem that makes this easy.
> Here there problem with the trac (and other plugin systems I've seen)
> approach:
>
> You need to define something like:
> |
> | class IAuthPlugin(Interface): [...]
> |
> in your blog software.

Why? Any reason you can't define it in a separate package the
blog-software depends on, as well as your wiki?


And then of course, this is not really needed. In Python, behavior
counts, not type-information. So you can get away without any explicit
declared interface. You might chose to not do that, for aestetic
reasons, or better documentation. But you aren't forced.

diez

Florian Ludwig

2/15/2010 10:12:00 PM

0

On Sun, 2010-02-14 at 18:47 +0100, Diez B. Roggisch wrote:
> > Here there problem with the trac (and other plugin systems I've
> seen)
> > approach:
> >
> > You need to define something like:
> > |
> > | class IAuthPlugin(Interface): [...]
> > |
> > in your blog software.
>
> Why? Any reason you can't define it in a separate package the
> blog-software depends on, as well as your wiki?

That's actually my point - most plugin systems I've seen, like the one
trac uses, are not encouraging you to do so. Having a module that just
defines an Interface is kind of weird - and in the real world no one is
doing it.

> And then of course, this is not really needed. In Python, behavior
> counts, not type-information. So you can get away without any explicit
> declared interface. You might chose to not do that, for aestetic
> reasons, or better documentation. But you aren't forced.

Actually some plugin-systems in python do force you and they check if
your "implementation" comply with the "interface".


Here is one solution I came up with. Based on the earlier example:
> > Lets say you program a wiki and like to allow different kind of
> > authentications. So you create two plugins (for example one for
> > OpenID and one for Shibboleth).
> >
> > Some time later you feel like reinventing the wheel again and
> > you program a blog. Again you like to allow different ways of
> > authentication and you already wrote plugins for exactly the
> > same for your wiki, right?

auth_openid.py - providing the authentication service
http://dpaste.com/ho...

wiki.py - providing the wiki
http://dpaste.com/ho...

Now putting both together:

> import pbus
>
> import wiki
> import auth_openid
> # or: import auth_shibboleth
>
> pbus.get("wiki").run()

No interface definitions.


What do you think? Any obvious pitfalls (besides reinventing something)?

Please keep in mind that syntax/api is not "done" or anything its just
an concept presentation.

Thanks,
Florian


--
Florian Ludwig <dino@phidev.org>

Paul Kölle

2/16/2010 3:14:00 PM

0

Am 15.02.2010 23:12, schrieb Florian Ludwig:
> On Sun, 2010-02-14 at 18:47 +0100, Diez B. Roggisch wrote:
[...]
>> And then of course, this is not really needed. In Python, behavior
>> counts, not type-information. So you can get away without any explicit
>> declared interface. You might chose to not do that, for aestetic
>> reasons, or better documentation. But you aren't forced.
>
> Actually some plugin-systems in python do force you and they check if
> your "implementation" comply with the "interface".
Which is a good thing IMO ;)

> Here is one solution I came up with.
[...]
Can you post working code? It's not clear what "pbus" is and how it works.

cheers
Paul


> What do you think? Any obvious pitfalls (besides reinventing something)?
>
> Please keep in mind that syntax/api is not "done" or anything its just
> an concept presentation.
>
> Thanks,
> Florian
>
>
>

Diez B. Roggisch

2/16/2010 6:20:00 PM

0

Am 15.02.10 23:12, schrieb Florian Ludwig:
> On Sun, 2010-02-14 at 18:47 +0100, Diez B. Roggisch wrote:
>>> Here there problem with the trac (and other plugin systems I've
>> seen)
>>> approach:
>>>
>>> You need to define something like:
>>> |
>>> | class IAuthPlugin(Interface): [...]
>>> |
>>> in your blog software.
>>
>> Why? Any reason you can't define it in a separate package the
>> blog-software depends on, as well as your wiki?
>
> That's actually my point - most plugin systems I've seen, like the one
> trac uses, are not encouraging you to do so. Having a module that just
> defines an Interface is kind of weird - and in the real world no one is
> doing it.

Just because nobody doesn't do it doesn't mean it's not desirable. IMHO
the ones using interfaces usually immediatly implement them for some
cases - so you actually get the interface for you authentication
framework, and also implement an OpenID plugin for it. And this forms
one package. Then you can add new packages that implement other things.

repoze.who and what are examples for such design.


>
>> And then of course, this is not really needed. In Python, behavior
>> counts, not type-information. So you can get away without any explicit
>> declared interface. You might chose to not do that, for aestetic
>> reasons, or better documentation. But you aren't forced.
>
> Actually some plugin-systems in python do force you and they check if
> your "implementation" comply with the "interface".

I didn't say nobody does it, I just said it isn't needed to get a
working plugin system. If you don't want to have a separate package
*and* don't want to pull in any unused code, this is pretty much your
only option.

> Here is one solution I came up with. Based on the earlier example:
>>> Lets say you program a wiki and like to allow different kind of
>>> authentications. So you create two plugins (for example one for
>>> OpenID and one for Shibboleth).
>>>
>>> Some time later you feel like reinventing the wheel again and
>>> you program a blog. Again you like to allow different ways of
>>> authentication and you already wrote plugins for exactly the
>>> same for your wiki, right?
>
> auth_openid.py - providing the authentication service
> http://dpaste.com/ho...
>
> wiki.py - providing the wiki
> http://dpaste.com/ho...
>
> Now putting both together:
>
>> import pbus
>>
>> import wiki
>> import auth_openid
>> # or: import auth_shibboleth
>>
>> pbus.get("wiki").run()
>
> No interface definitions.
>
>
> What do you think? Any obvious pitfalls (besides reinventing something)?

I don't know what pbus is supposed to do. Nor how it's laid out on a
python package level.

Diez

Florian Ludwig

2/16/2010 7:51:00 PM

0

On Tue, 2010-02-16 at 16:14 +0100, Paul Kölle wrote:
> Am 15.02.2010 23:12, schrieb Florian Ludwig:
> > On Sun, 2010-02-14 at 18:47 +0100, Diez B. Roggisch wrote:
> >> [...]
> >> And then of course, this is not really needed. In Python, behavior
> >> counts, not type-information. So you can get away without any
> >> explicit
> >> declared interface. You might chose to not do that, for aestetic
> >> reasons, or better documentation. But you aren't forced.
> >
> > Actually some plugin-systems in python do force you and they check
> > if your "implementation" comply with the "interface".

> Which is a good thing IMO ;)

Type checking might be good but not pythonic - imo. The problem with
having interfaces needed to be declared (and therefore the possibility
to check) is as I outlined in my first mail:
Where to put those interface descriptions if you want both sides
pluggable? You probably end up depending on other modules/projects just
to import the interfaces.

> > Here is one solution I came up with.
> [...]
> Can you post working code? It's not clear what "pbus" is and how it
> works.

Oh, sorry. The idea is similar to "d-bus" - hence the name. Its "the
communcation system". The code is pretty simple and quick. Please
remember its just to show of the concept not to use it in any production
software or anything.

Instead of using classes as connecting-piece I thought using strings
(maybe path "/plugin/wiki" or "org.example.wiki") might solve my
problem. It might be possible to add optional interface validation
though.


pbus.py - just a simple concept implementation of a framework
http://dpaste.com/ho...

auth_openid.py - providing the authentication service
http://dpaste.com/ho...

wiki.py - providing the wiki
http://dpaste.com/ho...

Now putting both together:

> import pbus
>
> import wiki
> import auth_openid
> # or: import auth_shibboleth
>
> pbus.get("wiki").run()



--
Florian Ludwig <dino@phidev.org>

Florian Ludwig

2/17/2010 9:30:00 AM

0

On Tue, 2010-02-16 at 19:20 +0100, Diez B. Roggisch wrote:
> Am 15.02.10 23:12, schrieb Florian Ludwig:
> > On Sun, 2010-02-14 at 18:47 +0100, Diez B. Roggisch wrote:
> >>> Here there problem with the trac (and other plugin systems I've
> >> seen)
> >>> approach:
> >>>
> >>> You need to define something like:
> >>> |
> >>> | class IAuthPlugin(Interface): [...]
> >>>
> >>> in your blog software.
> >>
> >> Why? Any reason you can't define it in a separate package the
> >> blog-software depends on, as well as your wiki?
> >
> > That's actually my point - most plugin systems I've seen, like the one
> > trac uses, are not encouraging you to do so. Having a module that just
> > defines an Interface is kind of weird - and in the real world no one is
> > doing it.
>
> Just because nobody doesn't do it doesn't mean it's not desirable.

Good point.

> IMHO the ones using interfaces usually immediatly implement them for some
> cases - so you actually get the interface for you authentication
> framework, and also implement an OpenID plugin for it. And this forms
> one package. Then you can add new packages that implement other things.

So if I want to implement another authentication mechanism but doesn't
use any of the auth-frameworks code you mentioned you still would depend
on it.

> repoze.who and what are examples for such design.

repoze.who is new for me, it might not actually solve my problem here
but it looks interesting for other tasks :) Thanks.


> >> And then of course, this is not really needed. In Python, behavior
> >> counts, not type-information. So you can get away without any explicit
> >> declared interface. You might chose to not do that, for aestetic
> >> reasons, or better documentation. But you aren't forced.
> >
> > Actually some plugin-systems in python do force you and they check if
> > your "implementation" comply with the "interface".
>
> I didn't say nobody does it, I just said it isn't needed to get a
> working plugin system. If you don't want to have a separate package
> *and* don't want to pull in any unused code, this is pretty much your
> only option.

Do you know somebody/some project/system/... who does it?

Pulling unused code I definitely want to avoid so I probably will stay
with this way.


> > [...]
> >
> > What do you think? Any obvious pitfalls (besides reinventing something)?
>
> I don't know what pbus is supposed to do. Nor how it's laid out on a
> python package level.

Its "the plugin system", connecting "service providers" (plugins) and
"service users". In my example the "users" also are providers at the
same time, the service "auth", provides the service "wiki".

Hope that clarifies my intentions a bit.

Thanks,
Florian


PS I removed you (diez) from the TO list as that bounced. Probably
related to the (not existing?) domain nospam.web.de


--
Florian Ludwig <dino@phidev.org>

Diez B. Roggisch

2/17/2010 7:46:00 PM

0

>> IMHO the ones using interfaces usually immediatly implement them for some
>> cases - so you actually get the interface for you authentication
>> framework, and also implement an OpenID plugin for it. And this forms
>> one package. Then you can add new packages that implement other things.
>
> So if I want to implement another authentication mechanism but doesn't
> use any of the auth-frameworks code you mentioned you still would depend
> on it.

This depends on your definition of depending... (pun intended).

Sure, the code floats around. But it isn't *used* by your application.
Why do you bother so much?

>> I didn't say nobody does it, I just said it isn't needed to get a
>> working plugin system. If you don't want to have a separate package
>> *and* don't want to pull in any unused code, this is pretty much your
>> only option.
>
> Do you know somebody/some project/system/... who does it?
>
> Pulling unused code I definitely want to avoid so I probably will stay
> with this way.

Well, it's simply duck-typing, so you find it all over the place. Out of
my head I only know about e.g. TurboMail, that allows you to specify
plugins via pkg_resources entry points & simply uses them.

> Its "the plugin system", connecting "service providers" (plugins) and
> "service users". In my example the "users" also are providers at the
> same time, the service "auth", provides the service "wiki".

So it's hypothetical?

To summarize my point again: if you want explicit type-checking or
zope-style interface implementation, but don't want any code that
implements this (at least not imported) - you need an explicit package.
If you don't want this, use duck-typing.

>
> PS I removed you (diez) from the TO list as that bounced. Probably
> related to the (not existing?) domain nospam.web.de

I post through NNTP, and the email is obfuscated for the obvious reasons.