[lnkForumImage]
TotalShareware - Download Free Software

Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.python

Distributed App - C++ with Python for Portability?

Roopan

3/9/2008 11:42:00 PM

Hello!

I am looking at developing an enterprise-grade distributed data
sharing application - key requirements are productivity and platform
portability.

Will it be sensible to use C++ for performance-critical sections and
Python for all the glue logic.

Pls comment from your *experiences* how Python scales to large
projects( > 200KLOC).
I assume the C++/Python binding is fairly painless.

Regards
Elam.
15 Answers

Paddy

3/10/2008 6:00:00 AM

0

On Mar 9, 11:41 pm, Roopan <elampoora...@gmail.com> wrote:
> Hello!
>
> I am looking at developing an enterprise-grade distributed data
> sharing application - key requirements are productivity and platform
> portability.
>
> Will it be sensible to use C++ for performance-critical sections and
> Python for all the glue logic.
>
> Pls comment from your *experiences* how Python scales to large
> projects( > 200KLOC).
> I assume the C++/Python binding is fairly painless.
>
> Regards
> Elam.

You might try prototyping as much as possible in Python as soon as
possible.
Then, and only after getting something that computes the right
results,
profile your prototype to see where the real bottlenecks are.
Sometimes it
it is not evident at the beginning where the bottlenecks in the
prototype
will be.
If you write most of the prototype in Python then you will write less
lines of code, in a shorter time and so should be more inclined to
experiment with different algorithms, and do more testing. (Doctest
can
be great).
After profiling their may be other ways to remove a bottleneck, such
as
using existing highly-optimised libraries such as Numpy; Psycho, an
optimising interpreter that can approach C type speeds for Python
code;
and you could create your own C++ based libraries.

You might want to ask the Mercurial development team how they got
their
impressive speed and functionality out of using mainly Python with
critical regions in C. - Or watch this:
http://video.google.com/videoplay?docid=-772429601...

- Paddy.


Matt Nordhoff

3/10/2008 7:47:00 AM

0

Paddy wrote:
> After profiling their may be other ways to remove a bottleneck, such
> as
> using existing highly-optimised libraries such as Numpy; Psycho, an
> optimising interpreter that can approach C type speeds for Python
> code;
> and you could create your own C++ based libraries.
>
> You might want to ask the Mercurial development team how they got
> their
> impressive speed and functionality out of using mainly Python with
> critical regions in C. - Or watch this:
> http://video.google.com/videoplay?docid=-772429601...

For what you do decide to rewrite in C, you can also use a language like
Cython [1] (which is a fork of Pyrex [2]). It looks mostly like Python,
and is translated to C without you having to write all of the
boilerplate Python C API stuff. Of course, not quite as efficient
well-tuned raw C, but much more pleasant to write.

(FWIW, Bazaar [3], another VCS written in Python similar to Mercurial,
has rewritten parts of two modules in Pyrex. Another one is in raw C,
because that's what was contributed.)

[1] <http://www.cytho...
[2] <http://www.cosc.canterbury.ac.nz/greg.ewing/python/...
[3] <http://bazaar-vc...
--

Stefan Behnel

3/10/2008 7:56:00 AM

0

Matt Nordhoff wrote:
> For what you do decide to rewrite in C, you can also use a language like
> Cython [1] (which is a fork of Pyrex [2]). It looks mostly like Python,
> and is translated to C without you having to write all of the
> boilerplate Python C API stuff. Of course, not quite as efficient
> well-tuned raw C, but much more pleasant to write.

And if you really need the efficiency of "well-tuned raw C", it's one function
call away in your Cython code.

Stefan

Matt Nordhoff

3/10/2008 8:19:00 AM

0

Stefan Behnel wrote:
> And if you really need the efficiency of "well-tuned raw C", it's one function
> call away in your Cython code.

What do you mean by that?

I know nothing about how Cython compares to C in performance, so I said
"well-tuned" because it must be possible to write C that is faster than
Cython, though it may take some effort.
--

Stefan Behnel

3/10/2008 8:32:00 AM

0

Matt Nordhoff wrote:
> Stefan Behnel wrote:
>> And if you really need the efficiency of "well-tuned raw C", it's one function
>> call away in your Cython code.
>
> What do you mean by that?
>
> I know nothing about how Cython compares to C in performance, so I said
> "well-tuned" because it must be possible to write C that is faster than
> Cython, though it may take some effort.

So, you write the hand-optimised function in plain C, declare it in Cython and
call it. That's what I meant. Since Cython compiles to C code, linking against
a C module is straight forward. And this still keeps you from having to write
all the Python API glue code in plain C.

Stefan

Aaron Brady

3/10/2008 9:46:00 AM

0

> >> And if you really need the efficiency of "well-tuned raw C", it's one function
> >> call away in your Cython code.
>
> > What do you mean by that?
>
> > I know nothing about how Cython compares to C in performance, so I said
> > "well-tuned" because it must be possible to write C that is faster than
> > Cython, though it may take some effort.
>
> So, you write the hand-optimised function in plain C, declare it in Cython and
> call it. That's what I meant. Since Cython compiles to C code, linking against
> a C module is straight forward. And this still keeps you from having to write
> all the Python API glue code in plain C.

Python was originally intended to just link C modules, right? (ITTRW
if that's the right word?) What are Python's roots? What are its
principles? What are its fundamentals? (And what does Python 8! look
like!?) We can even get multi-threading running outside of the Global
Interpreter Lock, if the thread only acquires it to access shared
object code... make that mananged objects.

One big decision is if you have a separate Python interpreter running
on every remote location, versus say as just a relay mechanism. C on
the rest. CMIIW, correct me if I'm wrong, but the productivity
bottlenecks vs. performance bottlenecks make a trade-off.

For my part, sometimes I get __getattr__ bottlenecks, and other times
object instantiation bottlenecks. There was an iterator one in there
too. But no I didn't have competing C code to test to compare. If I
understand OP correctly, C libraries will be running at all locations,
which means they'll need separate compilations. In fact, come to
think of it, I'm having trouble with the concept of cross-platform
distributed. Do OP mean that lots of programmers are sharing data?
Will all the platforms be performing all of the tasks? Or do they
specialize? (Not necessarily / no / yes ok.) Lastly, if you can get
a boost by buying more servers, then there's a resource bottleneck
breakpoint to consider too.

Diez B. Roggisch

3/10/2008 11:58:00 AM

0

Roopan wrote:

> Hello!
>
> I am looking at developing an enterprise-grade distributed data
> sharing application - key requirements are productivity and platform
> portability.
>
> Will it be sensible to use C++ for performance-critical sections and
> Python for all the glue logic.
>
> Pls comment from your *experiences* how Python scales to large
> projects( > 200KLOC).
> I assume the C++/Python binding is fairly painless.

It depends. There are good wrappers out there, I personally prefer SIP.
However, a mixed language environment is always a PITA, especially for
distribution.

If you can, write everything in python. Identify bottlenecks, and if you
must, I suggest using C + ctypes for performance-critical code.

Obviously it's a matter of taste, but C++ is a beast, and getting it to work
seamless under varying compilers and OSes could be avoided using plain C.

Diez

Bob Martin

3/10/2008 7:21:00 PM

0

in 337513 20080310 115744 "Diez B. Roggisch" <deets@nospam.web.de> wrote:
>Roopan wrote:
>
>> Hello!
>>
>> I am looking at developing an enterprise-grade distributed data
>> sharing application - key requirements are productivity and platform
>> portability.
>>
>> Will it be sensible to use C++ for performance-critical sections and
>> Python for all the glue logic.
>>
>> Pls comment from your *experiences* how Python scales to large
>> projects( > 200KLOC).
>> I assume the C++/Python binding is fairly painless.
>
>It depends. There are good wrappers out there, I personally prefer SIP.
>However, a mixed language environment is always a PITA, especially for
>distribution.
>
>If you can, write everything in python. Identify bottlenecks, and if you
>must, I suggest using C + ctypes for performance-critical code.
>
>Obviously it's a matter of taste, but C++ is a beast, and getting it to work
>seamless under varying compilers and OSes could be avoided using plain C.
>
>Diez

Java is more portable than most other languages, especially if your app needs a gui.

Stefan Behnel

3/10/2008 8:39:00 PM

0

Bob Martin wrote:
> Java is more portable than most other languages, especially if your app needs a gui.

I don't think anything is more portable for an App GUI than HTML through a
built-in web server.

That said, does gcj compile GUI Apps by now? Or does that not count as "portable"?

Stefan

Carl Banks

3/10/2008 8:59:00 PM

0

On Mar 9, 7:41 pm, Roopan <elampoora...@gmail.com> wrote:
> Hello!
>
> I am looking at developing an enterprise-grade distributed data
> sharing application - key requirements are productivity and platform
> portability.
>
> Will it be sensible to use C++ for performance-critical sections and
> Python for all the glue logic.
> Pls comment from your *experiences* how Python scales to large
> projects( > 200KLOC).
> I assume the C++/Python binding is fairly painless.


200K with C++ lines of code or Python lines of code? :)

I can comment on 50K lines of Python code, and considering how I write
Python and how most people write C++ that might be about the same as a
200K C++ program. But it's not an enterprise app, so take it as you
will.

I've had no real issues. 95% of the program in Python; with only a
couple sections written in C, and a few wrappers for C/C++ libraries.
My wrappers are as thin as I can reasonably make them; where needed I
make the interface prettier with a higher-level Python wrapper.

I'm writing regular extensions, not using ctypes, Cython, SWIG,
Boost::python or anything like that. I am using ctypes for some
related tools and it's pretty nice. The others I can't comment on,
except to say I sometimes didn't have enough memory to compile third-
party SWIG extensions.

I develop it on Linux, but every few months or so I'd copy it to
Windows and update it so that it works on both. This was a minor
hastle, which is not so bad considering it hadn't seen Windows for
months. (In fairness, I am using the same compiler, gcc, on both.
Getting mingw gcc to work on Windows was a major hassle, but a one-
time thing.)


Python's a lot better productivity-wise, and it's worked well as a
cross-platform solution for me. The interfacing is the main
drawback. It's always a bit of effort to interface things, and to get
it working on multiple platforms, but I thought it was worth it.


Carl Banks