Asp Forum - extending Python - passing nested lists

Christian Meesters

1/28/2008 3:10:00 PM

Hi,

I would like to write a C-extension function for an application of mine. For
this I need to pass a nested list (like: [[a, b, c], [d, e, f], ...], where
all letters are floats) to the C-function. Now, with the code I have the
compiler is complaining: "subscripted value is neither array nor pointer".
Can somebody tell me what's wrong?

Here a code snippet to reproduce this problem:

static PyObject *_foo(PyObject *self, PyObject *args) {
int i;
long lenght;
float ax, ay, az;
PyObject *dummy_list;

if (!PyArg_ParseTuple(args, "O", &dummy_list))
return NULL;
dummy_list = PySequence_Fast(dummy_list, "argument must be iterable");

lenght = PyObject_Length(dummy_list);

for (i=0; i < lenght; i++) {
// part which does not work:
ax = dummy_list[i][0];
ay = dummy_list[i][1];
az = dummy_list[i][2];
}
return 0;
}

TIA
Christian

9 Answers

Mark Dickinson

1/28/2008 6:21:00 PM

On Jan 28, 10:10 am, Christian Meesters <meest...@uni-mainz.de> wrote:
> Hi,
>
> I would like to write a C-extension function for an application of mine. For
> this I need to pass a nested list (like: [[a, b, c], [d, e, f], ...], where
> all letters are floats) to the C-function. Now, with the code I have the
> compiler is complaining: "subscripted value is neither array nor pointer".
> Can somebody tell me what's wrong?

Well, it's pretty clear: you misspelt "length" as "lenght". :)

PySequence_Fast doesn't return an array: it returns a PyObject---in
this case, a PyObject corresponding to a Python tuple. As the
compiler says, a PyObject is neither an array nor a pointer, so when
you write

dummy_list[i]

the compiler doesn't know what you mean. You probably want to use
PySequence_Fast_GET_ITEM. See the documentation at

http://docs.python.org/api/sequence.ht...

Mark

Christian Meesters

1/28/2008 6:30:00 PM

Mark Dickinson wrote:

> Well, it's pretty clear: you misspelt "length" as "lenght". :)
Well, that's not it ;-). (Damn copy & paste plague ...)
>
> PySequence_Fast doesn't return an array: it returns a PyObject---in
> this case, a PyObject corresponding to a Python tuple.
That's it. Thanks. Think I just need a different approach. I got completely
on the wrong track here.

Thanks
Christian

Christian Meesters

1/29/2008 12:49:00 PM

Think, that I'm still at the wrong track. Point is that I cannot find any
examples and don't know where to start here.
Perhaps my problem boils down to two questions:
I'd like to pass lists (in some cases nested ones) from Python to C and
convert those Python-lists to C-arrays (e. g. of doubles). My second wish
is to return a C-array of longs to a Python list.

My approach so far:

static PyObject *_foo(PyObject *self, PyObject *args) {
double *v;
if (!PyArg_Parse(args, "(d)", &v))
return NULL;
// then I can't access v like v[1] ...
<snip>
// then return *v
return with something like PyBuildValue (but didn't get so far)
}

Can somebody give me a hint here, please? Passing simple arguments to and
fro (e. g. single integer values) is no problem, but lists of unknown size?

TIA
Christian

Arnaud Delobelle

1/29/2008 1:07:00 PM

On Jan 29, 12:48 pm, Christian Meesters <meest...@uni-mainz.de> wrote:
> Think, that I'm still at the wrong track. Point is that I cannot find any
> examples and don't know where to start here.
> Perhaps my problem boils down to two questions:
> I'd like to pass lists (in some cases nested ones) from Python to C and
> convert those Python-lists to C-arrays (e. g. of doubles). My second wish
> is to return a C-array of longs to a Python list.
>
> My approach so far:
>
> static PyObject *_foo(PyObject *self, PyObject *args) {
> double *v;
> if (!PyArg_Parse(args, "(d)", &v))
> return NULL;
> // then I can't access v like v[1] ...

I'm not a C API guru and I may not understand properly what info you
are looking for but from http://docs.python.org/api/seq...:

PyObject* PySequence_GetItem( PyObject *o, Py_ssize_t i)

Return value: New reference.
Return the ith element of o, or NULL on failure. This is the
equivalent of the Python expression "o[i]".

> <snip>
> // then return *v
> return with something like PyBuildValue (but didn't get so far)

This allows you to create a list (from http://docs.python.org/api/listOb...):

PyObject* PyList_New( Py_ssize_t len)

Return value: New reference.
Return a new list of length len on success, or NULL on failure. Note:
If length is greater than zero, the returned list object's items are
set to NULL. Thus you cannot use abstract API functions such as
PySequence_SetItem() or expose the object to Python code before
setting all items to a real object with PyList_SetItem().

...and this allows you to populate it (from http://docs.python.org/api/listOb...):

int PyList_Append( PyObject *list, PyObject *item)

Append the object item at the end of list list. Return 0 if
successful; return -1 and set an exception if unsuccessful. Analogous
to list.append(item).

>
> }
>
> Can somebody give me a hint here, please? Passing simple arguments to and
> fro (e. g. single integer values) is no problem, but lists of unknown size?

HTH

--
Arnaud

Christian Meesters

1/29/2008 1:23:00 PM

Thanks. Point is that all such approaches would require lots(!) of calls to
the Python API - a way by which I won't gain the desired speed.

I've tried pyrex and the corresponding C-file is so convoluted with dummy
variables, incrementing & decrementing references, and other stuff, that I
want to try to write a C-function myself. My goal is not to avoid
PyObjects* and the corresponding reference handling - apart from the head
and end of the function. (I could use ctypes instead, but that again would
obfuscate the API of my package a bit.)

Christian

Arnaud Delobelle

1/29/2008 1:45:00 PM

On Jan 29, 1:22 pm, Christian Meesters <meest...@uni-mainz.de> wrote:
> Thanks. Point is that all such approaches would require lots(!) of calls to
> the Python API - a way by which I won't gain the desired speed.

You didn't mention speed in your original post. What about using
array.array? Unless I am mistaken, these are just a thin wrapper
around normal C arrays. Anyway you could always convert your list
into a c array, do lots and lots of fast calculations, then convert it
back again to a list.

--
Arnaud

Christian Meesters

1/29/2008 4:01:00 PM

> You didn't mention speed in your original post.
Sorry, perhaps I considered this self-evident - which it is, of course, not.

> What about using
> array.array? Unless I am mistaken, these are just a thin wrapper
> around normal C arrays.
The algorithm I want to implement requires several million floating point
operations. Neither the array-modules nor numpy's thin layer seem thin
enough for me. ;-)

> Anyway you could always convert your list
> into a c array, do lots and lots of fast calculations, then convert it
> back again to a list.
I guess I am too blind to see, but I couldn't discover a method description
like "double* PyList_toDouble". So, yes, my question is a C-API-newbie
question: What is the way to say "myCarray = SomePyMethod(InputPyList)"? Or
better, what should be here instead
static PyObject *_foo(PyObject *self, PyObject *args) {
double *v;
if (!PyArg_Parse(args, "(d)", &v))
return NULL;
to get a list as an array of doubles into 'v' (and back to Python)?

I did read the API-description, but still am lost at this point. I presume,
once I get to know the answer I'll bang my head on the table ... ;-)

Cheers
Christian

Mel

1/29/2008 4:25:00 PM

Christian Meesters wrote:
>> You didn't mention speed in your original post.
> Sorry, perhaps I considered this self-evident - which it is, of course, not.
>
>> What about using
>> array.array? Unless I am mistaken, these are just a thin wrapper
>> around normal C arrays.
> The algorithm I want to implement requires several million floating point
> operations. Neither the array-modules nor numpy's thin layer seem thin
> enough for me. ;-)
>
>> Anyway you could always convert your list
>> into a c array, do lots and lots of fast calculations, then convert it
>> back again to a list.
> I guess I am too blind to see, but I couldn't discover a method description
> like "double* PyList_toDouble". So, yes, my question is a C-API-newbie
> question: What is the way to say "myCarray = SomePyMethod(InputPyList)"? Or
> better, what should be here instead
> static PyObject *_foo(PyObject *self, PyObject *args) {
> double *v;
> if (!PyArg_Parse(args, "(d)", &v))
> return NULL;
> to get a list as an array of doubles into 'v' (and back to Python)?
>
> I did read the API-description, but still am lost at this point. I presume,
> once I get to know the answer I'll bang my head on the table ... ;-)

I haven't strictly tried this, but PyArg_ParseTuple and Py_BuildValue
seem to be the orthodox ways to do Python->C and C->Python conversions.
But if Numpy isn't fast enough, then any Python at all in the solution
might be too much. Perhaps keeping your values in a file and reading
them into the C programs will work.

Mel.

Arnaud Delobelle

1/29/2008 6:30:00 PM

On Jan 29, 4:00 pm, Christian Meesters <meest...@uni-mainz.de> wrote:
> > You didn't mention speed in your original post.
>
> Sorry, perhaps I considered this self-evident - which it is, of course, not.
>
> > What about using
> > array.array? Unless I am mistaken, these are just a thin wrapper
> > around normal C arrays.
>
> The algorithm I want to implement requires several million floating point
> operations. Neither the array-modules nor numpy's thin layer seem thin
> enough for me. ;-)

I'm not sure I understand. Here, taken from Modules/arraymodule.c, is
the definition of an arrayobject:

typedef struct arrayobject {
PyObject_VAR_HEAD
char *ob_item;
Py_ssize_t allocated;
struct arraydescr *ob_descr;
PyObject *weakreflist; /* List of weak references */
} arrayobject;

Now here is the function that gets an item from an array of floats:

static PyObject *
f_getitem(arrayobject *ap, Py_ssize_t i)
{
return PyFloat_FromDouble((double) ((float *)ap->ob_item)[i]);
}

This means that if you define an array of floats, call the arrayobject
'ap', then (float *)ap->ob_item seems to meexactly what you want: a c
array of floats. In what way is this not suitable?

> > Anyway you could always convert your list
> > into a c array, do lots and lots of fast calculations, then convert it
> > back again to a list.
>
> I guess I am too blind to see, but I couldn't discover a method description
> like "double* PyList_toDouble". So, yes, my question is a C-API-newbie
> question: What is the way to say "myCarray = SomePyMethod(InputPyList)"? Or
> better, what should be here instead
> static PyObject *_foo(PyObject *self, PyObject *args) {
> double *v;
> if (!PyArg_Parse(args, "(d)", &v))
> return NULL;
> to get a list as an array of doubles into 'v' (and back to Python)?

You can always iterate over the elements of the list an fill your c
array with the results. Look at the array_fromlist() function in
arraymodule.c, this function does exactly this.

HTH

--
Arnaud

comp.lang.python

extending Python - passing nested lists

Christian Meesters

Mark Dickinson

Christian Meesters

Christian Meesters

Arnaud Delobelle

Christian Meesters

Arnaud Delobelle

Christian Meesters

Mel

Arnaud Delobelle

x Login to ForumsZone