Asp Forum - mmap and shared memory

Matias Surdi

2/11/2008 11:42:00 PM

Suppose I've a process P1, which generates itself a lot of data , for
example 2Mb.
Then, I've a process P2 which must access P1 shared memory and,
probably, modify this data.
To accomplish this, I've been digging around python's mmap module, but I
can't figure how to use it without files.

Could anybody explain me how could this be accomplished? An example will
be very appreciated. Thanks a lot for your help.

8 Answers

Carl Banks

2/12/2008 1:04:00 AM

On Feb 11, 6:41 pm, Matias Surdi <matiassu...@gmail.com> wrote:
> Suppose I've a process P1, which generates itself a lot of data , for
> example 2Mb.
> Then, I've a process P2 which must access P1 shared memory and,
> probably, modify this data.
> To accomplish this, I've been digging around python's mmap module, but I
> can't figure how to use it without files.
>
> Could anybody explain me how could this be accomplished? An example will
> be very appreciated. Thanks a lot for your help.

In C you can use the mmap call to request a specific physical location
in memory (whence I presume two different processes can mmap anonymous
memory block in the same location), but Python doesn't expose this.

There isn't really as much drawback to using a file as you might
expect, and there are benefits. If you use an anonymous map the OS
could still write that memory to swap space, which, if your swap space
is fixed and limited, might negatively affect performance for the rest
of the system.

Carl Banks

Tim Roberts

2/12/2008 7:17:00 AM

Matias Surdi <matiassurdi@gmail.com> wrote:

>Suppose I've a process P1, which generates itself a lot of data , for
>example 2Mb.
>Then, I've a process P2 which must access P1 shared memory and,
>probably, modify this data.
>To accomplish this, I've been digging around python's mmap module, but I
>can't figure how to use it without files.

So, let it use a temporary file. What's the harm? An anonymous mmap
region is still mapped to the swap file. Might as well give it a name.
--
Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

greg

2/13/2008 4:58:00 AM

Carl Banks wrote:
> In C you can use the mmap call to request a specific physical location
> in memory (whence I presume two different processes can mmap anonymous
> memory block in the same location)

Um, no, it lets you specify the *virtual* address in the process's
address space at which the object you specify is to be mapped.

As far as I know, the only way two unrelated processes can share
memory via mmap is by mapping a file. An anonymous block is known
only to the process that creates it -- being anonymous, there's
no way for another process to refer to it.

However, if one process is forked from the other, the parent
can mmap an anonymous block and the child will inherit that
mapping.

(I suppose if both processes had sufficient privileges they
could map physical memory out of /dev/mem, but that would be
*really* dangerous!)

--
Greg

Jeff Schwab

2/13/2008 5:14:00 AM

greg wrote:
> Carl Banks wrote:
>> In C you can use the mmap call to request a specific physical location
>> in memory (whence I presume two different processes can mmap anonymous
>> memory block in the same location)
>
> Um, no, it lets you specify the *virtual* address in the process's
> address space at which the object you specify is to be mapped.
>
> As far as I know, the only way two unrelated processes can share
> memory via mmap is by mapping a file. An anonymous block is known
> only to the process that creates it -- being anonymous, there's
> no way for another process to refer to it.

On POSIX systems, you can create a shared memory object without a file
using shm_open. The function returns a file descriptor.

> However, if one process is forked from the other, the parent
> can mmap an anonymous block and the child will inherit that
> mapping.
>
> (I suppose if both processes had sufficient privileges they
> could map physical memory out of /dev/mem, but that would be
> *really* dangerous!)
>
> --
> Greg

Philip

2/14/2008 2:34:00 AM

In article <Ko2dncp9M48X4S_anZ2dnUVZ_ryqnZ2d@comcast.com>,
Jeff Schwab <jeff@schwabcenter.com> wrote:

> greg wrote:
> > Carl Banks wrote:
> >> In C you can use the mmap call to request a specific physical location
> >> in memory (whence I presume two different processes can mmap anonymous
> >> memory block in the same location)
> >
> > Um, no, it lets you specify the *virtual* address in the process's
> > address space at which the object you specify is to be mapped.
> >
> > As far as I know, the only way two unrelated processes can share
> > memory via mmap is by mapping a file. An anonymous block is known
> > only to the process that creates it -- being anonymous, there's
> > no way for another process to refer to it.
>
> On POSIX systems, you can create a shared memory object without a file
> using shm_open. The function returns a file descriptor.

Sorry I missed the OP, but you might be interested in this shared memory
module for Python:
http://NikitaTheS...p...

--
Philip
http://NikitaTheS...
Whole-site HTML validation, link checking and more

Jeff Schwab

2/14/2008 4:13:00 AM

Nikita the Spider wrote:
> In article <Ko2dncp9M48X4S_anZ2dnUVZ_ryqnZ2d@comcast.com>,
> Jeff Schwab <jeff@schwabcenter.com> wrote:
>
>> greg wrote:
>>> Carl Banks wrote:
>>>> In C you can use the mmap call to request a specific physical location
>>>> in memory (whence I presume two different processes can mmap anonymous
>>>> memory block in the same location)
>>> Um, no, it lets you specify the *virtual* address in the process's
>>> address space at which the object you specify is to be mapped.
>>>
>>> As far as I know, the only way two unrelated processes can share
>>> memory via mmap is by mapping a file. An anonymous block is known
>>> only to the process that creates it -- being anonymous, there's
>>> no way for another process to refer to it.
>> On POSIX systems, you can create a shared memory object without a file
>> using shm_open. The function returns a file descriptor.
>
> Sorry I missed the OP, but you might be interested in this shared memory
> module for Python:
> http://NikitaTheSpider.com/p...

Thanks; I just downloaded it. It seems to be missing the INSTALL file;
any idea where I could find that, or should I write to the author?

Philip

2/14/2008 8:54:00 PM

In article <C5ydnQgAo_Q8Ii7anZ2dnUVZ_uCinZ2d@comcast.com>,
Jeff Schwab <jeff@schwabcenter.com> wrote:

> Nikita the Spider wrote:
> > In article <Ko2dncp9M48X4S_anZ2dnUVZ_ryqnZ2d@comcast.com>,
> > Jeff Schwab <jeff@schwabcenter.com> wrote:
> >
> >> greg wrote:
> >>> Carl Banks wrote:
> >>>> In C you can use the mmap call to request a specific physical location
> >>>> in memory (whence I presume two different processes can mmap anonymous
> >>>> memory block in the same location)
> >>> Um, no, it lets you specify the *virtual* address in the process's
> >>> address space at which the object you specify is to be mapped.
> >>>
> >>> As far as I know, the only way two unrelated processes can share
> >>> memory via mmap is by mapping a file. An anonymous block is known
> >>> only to the process that creates it -- being anonymous, there's
> >>> no way for another process to refer to it.
> >> On POSIX systems, you can create a shared memory object without a file
> >> using shm_open. The function returns a file descriptor.
> >
> > Sorry I missed the OP, but you might be interested in this shared memory
> > module for Python:
> > http://NikitaTheS...p...
>
>
> Thanks; I just downloaded it. It seems to be missing the INSTALL file;
> any idea where I could find that, or should I write to the author?

The main author has been AWOL for some years now. I'm the current
maintainer. Sorry about the missing INSTALL file. Looks like I made
reference to it in the README but never created it. It installs with the
normal setup.py:

sudo python setup.py

I'll put out an updated package with an INSTALL file one of these days.
Thanks for pointing that out.

Cheers

--
Philip
http://NikitaTheS...
Whole-site HTML validation, link checking and more

Ryan Smith-Roberts

2/15/2008 8:18:00 PM

On Feb 11, 3:41 pm, Matias Surdi <matiassu...@gmail.com> wrote:
> Suppose I've a process P1, which generates itself a lot of data , for
> example 2Mb.
> Then, I've a process P2 which must access P1 shared memory and,
> probably, modify this data.
> To accomplish this, I've been digging around python's mmap module, but I
> can't figure how to use it without files.
>
> Could anybody explain me how could this be accomplished? An example will
> be very appreciated. Thanks a lot for your help.

A non-portable solution, for modern Linux systems, is to create and
mmap a file in the /dev/shm directory. This is a more "unix-y"
solution than the SysV SHM interface mentioned elsewhere in the
thread, since it gives you files you can cat, ls, chown, etc. Files
created in this directory may hit swap, but don't consume normal disk
space.

comp.lang.python

mmap and shared memory

Matias Surdi

Carl Banks

Tim Roberts

greg

Jeff Schwab

Philip

Jeff Schwab

Philip

Ryan Smith-Roberts

x Login to ForumsZone