Asp Forum - [ANN] ruby-dipus 0.1.2

Ilmari Heikkinen

12/27/2005 7:05:00 AM

DIPUS - Distributed IPC by Proxying UNIX Sockets================================================Sockets with user authentication, file permissions, encryption,service discovery and network transparent addressing.Using services:---host1$ dcat lame-svc foo.wav | dcat madplay.host3host1$ dcat sep://example.com/image/jpeg/to/image/png foo.jpeg | dcatssh://user@another.com/image/png/to/text/plain | lessCreating services:---host3$ dipus-local-pipe 'madplay -' madplayhost2$ dipus-export-pipe 'lame - -' lame-svcSome performance numbers:---Local throughput around 50MBps (or 100MBps, depending whether you usesocat as a client or not), unencrypted network throughput 9.8MBps in a100Mbps LAN, CPU-limited when using SSH (my other computer is slow.)In what languages can I use and create services?---Anything that supports a) UNIX domain sockets, and b) listingfilesystem directories. So, just about every language available. Ifyou install the helper scripts in bin, you don't even need those, justthe ability to read from stdin and write to stdout.Download: http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2.t...: http://www.cs.helsinki.fi/u/irkheikk/dipus-... repo: http://dark.fhtr.org/repos/ruby-d...: http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2/readme...: http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2/doc/dipus-design.t... spec: http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2/doc/dipus-specification.txt^^^^^^^^^^^^^^^^^^..., designed and implemented a distributed IPC system that takeslessons from UNIX (keep it simple) and HTTP (keep it simple.) Opinionson the design or anything else? I'm dying to get some feedback,writing stuff in a bubble can take you only so far.And, now, with the first version of this December project out of theway, I'm off to librend land~--Ilmari HeikkinenvvvvvvvvvvvvvvvvvvvvvvvvvvWhat is this for?-----------------You have a program. You want to access it remotely with the minimumhassle. It'd be nice to access it efficiently locally too. Withoutrewriting your connection code.And it'd be nice to put some connections over SSH. And be notifiedwhen a service dies or a new one is created. And not expose anyof the individual services to the network, or even to other userson the machine running them. And it'd be cool to be able to have adifferent name and protocol for each service without having to shoehornFTP over XML-RPC. Oh, also, display a real-time list of servicesrunning on the local network and on explicitly defined machines thataren't on the local network. And it'd be especially cool if all thesoftware could just use existing libraries like sockets and files.So, take UNIX sockets and sprinkle some sugar to make them usable fordistributed IPC.Supported systems-----------------Should work on Linux and *BSD, likely on OS X too.No Windows support, although it's theoretically possible.Required--------Ruby version 1.8.2+ (one released in 2005)catRecommended-----------OpenSSH version > 3.9 (for connection sharing support (ControlPath))ssh-agent setup (for maximum convenience)socatMore than one computer.How does it work?-----------------To create a service, make a listening socket in a subdir of/tmp/dipus, so that the path is like:/tmp/dipus/exported/audio/x-wav/to/audio/mpeg/lame.mybox..0To access a service, open its socket.To make a service only accessible to members of group 'audio':chgrp audio socket_namechmod 770 socket_nameListing services that you can see:tree /tmp/dipusDIPUS provides helper methods and daemons to manipulate thesockets in /tmp/dipus, e.g. create proxies to remote sockets,delete dead sockets, provide network access to exportedsockets, find sockets by their service name, hostname, orprotocol.Quick usage example-------------------# (optional) start dipusd to automatically discover and import services,# export services, and delete dead sockets.#[kig@desktop] dipusd[kig@encode] dipusd# Create local madplay service for playing MP3s on# the computer connected to speakers.# The first argument is the command to run for each incoming connection.# The second argument is service_name/protocol, e.g. my_www/http.## The following command creates a socket with the path# /tmp/dipus/local/audio/mpeg/to/audio/out/madplay.stereo..0# and runs 'madplay -' for each incoming connection, setting the# connection to madplay's stdin and stdout.#[kig@stereo] dipus-local-pipe 'madplay -' madplay/audio/mpeg/to/audio/out# Create an exported lame service for encoding wavs to MP3s.#[kig@encode] dipus-export-pipe 'lame - -' lame/audio/x-wav/to/audio/mpeg# Manually import the services from stereo.# Import over SSH to gain access to all services and not just exported ones.#[kig@desktop] dipus-import ssh://stereo# Dipusd should've discovered and imported the lame service from encode,# so we don't need to manually import that. But if that didn't happen, we# can import it by doing `dipus-import sep://encode`, sep being# Socket Export Protocol (see doc/dipus-specification.txt)# Now, let's pipe some wavs to the stereo.# First encode the wav to MP3, using any service with a protocol that says# that the service converts audio/x-wav to audio/mpeg.# Play using a service named madplay running on stereo.#[kig@desktop] dcat audio/x-wav/to/audio/mpeg *.wav | dcat madplay.stereo# We could also access the services without importing them by giving the# service URI as the service name.## Here's the above piping example using service URIs.# Dcat lists the services matching the URI and uses the first match.#[kig@desktop] dcat sep://encode/audio/x-wav/to/audio/mpeg *.wav | dcat ssh://stereo/audio/mpeg/to/audio/out/madplay.stereo..0List of DIPUS commands---------------------- Pass --help as an argument to get the full USAGE. * dls -- list services dls sep://foobar/audio dls ssh://barfoo * dcat -- pipe stdin and/or given files to a DIPUS socket dcat lame foo.wav bar.wav > foobar.mp3 dcat ssh://foobar/image/png/to/image/jpeg cat.png > cat.jpg * dipus-import -- imported all found service from an URI dipus-import ssh://foobar/audio dipus-import sep://barfoo * dipusd -- start exporter, importer, advertiser, discoverer, dead socket monitor, and default services * dipus-exporter -- start exporter * dipus-discoverer -- discover exporters on the LAN and print their addresses * dipus-service-monitor -- monitor for service changes on localhost, used by SSH importers * dipus-ssh-importer -- import services over SSH to user's DIPUS subdirectory (/tmp/dipus/username), can use discovery to find exporter-running hosts on localnet * dipus-local-pipe -- create a local DIPUS service that forks the given command for the connection dipus-local-pipe 'head' head/text * dipus-export-pipe -- create an exported DIPUS service like above dipus-export-pipe 'madplay -' madplay/audio/mpeg/to/audio/outUsing socat to create and connect to DIPUS services---------------------------------------------------# proxy a web servermkdir -p /tmp/dipus/exported/httpsocat UNIX-LISTEN:/tmp/dipus/exported/http/foo.foobar..0,fork TCP4-CONNECT:foo.example.com:80rm /tmp/dipus/exported/http/foo.foobar..0# using a lame service to encode a wavsocat foo.wav UNIX-CONNECT:/tmp/dipus/local/audio/x-wav/to/audio/mpeg/lame.foobar..0 > foo.mp3# or by using dls to find the filenamesocat foo.wav UNIX-CONNECT:`dls -1 -f lame` > foo.mp3# proxy a DIPUS service to a TCP socketsocat TCP4-LISTEN:1234,fork EXEC:'dcat lame'Possibilities-------------Carry around a mobile phone with an Internet connection. Use it torun a media player. The media player sees files on your home computerand the files on the local network. Then it pipes the files in itsplaylist to the closest encode server, and from there to the wantedoutput (headphones, speakers, file, streaming server...)Send your 3D animation file over the Internet to a thousand idlemachines to render.Walk into an auditorium and display slides on the overhead projector,streamed from a network mount, controlled from your mobile.Run a search engine, giving URIs to remote services, making automatablethe task of finding and utilizing a network service.

2 Answers

Pau Garcia i Quiles

12/27/2005 9:02:00 AM

Quoting Ilmari Heikkinen <ilmari.heikkinen@gmail.com>:

Wow, that's exactly what I was looking for! Thank you!

Just one "but": I think using a known path under /tmp/dipus may be a security
risk.

> DIPUS - Distributed IPC by Proxying UNIX Sockets
> ================================================
> Sockets with user authentication, file permissions, encryption,
> service discovery and network transparent addressing.
>
> Using services:
> ---
> host1$ dcat lame-svc foo.wav | dcat madplay.host3
> host1$ dcat sep://example.com/image/jpeg/to/image/png foo.jpeg | dcat
> ssh://user@another.com/image/png/to/text/plain | less
>
> Creating services:
> ---
> host3$ dipus-local-pipe 'madplay -' madplay
> host2$ dipus-export-pipe 'lame - -' lame-svc
>
> Some performance numbers:
> ---
> Local throughput around 50MBps (or 100MBps, depending whether you use
> socat as a client or not), unencrypted network throughput 9.8MBps in a
> 100Mbps LAN, CPU-limited when using SSH (my other computer is slow.)
>
> In what languages can I use and create services?
> ---
> Anything that supports a) UNIX domain sockets, and b) listing
> filesystem directories. So, just about every language available. If
> you install the helper scripts in bin, you don't even need those, just
> the ability to read from stdin and write to stdout.
>
> Download: http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0....
> RDocs: http://www.cs.helsinki.fi/u/irkheikk/d...
> Darcs repo: http://dark.fhtr.org/repos/...
>
> Readme: http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2/...
> Design:
> http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2/doc/dipus-...
> Protocol spec:
> http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2/doc/dipus-specifi...
>
> ^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> Hi, designed and implemented a distributed IPC system that takes
> lessons from UNIX (keep it simple) and HTTP (keep it simple.) Opinions
> on the design or anything else? I'm dying to get some feedback,
> writing stuff in a bubble can take you only so far.
>
> And, now, with the first version of this December project out of the
> way, I'm off to librend land~
> --
> Ilmari Heikkinen
>
> vvvvvvvvvvvvvvvvvvvvvvvvvv
>
> What is this for?
> -----------------
> You have a program. You want to access it remotely with the minimum
> hassle. It'd be nice to access it efficiently locally too. Without
> rewriting your connection code.
>
> And it'd be nice to put some connections over SSH. And be notified
> when a service dies or a new one is created. And not expose any
> of the individual services to the network, or even to other users
> on the machine running them. And it'd be cool to be able to have a
> different name and protocol for each service without having to shoehorn
> FTP over XML-RPC. Oh, also, display a real-time list of services
> running on the local network and on explicitly defined machines that
> aren't on the local network. And it'd be especially cool if all the
> software could just use existing libraries like sockets and files.
>
> So, take UNIX sockets and sprinkle some sugar to make them usable for
> distributed IPC.
>
>
> Supported systems
> -----------------
> Should work on Linux and *BSD, likely on OS X too.
> No Windows support, although it's theoretically possible.
>
>
> Required
> --------
> Ruby version 1.8.2+ (one released in 2005)
> cat
>
>
> Recommended
> -----------
> OpenSSH version > 3.9 (for connection sharing support (ControlPath))
> ssh-agent setup (for maximum convenience)
> socat
> More than one computer.
>
>
> How does it work?
> -----------------
> To create a service, make a listening socket in a subdir of
> /tmp/dipus, so that the path is like:
> /tmp/dipus/exported/audio/x-wav/to/audio/mpeg/lame.mybox..0
>
> To access a service, open its socket.
>
> To make a service only accessible to members of group 'audio':
> chgrp audio socket_name
> chmod 770 socket_name
>
> Listing services that you can see:
> tree /tmp/dipus
>
> DIPUS provides helper methods and daemons to manipulate the
> sockets in /tmp/dipus, e.g. create proxies to remote sockets,
> delete dead sockets, provide network access to exported
> sockets, find sockets by their service name, hostname, or
> protocol.
>
>
> Quick usage example
> -------------------
> # (optional) start dipusd to automatically discover and import services,
> # export services, and delete dead sockets.
> #
> [kig@desktop] dipusd
> [kig@encode] dipusd
>
> # Create local madplay service for playing MP3s on
> # the computer connected to speakers.
> # The first argument is the command to run for each incoming connection.
> # The second argument is service_name/protocol, e.g. my_www/http.
> #
> # The following command creates a socket with the path
> # /tmp/dipus/local/audio/mpeg/to/audio/out/madplay.stereo..0
> # and runs 'madplay -' for each incoming connection, setting the
> # connection to madplay's stdin and stdout.
> #
> [kig@stereo] dipus-local-pipe 'madplay -' madplay/audio/mpeg/to/audio/out
>
> # Create an exported lame service for encoding wavs to MP3s.
> #
> [kig@encode] dipus-export-pipe 'lame - -' lame/audio/x-wav/to/audio/mpeg
>
> # Manually import the services from stereo.
> # Import over SSH to gain access to all services and not just exported ones.
> #
> [kig@desktop] dipus-import ssh://stereo
>
> # Dipusd should've discovered and imported the lame service from encode,
> # so we don't need to manually import that. But if that didn't happen, we
> # can import it by doing `dipus-import sep://encode`, sep being
> # Socket Export Protocol (see doc/dipus-specification.txt)
>
> # Now, let's pipe some wavs to the stereo.
> # First encode the wav to MP3, using any service with a protocol that says
> # that the service converts audio/x-wav to audio/mpeg.
> # Play using a service named madplay running on stereo.
> #
> [kig@desktop] dcat audio/x-wav/to/audio/mpeg *.wav | dcat madplay.stereo
>
> # We could also access the services without importing them by giving the
> # service URI as the service name.
> #
> # Here's the above piping example using service URIs.
> # Dcat lists the services matching the URI and uses the first match.
> #
> [kig@desktop] dcat sep://encode/audio/x-wav/to/audio/mpeg *.wav | > dcat ssh://stereo/audio/mpeg/to/audio/out/madplay.stereo..0
>
>
> List of DIPUS commands
> ----------------------
> Pass --help as an argument to get the full USAGE.
>
> * dls -- list services
> dls sep://foobar/audio
> dls ssh://barfoo
>
> * dcat -- pipe stdin and/or given files to a DIPUS socket
> dcat lame foo.wav bar.wav > foobar.mp3
> dcat ssh://foobar/image/png/to/image/jpeg cat.png > cat.jpg
>
> * dipus-import -- imported all found service from an URI
> dipus-import ssh://foobar/audio
> dipus-import sep://barfoo
>
> * dipusd -- start exporter, importer, advertiser, discoverer,
> dead socket monitor, and default services
>
> * dipus-exporter -- start exporter
>
> * dipus-discoverer -- discover exporters on the LAN and print
> their addresses
>
> * dipus-service-monitor -- monitor for service changes on localhost,
> used by SSH importers
>
> * dipus-ssh-importer -- import services over SSH to user's DIPUS
> subdirectory (/tmp/dipus/username), can use
> discovery to find exporter-running hosts on
> localnet
>
> * dipus-local-pipe -- create a local DIPUS service that forks the
> given command for the connection
> dipus-local-pipe 'head' head/text
>
> * dipus-export-pipe -- create an exported DIPUS service like above
> dipus-export-pipe 'madplay -' madplay/audio/mpeg/to/audio/out
>
>
> Using socat to create and connect to DIPUS services
> ---------------------------------------------------
> # proxy a web server
> mkdir -p /tmp/dipus/exported/http
> socat UNIX-LISTEN:/tmp/dipus/exported/http/foo.foobar..0,fork > TCP4-CONNECT:foo.example.com:80
> rm /tmp/dipus/exported/http/foo.foobar..0
>
> # using a lame service to encode a wav
> socat foo.wav > UNIX-CONNECT:/tmp/dipus/local/audio/x-wav/to/audio/mpeg/lame.foobar..0 > > foo.mp3
>
> # or by using dls to find the filename
> socat foo.wav UNIX-CONNECT:`dls -1 -f lame` > foo.mp3
>
> # proxy a DIPUS service to a TCP socket
> socat TCP4-LISTEN:1234,fork EXEC:'dcat lame'
>
>
> Possibilities
> -------------
> Carry around a mobile phone with an Internet connection. Use it to
> run a media player. The media player sees files on your home computer
> and the files on the local network. Then it pipes the files in its
> playlist to the closest encode server, and from there to the wanted
> output (headphones, speakers, file, streaming server...)
>
> Send your 3D animation file over the Internet to a thousand idle
> machines to render.
>
> Walk into an auditorium and display slides on the overhead projector,
> streamed from a network mount, controlled from your mobile.
>
> Run a search engine, giving URIs to remote services, making automatable
> the task of finding and utilizing a network service.
>

Ilmari Heikkinen

12/27/2005 8:01:00 PM

On 12/27/05, Pau Garcia i Quiles <pgquiles@elpauer.org> wrote:> Quoting Ilmari Heikkinen <ilmari.heikkinen@gmail.com>:>> Wow, that's exactly what I was looking for! Thank you!Thank you for trying it out, let me know if / when you find bugs ;)> Just one "but": I think using a known path under /tmp/dipus may be a security> risk.How do you mean?

comp.lang.ruby

[ANN] ruby-dipus 0.1.2

Ilmari Heikkinen

Pau Garcia i Quiles

Ilmari Heikkinen

x Login to ForumsZone