[lnkForumImage]
TotalShareware - Download Free Software

Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.python

Socket Performance

sleddd

3/13/2008 4:47:00 AM

Can anyone explain why socket performance (throughput) varies
depending on the amount of data send and recv are called with?

For example: try creating a local client/server (running on the same
computer) where the server sends the client a fixed amount of data.
Using method A, recv(8192) and sendall( ) with 8192 bytes worth of
data. Do this 100 times. Using method B, recv(1) and sendall( ) with 1
byte worth of data. Do this 819200 times.

If you time both methods, method A has much greater throughput than
method B.

Server:

import socket
import random
import string
import time

HOST = 'localhost'
PORT = 50023
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print 'Connected by', addr

string_1 = 'A'
string_8192 = ''.join([random.choice(string.letters + string.digits)
for i in range(8192)])

conn.sendall('Start')

start = time.clock()
total_data = 0
for i in range(0,100):
conn.sendall(string_8192)
total_data += len(string_8192)
print 'Send Speed (Long String): ' + str( total_data / (time.clock() -
start) / 1024 / 1024 ) + ' MB/sec\n\n'

start = time.clock()
total_data = 0
for i in range(0,819200):
conn.sendall(string_1)
total_data += len(string_1)
print 'Send Speed (Short String): ' + str( total_data / (time.clock()
- start) / 1024 / 1024 ) + ' MB/sec'

conn.close()


Client:

import socket
import time

HOST = 'localhost' # The remote host
PORT = 50023 # The same port as used by the server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, 50026))
s.connect((HOST, PORT))

data = s.recv(5)
print 'From Server: ' + data

start = time.clock()
total_data = 0
while total_data < 819200:
data = s.recv(8192)
total_data += len(data)
print 'Receive Speed (Long String): ' + str( total_data /
(time.clock() - start) / 1024 / 1024 ) + ' MB/sec\n\n'

start = time.clock()
total_data = 0
while total_data < 819200:
data = s.recv(1)
total_data += len(data)
print 'Receive Speed (Short String): ' + str( total_data /
(time.clock() - start) / 1024 / 1024 ) + ' MB/sec'
s.close()
14 Answers

Brian Smith

3/13/2008 1:34:00 PM

0

sleddd@gmail.com wrote:
> Sent: Wednesday, March 12, 2008 9:47 PM
> To: python-list@python.org
> Subject: Socket Performance
>
> Can anyone explain why socket performance (throughput) varies
> depending on the amount of data send and recv are called with?
>
> For example: try creating a local client/server (running on the same
> computer) where the server sends the client a fixed amount of data.
> Using method A, recv(8192) and sendall( ) with 8192 bytes
> worth of data. Do this 100 times. Using method B, recv(1) and
> sendall( ) with 1 byte worth of data. Do this 819200 times.
>
> If you time both methods, method A has much greater
> throughput than method B.

Why is it faster to drink a liter of water a cupful at a time than to
drink it out of an eyedropper?

- Brian

sleddd

3/13/2008 5:19:00 PM

0

On Mar 13, 9:33 am, "Brian Smith" <br...@briansmith.org> wrote:
> sle...@gmail.com wrote:
> > Sent: Wednesday, March 12, 2008 9:47 PM
> > To: python-l...@python.org
> > Subject: Socket Performance
>
> > Can anyone explain why socket performance (throughput) varies
> > depending on the amount of data send and recv are called with?
>
> > For example: try creating a local client/server (running on the same
> > computer) where the server sends the client a fixed amount of data.
> > Using method A, recv(8192) and sendall( ) with 8192 bytes
> > worth of data. Do this 100 times. Using method B, recv(1) and
> > sendall( ) with 1 byte worth of data. Do this 819200 times.
>
> > If you time both methods, method A has much greater
> > throughput than method B.
>
> Why is it faster to drink a liter of water a cupful at a time than to
> drink it out of an eyedropper?
>
> - Brian

Well, lets say you have a situation where you're going to be
alternating between sending large and small chunks of data. Is the
solution to create a NetworkBuffer class and only call send when the
buffer is full, always recv(8192)?

Grant Edwards

3/13/2008 6:38:00 PM

0

On 2008-03-13, sleddd@gmail.com <sleddd@gmail.com> wrote:

>> > For example: try creating a local client/server (running on the same
>> > computer) where the server sends the client a fixed amount of data.
>> > Using method A, recv(8192) and sendall( ) with 8192 bytes
>> > worth of data. Do this 100 times. Using method B, recv(1) and
>> > sendall( ) with 1 byte worth of data. Do this 819200 times.
>>
>> > If you time both methods, method A has much greater
>> > throughput than method B.
>>
>> Why is it faster to drink a liter of water a cupful at a time than to
>> drink it out of an eyedropper?

> Well, lets say you have a situation where you're going to be
> alternating between sending large and small chunks of data. Is the
> solution to create a NetworkBuffer class and only call send when the
> buffer is full, always recv(8192)?

If you need to send large and small chumks of data, the
solution is to send large and small chunks of data.

--
Grant

Dennis Lee Bieber

3/14/2008 6:36:00 AM

0

On Thu, 13 Mar 2008 10:18:44 -0700 (PDT), sleddd@gmail.com declaimed the
following in comp.lang.python:


> Well, lets say you have a situation where you're going to be
> alternating between sending large and small chunks of data. Is the
> solution to create a NetworkBuffer class and only call send when the
> buffer is full, always recv(8192)?

Or create a protocol where the first 16 bits (in network byte order)
contain a length value for the subsequent data, and use a receive
process that consists of:

leng = ntoh(socket.recv(2))
data = socket.receive(leng)

(the send can combine the length with the data into a single packet)
--
Wulfraed Dennis Lee Bieber KD6MOG
wlfraed@ix.netcom.com wulfraed@bestiaria.com
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: web-asst@bestiaria.com)
HTTP://www.bestiaria.com/

Aaron Brady

3/14/2008 7:02:00 AM

0

> > Well, lets say you have a situation where you're going to be
> > alternating between sending large and small chunks of data. Is the
> > solution to create a NetworkBuffer class and only call send when the
> > buffer is full, always recv(8192)?
>
>         Or create a protocol where the first 16 bits (in network byte order)
> contain a length value for the subsequent data, and use a receive
> process that consists of:
>
> leng = ntoh(socket.recv(2))
> data = socket.receive(leng)
>
> (the send can combine the length with the data into a single packet)

Are two 'sends' guaranteed to arrive as at least two 'receives'?

Send-3: xxx
Send-3: yyy
Receive-6: xxxyyy

Bryan Olson

3/14/2008 7:42:00 AM

0

castironpi@gmail.com wrote:
[Dennis Lee Bieber had written:]
>> Or create a protocol where the first 16 bits (in network byte order)
>> contain a length value for the subsequent data, and use a receive
>> process that consists of:
>>
>> leng = ntoh(socket.recv(2))
>> data = socket.receive(leng)
>>
>> (the send can combine the length with the data into a single packet)
>
> Are two 'sends' guaranteed to arrive as at least two 'receives'?

No. Nor are they guaranteed to arrive as at least most two.

> Send-3: xxx
> Send-3: yyy
> Receive-6: xxxyyy

Can happen, though I think the problem with Dennis's code is the
other way. The recv in

leng = ntoh(socket.recv(2))

might return one byte of data, not two. The latter recv is similar.


--
--Bryan

Bryan Olson

3/14/2008 8:07:00 AM

0

sleddd@gmail.com wrote:
> Well, lets say you have a situation where you're going to be
> alternating between sending large and small chunks of data. Is the
> solution to create a NetworkBuffer class and only call send when the
> buffer is full, always recv(8192)?

Buffering can often improve performance, but with the above we'd
to too quick to prematurely jump to an as yet unwarranted conclusion.

You measured the one-large and many-small cases, but not the case
you actually have. You might be able to take various measurements
and generally characterize speed as a function of the number of
calls and the size of the send. I wouldn't be surprised if the
result is well approximated by a time per call plus a time per
byte.

In optimization, guessing is bad. Where you cannot reason with
mathematical rigor, measure. Where you can derive a purely
analytic result, taking the occasional measurements still isn't a
bad idea (but don't tell my algorithms students).

--
--Bryan

Dennis Lee Bieber

3/15/2008 5:40:00 AM

0

On Fri, 14 Mar 2008 07:42:05 GMT, Bryan Olson <fakeaddress@nowhere.org>
declaimed the following in comp.lang.python:

>
> Can happen, though I think the problem with Dennis's code is the
> other way. The recv in
>
> leng = ntoh(socket.recv(2))
>
> might return one byte of data, not two. The latter recv is similar.

Concede I took a simple and quick view -- the main point is that one
probably needs to define a protocol that somehow defines or specifies
when a variable length data packet is complete...

SMTP, as I recall, uses the convention

\r\n.\r\n

to signal the end of the message body.

In my example, if you know that all packets are prefaced by a two
byte length, even if the network feels like a tease and is releasing
things one byte at a time, one knows they must collect at least two
bytes to make up a length value, and then can loop over receives for
that length, collecting the data into a buffer.
--
Wulfraed Dennis Lee Bieber KD6MOG
wlfraed@ix.netcom.com wulfraed@bestiaria.com
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: web-asst@bestiaria.com)
HTTP://www.bestiaria.com/

Gabriel Genellina

3/15/2008 8:34:00 AM

0

En Thu, 13 Mar 2008 15:18:44 -0200, <sleddd@gmail.com> escribió:

> Well, lets say you have a situation where you're going to be
> alternating between sending large and small chunks of data. Is the
> solution to create a NetworkBuffer class and only call send when the
> buffer is full, always recv(8192)?

No need to reinvent the wheel. socket objects already have a makefile
method returning a file-like object, which behaves like a buffered socket.

--
Gabriel Genellina

Aaron Brady

3/15/2008 9:01:00 AM

0

On Mar 15, 3:33 am, "Gabriel Genellina" <gagsl-...@yahoo.com.ar>
wrote:
> En Thu, 13 Mar 2008 15:18:44 -0200, <sle...@gmail.com> escribió:
>
> > Well, lets say you have a situation where you're going to be
> > alternating between sending large and small chunks of data. Is the
> > solution to create a NetworkBuffer class and only call send when the
> > buffer is full, always recv(8192)?
>
> No need to reinvent the wheel. socket objects already have a makefile  
> method returning a file-like object, which behaves like a buffered socket.

Newbie question: Can you write to the 'file-like object' a pickle,
and receive it intact-- as one string with nothing else?

I want to know because I want to send two pickles.