[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

Better performance than native unix commands?

ntwrkd

3/4/2008 1:11:00 AM

I have an issue with nfs and listing a large amount of files that I am
wondering if I can solve with ruby. I know that ruby has better
performance in some instances than a bash shell script so I am wondering
if anyone has experience or some examples of how this might work...

I have a directory with a large amount of files that is mounted over
nfs. The network is gigabit and so are the nic's. The servers are very
high end so the hardware / network performance is not the bottleneck.
However, sometimes it can take a very long time to list out the files
contained within a directory.

I will do an ls and it might take 5 minutes to retrieve a response.

Would this be faster if I were to do the same ls function using the ruby
api?
--
Posted via http://www.ruby-....
15 Answers

Markus Arike

3/4/2008 4:21:00 AM

0

Ma Sa wrote:
> I have an issue with nfs and listing a large amount of files that I am
> wondering if I can solve with ruby. I know that ruby has better
> performance in some instances than a bash shell script so I am wondering
> if anyone has experience or some examples of how this might work...
>
> I have a directory with a large amount of files that is mounted over
> nfs. The network is gigabit and so are the nic's. The servers are very
> high end so the hardware / network performance is not the bottleneck.
> However, sometimes it can take a very long time to list out the files
> contained within a directory.
>
> I will do an ls and it might take 5 minutes to retrieve a response.
>
> Would this be faster if I were to do the same ls function using the ruby
> api?

The only way to know for sure would be make some simple tests. I just
ran these using the Dir class:

Macintosh:~ marike$ time ruby -e 'd = Dir.new("."); d.each {|x| puts x}'
| wc
162 166 1761

real 0m0.027s
user 0m0.019s
sys 0m0.010s
Macintosh:~ marike$ time ls -A | wc
160 164 1756

real 0m0.008s
user 0m0.005s
sys 0m0.006s

ls does appear to be faster, but I'm sure others on this list know a
better Ruby way than that iterating over a Dir object.

Markus


--
Posted via http://www.ruby-....

Arlen Cuss

3/4/2008 5:41:00 AM

0

[Note: parts of this message were removed to make it a legal post.]

Hi,

On Tue, Mar 4, 2008 at 12:10 PM, Ma Sa <ntwrkd@gmail.com> wrote:

> I have an issue with nfs and listing a large amount of files that I am
> wondering if I can solve with ruby. I know that ruby has better
> performance in some instances than a bash shell script so I am wondering
> if anyone has experience or some examples of how this might work...
>
>
You may want to check your options. In some cases, I found using TCP instead
of UDP to increase speed quite a bit - it really depends on your network. It
also fixed a problem involving dropped packets over wireless leading to
connections never finishing their job. Also check your sync/async options.

Arlen

Dan Fitzpatrick

3/4/2008 5:44:00 AM

0

Ma Sa wrote:
> I have an issue with nfs and listing a large amount of files that I am
> wondering if I can solve with ruby. I know that ruby has better
> performance in some instances than a bash shell script so I am wondering
> if anyone has experience or some examples of how this might work...
>
> I have a directory with a large amount of files that is mounted over
> nfs. The network is gigabit and so are the nic's. The servers are very
> high end so the hardware / network performance is not the bottleneck.
> However, sometimes it can take a very long time to list out the files
> contained within a directory.
>
> I will do an ls and it might take 5 minutes to retrieve a response.
>
> Would this be faster if I were to do the same ls function using the ruby
> api?
>
What OS are you running? Something seems like it's not working. I run
FreeBSD 6.2 and I have almost 200,000 files in one folder on an NFS
mount and ls is pretty much an instant response. I doubt Ruby will be
any faster than a compiled C app (ls). Could be a firewall issue (pf and
nfs do not get along to well for example), locking issue, network
congestion or misconfiguration. You could try to see where the
bottleneck is by doing the following:

On the NFS server:

time ls > temp_file

Make sure the temp file is on the server too. This will tell you if the
bottleneck is with NFS/network or the server's file system.

On the client (with temp_file on the server):

time ls
time cat temp_file

This will tell you if it is the ls command or the NFS/network time to
send the data over the network.

Copy the temp_file to the client and see what the speed is on the client.

time cat temp_file


Here are my numbers for reference:

ls | wc -l
197621

time ls > /dev/null (local)
real 0m0.669s
user 0m0.550s
sys 0m0.114s

time ls > /dev/null (over nfs)
real 0m0.716s
user 0m0.606s
sys 0m0.070s


Dan

Lionel Bouton

3/4/2008 11:00:00 AM

0

Ma Sa wrote:
> I have an issue with nfs and listing a large amount of files that I am
> wondering if I can solve with ruby. I know that ruby has better
> performance in some instances than a bash shell script so I am wondering
> if anyone has experience or some examples of how this might work...
>
> I have a directory with a large amount of files that is mounted over
> nfs. The network is gigabit and so are the nic's. The servers are very
> high end so the hardware / network performance is not the bottleneck.
> However, sometimes it can take a very long time to list out the files
> contained within a directory.
>
> I will do an ls and it might take 5 minutes to retrieve a response.

Do you use any external name service (LDAP/NIS/...)? If so your problem
could be uid number to user name resolution taking too much time on so
many files.

You can check this by running 'ls -n', if it's much faster you'll have
to either:
- speed up name resolution (use a cache like nscd - which should be
installed or readily available on any Linux distribution - for example).
- don't resolve names when you don't need them.

Lionel

7stud --

3/4/2008 1:13:00 PM

0

Ma Sa wrote:
> I know that ruby has better
> performance....

lol.
--
Posted via http://www.ruby-....

Robert Klemme

3/4/2008 6:02:00 PM

0

On 04.03.2008 02:10, Ma Sa wrote:
> I have an issue with nfs and listing a large amount of files that I am
> wondering if I can solve with ruby. I know that ruby has better
> performance in some instances than a bash shell script so I am wondering
> if anyone has experience or some examples of how this might work...
>
> I have a directory with a large amount of files that is mounted over
> nfs. The network is gigabit and so are the nic's. The servers are very
> high end so the hardware / network performance is not the bottleneck.
> However, sometimes it can take a very long time to list out the files
> contained within a directory.
>
> I will do an ls and it might take 5 minutes to retrieve a response.
>
> Would this be faster if I were to do the same ls function using the ruby
> api?

Did you try it out? What happened? My guess is that it's as slow
because likely the timing is dominated by IO.

Kind regards

robert

Daniel Brumbaugh Keeney

3/5/2008 8:09:00 PM

0

On Mon, Mar 3, 2008 at 10:21 PM, Markus Arike <marike1@optonline.net> wrote:
>
> Ma Sa wrote:
> > I have an issue with nfs and listing a large amount of files that I am
> > wondering if I can solve with ruby. I know that ruby has better
> > performance in some instances than a bash shell script so I am wondering
> > if anyone has experience or some examples of how this might work...
> >
> > I have a directory with a large amount of files that is mounted over
> > nfs. The network is gigabit and so are the nic's. The servers are very
> > high end so the hardware / network performance is not the bottleneck.
> > However, sometimes it can take a very long time to list out the files
> > contained within a directory.
> >
> > I will do an ls and it might take 5 minutes to retrieve a response.
> >
> > Would this be faster if I were to do the same ls function using the ruby
> > api?
>
> The only way to know for sure would be make some simple tests.

I must beg to differ. Ruby doesn't stand a chance of being as fast as
your system's ls.

> ls does appear to be faster, but I'm sure others on this list know a
> better Ruby way than that iterating over a Dir object.
>
> Markus

The method you're looking for is Dir.foreach('.')

Daniel Brumbaugh Keeney

Rodrigo Bermejo

3/5/2008 8:32:00 PM

0

If you just need to list the name of the files, du is faster than ls
du -a should make it.

-r.

--
Posted via http://www.ruby-....

Daniel Brumbaugh Keeney

3/5/2008 8:55:00 PM

0

On Wed, Mar 5, 2008 at 2:32 PM, Rodrigo Bermejo
<rodrigo.bermejo@ps.ge.com> wrote:
> If you just need to list the name of the files, du is faster than ls
> du -a should make it.
>
> -r.

First of all, that's using the wrong tool for the job. The job of ls
is to list files, du is for estimating file space usage. Second of
all, on my system at least, you're wrong about speed:

$ time ls -1>/dev/null
real 0m0.013s
user 0m0.004s
sys 0m0.008s
$ time du -a>/dev/null
real 0m0.080s
user 0m0.004s
sys 0m0.020s

the -1 on ls is to prevent formatting of the results

Daniel Brumbaugh Keeney

Kyle Schmitt

3/5/2008 9:16:00 PM

0

From experience:
Your reverse DNS isn't setup properly.