Asp Forum - Do arrays use up alot of memory space?

Clement Ow

4/7/2008 1:27:00 AM

I just started ruby not too long ago and I'm really new to the language.
But I came out with something; to copy files from path names to path
names using each respecive array elements.

--------------------------------------------------------------------------------
$options=
["2008*", "2008*", "700*", "2008*", "2008*"]

$source=
%w[C:/movtest/testing
C:/movtest/testing/new
U:/movtest/source
U:/movtest/new
U:/movtest/new1]

$dest=
%w[U:/test_1/
U:/dest1/
U:/dest2/
U:/dest3/
U:/dest4/]

while i<=j && i1<=j1 && i2<=j2

Dir.chdir($source[i])
print "\nSource: " + Dir.getwd + "\t\n"
print "Dest: " + $dest[i1] + "\n"
print "Options: " + $options[i2] +"\n"
FileUtils.cp_r Dir.glob($options[i2]), $dest[i1]
print "File Mov Test:Success"
i+=1
i1+=1
i2+=1
end

--------------------------------------------------------------------------------
Apparently, when moving large files (i.e file size 50mb) it takes
relatively long as when i use ROBOCOPY (a robust copying software by
MSServer), but I cant use robocopy because it has it's limitations,hence
i used ruby. As I will be running the ruby program on a server, I cant
afford to use too much memory while transferring files from one folder
to another as the server is used for more impt tasks like running impt
applications in the office.

So is there anything that i could do to quicken the copying process
without compromising on the ability to have different path names and
options for copying files? (i understand that arrays do take up more
memory)
Thanks in advance. =)
--
Posted via http://www.ruby-....

13 Answers

7stud --

4/7/2008 4:29:00 AM

Clement Ow wrote:
> --------------------------------------------------------------------------------
> $options=
> ["2008*", "2008*", "700*", "2008*", "2008*"]
>
> $source=
> %w[C:/movtest/testing
> C:/movtest/testing/new
> U:/movtest/source
> U:/movtest/new
> U:/movtest/new1]
>
> $dest=
> %w[U:/test_1/
> U:/dest1/
> U:/dest2/
> U:/dest3/
> U:/dest4/]
>

You have 3 arrays with 5 short strings in them. A character in a string
takes up 1 byte of memory. I don't know exactly what the overhead for
an array is, but it is comparatively tiny--although you never know with
ruby. I would guess all your arrays would take up less than 1,000
bytes or 1 kB in memory. That's tiny.

> while i<=j && i1<=j1 && i2<=j2
>
> Dir.chdir($source[i])
> print "\nSource: " + Dir.getwd + "\t\n"
> print "Dest: " + $dest[i1] + "\n"
> print "Options: " + $options[i2] +"\n"
> FileUtils.cp_r Dir.glob($options[i2]), $dest[i1]
> print "File Mov Test:Success"
> i+=1
> i1+=1
> i2+=1
> end
>
> --------------------------------------------------------------------------------
> Apparently, when moving large files (i.e file size 50mb) it takes
> relatively long as when i use ROBOCOPY (a robust copying software by
> MSServer), but I cant use robocopy because it has it's limitations,hence
> i used ruby. As I will be running the ruby program on a server, I cant
> afford to use too much memory while transferring files from one folder
> to another as the server is used for more impt tasks like running impt
> applications in the office.
>
> So is there anything that i could do to quicken the copying process
> without compromising on the ability to have different path names and
> options for copying files? (i understand that arrays do take up more
> memory)
> Thanks in advance. =)
--
Posted via http://www.ruby-....

Robert Dober

4/7/2008 7:27:00 PM

On Mon, Apr 7, 2008 at 6:29 AM, 7stud -- <bbxx789_05ss@yahoo.com> wrote:
> Clement Ow wrote:
> > --------------------------------------------------------------------------------
> > $options=
> > ["2008*", "2008*", "700*", "2008*", "2008*"]
> >
> > $source=
> > %w[C:/movtest/testing
> > C:/movtest/testing/new
> > U:/movtest/source
> > U:/movtest/new
> > U:/movtest/new1]
> >
> > $dest=
> > %w[U:/test_1/
> > U:/dest1/
> > U:/dest2/
> > U:/dest3/
> > U:/dest4/]
> >
>
> You have 3 arrays with 5 short strings in them. A character in a string
> takes up 1 byte of memory. I don't know exactly what the overhead for
> an array is, but it is comparatively tiny--although you never know with
> ruby.
A well judged statement again....

For Ruby 1.9 the overhead of an array is one long and 4 VALUES, I am
not a VM Guru but it would surprise me a lot if a VALUE used more than
4 bytes given their semantic meanings shared flags and klass, the
fourth being a pointer to the content.
I am sure that someone will correct me if I am wrong but I estimate
the overhead of less than 21 bytes.

I would guess all your arrays would take up less than 1,000
>yes that seems indeed a save guess....
HTH
Robert
--
http://ruby-smalltalk.blo...

---
Whereof one cannot speak, thereof one must be silent.
Ludwig Wittgenstein

Clement Ow

4/8/2008 1:36:00 AM

Robert Dober wrote:
> On Mon, Apr 7, 2008 at 6:29 AM, 7stud -- <bbxx789_05ss@yahoo.com> wrote:
>> > U:/movtest/new1]
>> takes up 1 byte of memory. I don't know exactly what the overhead for
>> an array is, but it is comparatively tiny--although you never know with
>> ruby.
> A well judged statement again....
>
> For Ruby 1.9 the overhead of an array is one long and 4 VALUES, I am
> not a VM Guru but it would surprise me a lot if a VALUE used more than
> 4 bytes given their semantic meanings shared flags and klass, the
> fourth being a pointer to the content.
> I am sure that someone will correct me if I am wrong but I estimate
> the overhead of less than 21 bytes.
>
> I would guess all your arrays would take up less than 1,000
>>yes that seems indeed a save guess....
> HTH
> Robert
> --
> http://ruby-smalltalk.blo...
>
> ---
> Whereof one cannot speak, thereof one must be silent.
> Ludwig Wittgenstein

Thanks for the prompt reply, 7student and Robert =) But while
transferring files, it does take a significant amount time though, so
the primary reason does not lie in the arrray? If so, is there any way I
can quicken the process of transferring files?

--
Posted via http://www.ruby-....

Phillip Gawlowski

4/8/2008 2:56:00 AM

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Clement Ow wrote:

|
| Thanks for the prompt reply, 7student and Robert =) But while
| transferring files, it does take a significant amount time though, so
| the primary reason does not lie in the arrray? If so, is there any way I
| can quicken the process of transferring files?

It probably isn't the array. You probably are IO bound (i.e. speed of
the hard drive(s)/network connection).

Are you copying the files on the same disk, or from disk to disk, and on
different IDE/SATA channels?

If you are copying across a network, how good is the connection?

- --
Phillip Gawlowski
Twitter: twitter.com/cynicalryan

:zorkmid: /zork'mid/ n. The canonical unit of currency in
~ hacker-written games. This originated in {zork} but has spread
~ to {nethack} and is referred to in several other games.
~ -- The AI Hackers Dictionary
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail....

iEYEARECAAYFAkf63s8ACgkQbtAgaoJTgL8IUQCdEx+pcVfXapfAMdCszaz8D/bx
7zQAnjL4OJxQ+/Z3MMQ4tym7AipPLKfs
=dnhS
-----END PGP SIGNATURE-----

Clement Ow

4/9/2008 2:03:00 AM

Thanks Botp and Philip for the help! ;) anyway, ans philip, the network
i'm running is quite optimal though not really sure about the speeds, as
I'm running this program in a company with alot of servers dealing with
loads of jobs a day :)

PeÃ±a, Botp wrote:
> From: Clement Ow [mailto:clement.ow@asia.bnpparibas.com]
> # $options=
> you may remove the "$" sign ;)

i use the "$" sign because I have a config file that holds all my
values, but I didnt include it for the purpose of this example ;)
>
>
> # U:/movtest/source
> # U:/movtest/new
> # U:/movtest/new1]
> #
> # $dest=
> # %w[U:/test_1/
> # U:/dest1/
> # U:/dest2/
> # U:/dest3/
> # U:/dest4/]
> #
> # while i<=j && i1<=j1 && i2<=j2
>
> hmmm, you just created 3 parallel arrays. Either you can create one
> multi-dimensional or a hash.
>
>
> # Dir.chdir($source[i])
>
> you may not need this
>
> # print "\nSource: " + Dir.getwd + "\t\n"
> # print "Dest: " + $dest[i1] + "\n"
> # print "Options: " + $options[i2] +"\n"
> # FileUtils.cp_r Dir.glob($options[i2]), $dest[i1]
> # print "File Mov Test:Success"
> # i+=1
> # i1+=1
> # i2+=1
>
> if you modify your code, you may not need those increments ;)
>
> # end
> # ------------------
> # Apparently, when moving large files (i.e file size 50mb) it takes
> # relatively long as when i use ROBOCOPY (a robust copying software by
> # MSServer),
>
> robocopy seems to do stat on the filetransfer speed to get the optimum
> chunk to transfer. You can do something like that in ruby, but you'll
> have to work harder (since your code/logic will get a bit longer)
>
> # but I cant use robocopy because it has it's limitations
>
> it's not perfect, and w ruby, you can beat it :)
yea, limitations being that I cant have exceptions while selecting a
serious of folders. EG. I wanna select files and folders that start with
2008 but I want to keep the last day of the month untouched.(still yet
to figure out this part though)
>
> So, if i were to tackle this, i'd first do, (note, this is ruby
> pseudocode, ergo untested since i do not have time and space to test it
> now...)
>
>
> source=%w[C:/movtest/testing
> C:/movtest/testing/new
> U:/movtest/source
> U:/movtest/new
> U:/movtest/new1]
>
> dest=%w[U:/test_1/
> U:/dest1/
> U:/dest2/
> U:/dest3/
> U:/dest4/]
>
> selections=["2008*", "2008*", "700*", "2008*", "2008*"]
>
> # here i combine those 3 arrays into 1 multi-dim array
> sd_a=source.zip(dest,options)

hmmm, would combining the arrays be feasible if i were to add on more
values to the either of the arrays?

>
> sd_a.each do |sd|
> source, destination, selections = sd
> src = File.join source,selections
> puts "Source: #{src}"
> puts "Dest: #{destination}"
>
> # i comment the 2 lines below for you to choose bw recurse or not
> # again, if you recurse, check for directory overlaps
>
> # FileUtils.cp src, destination
> # FileUtils.cp_r src, destination

This command runs an error because the value of src is
"C:/movtest/test/2008*" and it doesnt recognise this wildcard but
instead looks for the file which has the name 2008*.
so i tried to chang it alil:

Dir.chdir(source)
FileUtils.cp_r Dir.glob('2008*'), source, destination

But still runs an error here showing :
Source: C:/movtest/testing/2008*
Dest: U:/test_1/
C:/movtest/testing
c:/ruby/lib/ruby/1.8/fileutils.rb:1438:in `delete': can't convert Symbol
into St
ring (TypeError)
from c:/ruby/lib/ruby/1.8/fileutils.rb:1438:in
`fu_check_options'
from c:/ruby/lib/ruby/1.8/fileutils.rb:1437:in `each'
from c:/ruby/lib/ruby/1.8/fileutils.rb:1437:in
`fu_check_options'
from c:/ruby/lib/ruby/1.8/fileutils.rb:418:in `cp_r'
from testing.rb:44
from testing.rb:32:in `each'
from testing.rb:32

I have tried ways and means to try correct it but to no avail. Is there
anything majorly wrong with my code?
--
Posted via http://www.ruby-....

Clement Ow

4/9/2008 3:12:00 AM

PeÃ±a, Botp wrote:
> From: list-bounce@example.com
> # > # here i combine those 3 arrays into 1 multi-dim array
> # > sd_a=source.zip(dest,options)
> #
> # hmmm, would combining the arrays be feasible if i were to add on more
> # values to the either of the arrays?
>
> i zipped it because,
> 1. i was too lazy to retype it again
> 2. i knew it would work since your arrays are one-to-one on each other
>
> # > sd_a.each do |sd|
> # > source, destination, selections = sd
> # > src = File.join source,selections
> # > puts "Source: #{src}"
> # > puts "Dest: #{destination}"
> # >
> # > # i comment the 2 lines below for you to choose bw recurse or not
> # > # again, if you recurse, check for directory overlaps
> # >
> # > # FileUtils.cp src, destination
> # > # FileUtils.cp_r src, destination
> #
> # This command runs an error because the value of src is
> # "C:/movtest/test/2008*" and it doesnt recognise this wildcard but
> # instead looks for the file which has the name 2008*.
> # so i tried to chang it alil:
>
> no problem, just do
>
> FileUtils.cp Dir.glob(src), destination
>
> that should work because i just tested it here :)
>
> kind regards -botp

it works ;)
But one issue, when i changed to FileUtils.mv Dir.glob(src), destination
because i needed to move the files, it shows a big error msg:
c:/ruby/lib/ruby/1.8/fileutils.rb:505:in `rename': Permission denied -
C:/movtes
t/testing/20080321 or U:/test_1/20080321 (Errno::EACCES)
from c:/ruby/lib/ruby/1.8/fileutils.rb:505:in `move'
from c:/ruby/lib/ruby/1.8/fileutils.rb:1395:in
`fu_each_src_dest'
from c:/ruby/lib/ruby/1.8/fileutils.rb:1404:in
`fu_each_src_dest0'
from c:/ruby/lib/ruby/1.8/fileutils.rb:1402:in `each'
from c:/ruby/lib/ruby/1.8/fileutils.rb:1402:in
`fu_each_src_dest0'
from c:/ruby/lib/ruby/1.8/fileutils.rb:1393:in
`fu_each_src_dest'
from c:/ruby/lib/ruby/1.8/fileutils.rb:494:in `move'
from testing.rb:42
from testing.rb:32:in `each'
from testing.rb:32

It seems like some problem with the access rights.. Any advice?
I'm still quite new to ruby so ya, but I'm really thankful for the help
so far though. ;)

--
Posted via http://www.ruby-....

Phillip Gawlowski

4/9/2008 3:18:00 AM

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Clement Ow wrote:

|
| It seems like some problem with the access rights.. Any advice?
| I'm still quite new to ruby so ya, but I'm really thankful for the help
| so far though. ;)

Yeah, contact your sysadmin, if you have write permission. :P

If you are the admin (and executing the script with admin rights), make
sure no other programs are using the directory/file (file locks).

That means, you can't have the script in the directory that you move. ;)

(Sounds obvious, but it happens to the best...)

- --
Phillip Gawlowski
Twitter: twitter.com/cynicalryan

You were ever good at sudden commendations.
~ -- William Shakespeare (1564-1616), King Henry VIII
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail....

iEYEARECAAYFAkf8NYIACgkQbtAgaoJTgL+/8wCfbBnRMgzG/oV0pkaAa67EmTLX
eHcAn2db9lJ0bNi6Iti/NUQU71lIn3w6
=SjNH
-----END PGP SIGNATURE-----

Clement Ow

4/9/2008 3:28:00 AM

Phillip Gawlowski wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Clement Ow wrote:
>
> |
> | It seems like some problem with the access rights.. Any advice?
> | I'm still quite new to ruby so ya, but I'm really thankful for the help
> | so far though. ;)
>
> Yeah, contact your sysadmin, if you have write permission. :P
>
> If you are the admin (and executing the script with admin rights), make
> sure no other programs are using the directory/file (file locks).
Im running the script with admin rights.
>
> That means, you can't have the script in the directory that you move. ;)
The script is residing in C:\rubytest which is not in any of the path
names above.
And I'm quite sure it's not due to the options i.e Verbose, force or
noop.
Tried it with the force option (returns no error) but no file move was
done.
hmmm, which makes it more puzzling..

--
Posted via http://www.ruby-....

Phillip Gawlowski

4/9/2008 3:33:00 AM

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Clement Ow wrote:

| The script is residing in C:\rubytest which is not in any of the path
| names above.
| And I'm quite sure it's not due to the options i.e Verbose, force or
| noop.
| Tried it with the force option (returns no error) but no file move was
| done.
| hmmm, which makes it more puzzling..

Are you sure there are no open file handles? FileMon should reveal this:
http://technet.microsoft.com/en-us/sysinternals/bb8...

If there are no file handles shown, check if you can move the
directories/files via explorer.

- --
Phillip Gawlowski
Twitter: twitter.com/cynicalryan

[Abstract art is] a product of the untalented, sold by the unprincipled
~ to the utterly bewildered.
~ -- Al Capp
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail....

iEYEARECAAYFAkf8OPUACgkQbtAgaoJTgL9anQCbBvBvvBkraRKzy5flCuycYOeS
D3sAn16d9VsMJzZx6wm8TkVfC4BiIS3k
=24fF
-----END PGP SIGNATURE-----

Clement Ow

4/9/2008 4:18:00 AM

Phillip Gawlowski wrote:

> Are you sure there are no open file handles? FileMon should reveal this:
> http://technet.microsoft.com/en-us/sysinternals/bb8...
>
> If there are no file handles shown, check if you can move the
> directories/files via explorer.

Hmmm, i dont quite understand about the open file handles..
Do you mean if the folder is being in use without the knowledge of the
user which in this case is myslelf? If so, I dont see the files being
used by other programs whatsoever. I even closed explorer and run the
script, but to no avail..

-in desperate need for help

--
Posted via http://www.ruby-....

comp.lang.ruby

Do arrays use up alot of memory space?

Clement Ow

7stud --

Robert Dober

Clement Ow

Phillip Gawlowski

Clement Ow

Clement Ow

Phillip Gawlowski

Clement Ow

Phillip Gawlowski

Clement Ow

x Login to ForumsZone