Asp Forum - Too many open files on OpenBSD

Edd Barrett

3/6/2008 12:33:00 AM

Hi,

I am the maintainer of the TeXLive typesetting suite for OpenBSD. Each
year I must parse a database in order to calculate subsets of the
TeXLive system for packaging purposes.

The database format changed this year and I have thrown away my python
script and had a go at it with ruby. I am quite new to ruby, but not to
programming (previously C, C++, Java, BASIC..... )

This is the error I am stuck with:

---8<---
/tlpsrcnode.rb:19:in `initialize': Too many open files -
/home/tl/tl/Master/tlpkg/tlpsrc/wordcount.tlpsrc (Errno::EMFILE)
from ./tlpsrcnode.rb:19:in `new'
from ./tlpsrcnode.rb:19:in `parse'
from ./tlpsrcnode.rb:30:in `parse'
from ./tlpsrcnode.rb:20:in `each'
from ./tlpsrcnode.rb:20:in `parse'
from ./tlpsrcnode.rb:30:in `parse'
from ./tlpsrcnode.rb:20:in `each'
from ./tlpsrcnode.rb:20:in `parse'
... 3055 levels...
from ./tlpsrcnode.rb:20:in `each'
from ./tlpsrcnode.rb:20:in `parse'
from ./roottlpsrcnode.rb:53:in `startParse'
from ./rbmfsplit:54
---8<---

This appears to happen after 297 files are concurrently opened.

Immediately I dump myself into the staff login class and uncap the file
descriptor count in login.conf. Log out then back in.

---8<---
puff% ulimit -a
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) 524288
-s: stack size (kbytes) 4096
-c: core file size (blocks) unlimited
-m: resident set size (kbytes) 1747436
-l: locked-in-memory size (kb) 582993
-u: processes 128
-n: file descriptors 1024
---8<---

Uncapped in this case appears to be 1024. Plenty.

The error still occurs after 297 files are open.

Is there a workaround for this or is ruby not suitable for this nature
of processing?

Thanks

Edd
--
Posted via http://www.ruby-....

4 Answers

Edd Barrett

3/6/2008 12:38:00 AM

Edd Barrett wrote:
> ---8<---
> ./tlpsrcnode.rb:19:in `initialize': Too many open files -
> /home/tl/tl/Master/tlpkg/tlpsrc/wordcount.tlpsrc (Errno::EMFILE)
> from ./tlpsrcnode.rb:19:in `new'
> from ./tlpsrcnode.rb:19:in `parse'
> from ./tlpsrcnode.rb:30:in `parse'
> from ./tlpsrcnode.rb:20:in `each'
> from ./tlpsrcnode.rb:20:in `parse'
> from ./tlpsrcnode.rb:30:in `parse'
> from ./tlpsrcnode.rb:20:in `each'
> from ./tlpsrcnode.rb:20:in `parse'
> ... 3055 levels...
> from ./tlpsrcnode.rb:20:in `each'
> from ./tlpsrcnode.rb:20:in `parse'
> from ./roottlpsrcnode.rb:53:in `startParse'
> from ./rbmfsplit:54
> ---8<---
>
> This appears to happen after 297 files are concurrently opened.

Sorry this statement is inaccurate.

297 files have previously been opened and after each one has been parsed
is immediately closed.

The call stack should be an indication of concurrently opened files? In
this case 5 files. (Its recursive).

i suppose the src of the parse function helps (probably disobeying all
coding conventions, sorry):
---8<---
def parse()
fileHandle = File.new @path
for line in fileHandle do

line =~ /^(.*?) (.*)/
cmd = $~[1]
arg = $~[2]

if cmd =~ /^#.*/ then
nil # comment
elsif cmd == "depend" then
newNode = TlpSrcNode.new @basePath + arg + ".tlpsrc", @rootNode
newNode.parse # and so recursion starts
@rootNode.incrementDeps
elsif cmd == "runpattern" then
@rootNode.addRunFile arg
elsif cmd == "docpattern" then
@rootNode.addDocFile arg
elsif cmd == "srcpattern" then
@rootNode.addSrcFile arg
elsif @nothingCmds.detect { |i| i == cmd } then
nil # OK
elsif @todoCmds.detect { |i| i == cmd } then
nil # XXX work out what these are
else
puts "*error: unknown cmd: '#{cmd}' arg='#{arg}'"
exit 1
end
end

fileHandle.close
end
---8<---
--
Posted via http://www.ruby-....

Gerardo Santana Gómez Garrido

3/6/2008 1:04:00 AM

Hello Edd, an OpenBSD user here too.

The error messages speaks of more than 3055 recursion levels. Not
every level is a call to parse, but if you take out the #new and #each
methods I'm sure there are a lot more than just 5 files opened at the
same time.

Are there that many dependencies? I don't think so. I haven't seen
those .tlpsrc files yet. I'll take a look.

On 3/5/08, Edd Barrett <vext01@gmail.com> wrote:
> Edd Barrett wrote:
> > ---8<---
> > ./tlpsrcnode.rb:19:in `initialize': Too many open files -
> > /home/tl/tl/Master/tlpkg/tlpsrc/wordcount.tlpsrc (Errno::EMFILE)
> > from ./tlpsrcnode.rb:19:in `new'
> > from ./tlpsrcnode.rb:19:in `parse'
> > from ./tlpsrcnode.rb:30:in `parse'
> > from ./tlpsrcnode.rb:20:in `each'
> > from ./tlpsrcnode.rb:20:in `parse'
> > from ./tlpsrcnode.rb:30:in `parse'
> > from ./tlpsrcnode.rb:20:in `each'
> > from ./tlpsrcnode.rb:20:in `parse'
> > ... 3055 levels...
> > from ./tlpsrcnode.rb:20:in `each'
> > from ./tlpsrcnode.rb:20:in `parse'
> > from ./roottlpsrcnode.rb:53:in `startParse'
> > from ./rbmfsplit:54
> > ---8<---
> >
> > This appears to happen after 297 files are concurrently opened.
>
> Sorry this statement is inaccurate.
>
> 297 files have previously been opened and after each one has been parsed
> is immediately closed.
>
> The call stack should be an indication of concurrently opened files? In
> this case 5 files. (Its recursive).
>
> i suppose the src of the parse function helps (probably disobeying all
> coding conventions, sorry):
> ---8<---
> def parse()
> fileHandle = File.new @path
> for line in fileHandle do
>
> line =~ /^(.*?) (.*)/
> cmd = $~[1]
> arg = $~[2]
>
> if cmd =~ /^#.*/ then
> nil # comment
> elsif cmd == "depend" then
> newNode = TlpSrcNode.new @basePath + arg + ".tlpsrc", @rootNode
> newNode.parse # and so recursion starts
> @rootNode.incrementDeps
> elsif cmd == "runpattern" then
> @rootNode.addRunFile arg
> elsif cmd == "docpattern" then
> @rootNode.addDocFile arg
> elsif cmd == "srcpattern" then
> @rootNode.addSrcFile arg
> elsif @nothingCmds.detect { |i| i == cmd } then
> nil # OK
> elsif @todoCmds.detect { |i| i == cmd } then
> nil # XXX work out what these are
> else
> puts "*error: unknown cmd: '#{cmd}' arg='#{arg}'"
> exit 1
> end
> end
>
> fileHandle.close
> end
> ---8<---
> --
> Posted via http://www.ruby-....
>
>

--
Gerardo Santana

Edd Barrett

3/6/2008 9:43:00 AM

Hello,

Thanks for all the replies so far. I will later try the suggestion to
read the file into memory straight away then close the file.

Gerardo Santana GÃ³mez Garrido wrote:
> Hello Edd, an OpenBSD user here too.

Great :) Ready for 4.3?

>
> The error messages speaks of more than 3055 recursion levels. Not
> every level is a call to parse, but if you take out the #new and #each
> methods I'm sure there are a lot more than just 5 files opened at the
> same time.

Ah i see. So each call frame is a "file" in ruby? I probably
misunderstood.

>
> Are there that many dependencies? I don't think so. I haven't seen
> those .tlpsrc files yet. I'll take a look.

http://tug.org/svn/texlive/trunk/Master/tlp...

More than 1000 last time I checked.

This means I will be soring a lot of strings. Is the GC in ruby
effecient, or should I be encouraging it Java style. System.gc() :P

Thanks

Edd
--
Posted via http://www.ruby-....

Gerardo Santana Gómez Garrido

3/6/2008 4:00:00 PM

Sorry for top posting, Gmail doesn't support my browser.

Indeed, I'm waiting for that OPENBSD_4_3_BASE tag ;-)

What I meant is that you are opening a file in every call to parse. If
you have many recursive calls to parse, you'll end up with many files
open.

As some others have suggested, you could transform recursion into iteration=

comp.lang.ruby

Too many open files on OpenBSD

Edd Barrett

Edd Barrett

Gerardo Santana Gómez Garrido

Edd Barrett

Gerardo Santana Gómez Garrido

x Login to ForumsZone