Eric Hodel
12/16/2004 7:17:00 PM
On 16 Dec 2004, at 10:37, The Weeg wrote:
> I wrote a pretty simple web scraping tool, and i'm having it organize
> the results into folders based upon search queries. After six hundred
> or so of these folders are created, i get a bizarre, "too many files
> open" error(Errno::EMFILE).
>
> This is the line of code it crashes on:
> `mkdir #{foldername}`
This opens a file to recieve the shell's response, instead use
Dir.mkdir or FileUtils.
> And the only time I ever open and close files in my program is in
> these few lines, but the program isn't crashing on these lines:
>
> fileTempFile = File.new(filePath + "full_" + fileName +
> ".html","w")
> fileTempFile.puts(response)
> fileLinkText = File.new(filePath + "relevant_" + fileName +
> ".txt","w")
> fileLinkText.puts(link + "\n" +
> match.to_s.gsub(/<.*?>/,"").gsub(/&[a-zA-Z0-9#]*?;/,''))
> fileLinkText.close
> fileTempFile.close
You should be using blocks here, to ensure the file is really closed.
match = match.to_s
match = match.gsub(/<.*?>/, "")
match = match.gsub(/&[a-zA-Z0-9#]*?;/, "")
link_text = "#{link}\n#{match}"
File.open "#{file_path}full_#{file_name}.html", "w" do |temp_file|
temp_file.puts response
end
File.open "#{file_path}relevant_#{file_name}.txt", "w" do |link_file|
link_file.puts link_text
end
> As you can see, the files are being opened and closed almost
> immediately.
If an exception is raised, your File may not be closed if you don't use
the block form.
> Is there a different way I should be creating directories rather than
> through a shell command?
Dir.mkdir, and FileUtils has (I believe) mkpath as well.