Ruby error “Too many open files” and tricky code blocks

I wrote a ruby script recently that was executing thousands of command lines using popen as follow:

cmdLineStr = "some stuffs to execute"
IO.popen(cmdLineSr).each do |l|
  puts "LOG: " + l.strip
end

I have been using this syntax for long, but this very script would randomly crash with the error message:

Too many open files

After digging a bit on Internet, I found a way to query the garbage collector for non-closed files in live variables at this link: link. The code is the following:

ObjectSpace.each_object(File) do |f|
  unless f.closed?
    WARNING "This file is still open: %s: %d\n" % [f.path, f.fileno]
  end
end

This is sexy code, but it failed to find any opened files in my case. I got back to Popen’s documentation (here) and re-read the following sentence:

If a block is given, Ruby will run the command as a child connected to Ruby with a pipe. Ruby’s end of the pipe will be passed as a parameter to the block. At the end of block, Ruby close the pipe and sets $?. In this case IO.popen returns the value of the block.

I realized that I was not using blocks in here although I intended to do so. Indeed, by “giving a block”, the documentation means

IO.popen(cmdLineSr) do |io|

while I was using :

IO.popen(cmdLineSr).each do |l|

which is a synonym for

io = IO.popen(cmdLineSr)
io.each do |l|

In other words, I was not closing the file and ended up with zombie processes. Both codes look very similar as they only differ by .each. But they are very different. Ruby is a powerful language but the syntax is so dense that sometimes bugs are not easy to spot.

Anyway, the correct code is:

cmdLineStr = "some stuffs to execute"
IO.popen(cmdLineSr).each do |io|
  io.each do |l|
    puts "LOG: " + l.strip
  end
end
Advertisements

1 thought on “Ruby error “Too many open files” and tricky code blocks

  1. Pingback: Notes on uploading images via sinatra in ruby | PlanB

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s