While developing a new project that utilizes node.js as the webserver and riak as the datastore, I encountered this error (here is a relevant dump):
InnoDB: Operating system error number 24 in a file operation.
InnoDB: Error number 24 means 'Too many open files'.
InnoDB: Check InnoDB website for details
My developer platform is a beefy mac with lots of ram that lets me run many virtual machines at the same time, but I digress. Thankfully, riak dumps operations to a log file in its log folder so basically all you need to do to get that info is to check your most recent log like so:
$cat < `ls -tr log/erlang.log.* | tail -n 1`
The problem has to do with the fact that the Mac (and other *nix's) limits the maximum number of files you can have open at any one time. Unfortunately, as my testing has shown, riak likes to hog files. Especially when using the innostore backend. More or less, as of this writing (riak v0.8), the innostore backend to riak will open 64 files for every bucket that you use. Even if there is no data in the bucket. A call to "http://host:port/riak/someBucket" will instruct innostore to create the 64 files if they do not exist and open them. Each file is about 100KB with no real data in them, aka empty, in case you were wondering.
So how do I keep testing while Basho figures out a way to fix this issue? By upping my max file limit, of course! First let us check what are current limits are:
$launchctl limit
Note that that limit is a user specific limit. The kernel has its own limit which is check like so:
$sysctl -A | grep kern.maxfiles
I got most of this information from krypted.com, which has the most cohesive description in one place that I have yet to find on this subject outside of kernel-developer mailing lists. I got the userland shell info from here. Now for the eagle eyed, those two pages may have some overlapping yet conflicting information. I employed the following method, which may not be best for all users, YMMV. Be warned.
Borrowing heavily from the aforementioned posts, I simply created an /etc/launchd.conf file like so:
$sudo touch /etc/launchd.conf
Then edit the launchd.conf file to include these lines:
limit maxproc 1024 2048
limit maxfiles 2048 4096
The two columns are a soft limit and a hard limit. The distinction being... eh whatever, google is your friend. But importantly, the soft limit must be a number smaller than the hard limit and the hard limit may be "unlimited". I also added a maxproc line in there for good measure while I was at it. Of note, after reboot, sysctl and launchctl command both return the new values. If I didn't have so many windows open right now I would do more testing about that, but eh, whatever.
[UPDATE 10MAR2010]
Got an email from Jon Meredith, a Basho employee stating the following:
You can control the maximum number of file descriptors that Embedded
InnoDB will use by setting open_files in the innostore configuration.
The default value compiled in is 300 which exceeds the OS X default of
256.In riak/etc/app.config add this to your innostore section{innostore, [
%% Other Inno settings
{open_files, 100} %% Limit to 100 open file handles
]}
So there you have it. A number of ways to work around the open files limit on macs.