When the casper-node launches, it tries to set the maximum open files limit (nofile) for the process to 64000. With some systems, this limit will be larger than the default hard limit of 4096.
The node software uses file handles for both files and network connections. Since network connections are unpredictable, running out of file handles can stop critical file writes from occurring. Therefore, the default nofile limit needs to be increased.
With the casper-node-launcher running, you can see what the system allocated by finding the process ID (PID) for the casper-node with the following command.
pgrep "casper-node$"
If you do not get a value in return, you do not have the casper-node-launcher running correctly.
To find the current nofile (number of open files) hard limit, run prlimit with the PID from the previous command:
sudo prlimit -n -p <PID>
You can also embed both commands as shown here:
sudo prlimit -n -p $(pgrep "casper-node$")
If you receive prlimit: option requires an argument -- 'p', then pgrep "casper-node$" is not returning anything because the casper-node-launcher is no longer running.
Run the command below to set the nofile limit for an active process without restarting the casper-node-launcher and casper-node processes. Note that this setting is active only while the casper-node process runs. To make this setting permanent, update the limits.conf file instead.
sudo prlimit --nofile=64000 --pid=$(pgrep "casper-node$")`
Next, check that the prlimit has changed:
sudo prlimit -n -p $(pgrep "casper-node$")
It is possible to persist the nofile limit across server reboots, casper-node-launcher restarts, and protocol upgrades, by adding the nofile setting for the casper user in /etc/security/limits.conf.
Add the following row to the bottom of the /etc/security/limits.conf file:
casper hard nofile 64000
Afterward, log out of any shells to enable this change. Restarting the node should maintain the correct nofile setting.