On UNIX, you may get a “Too many open files” error if you are crawling several data sources simultaneously.
The relevant line in the error’s stack trace is the following:
Caused by: java.io.FileNotFoundException: /localdisk/jsmith/endeca/CAS/workspace/state/test_data_multiseeds/data/dictionary/seg0/c3a1.dat (Too many open files)
The error occurs because the operating system has reached the per-process limit for the number of files the process can have open at once.
To resolve this problem, you can increase the number of file handles available. For more information about how to increase the number of available file handles, refer to the documentation for your operating system.
Note
There is no single recommended range of file handles values that will fit all situations. File/socket requirements can depend on a number of metrics, such as processes managed, nodes, files transferred, and system status queries. Therefore, determining a new limit experimentally, through trial and error, is the simplest resolution.
