What to do if you have an Internal Server Error 500 (Too many open files) ?

Jerome.Jargot's picture
Jerome.Jargot
Tuesday, June 4, 2019 - 00:00
Blog Categories: 
Troubleshooting

Component: Linux or java process | Version: N/A

Question

The failure occurs sporadically.

The two servers are active and the service is available.

All of a sudden, the load balancer fails to find an healthy bonita: Internal Server Error 500 is received in response to the HTTP requests.

These messages are generated in the log files:

2019-03-19 13:49:08.597 -0300 WARNING: org.bonitasoft.console.common.server.servlet.ResourceServlet /opt/BonitaBPMSubscription-7.5.4-Tomcat-7.0.76/server/temp/bonita_portal_108793@ProdServer01.fr/tenants/1/pages/custompage_validate/resources/font/opensans-light-webfont.woff (Too many open files)
...
2019-03-19 13:49:38.099 -0300 WARNING: org.apache.catalina.loader.WebappClassLoaderBase Failed to open JAR [null]
java.io.FileNotFoundException: /opt/BonitaBPMSubscription-7.5.4-Tomcat-7.0.76/server/webapps/assistance/WEB-INF/lib/bonita-client-7.5.4.jar (Too many open files)
    at java.util.zip.ZipFile.open(Native Method)
...
2019-03-19 13:49:39.070 -0300 SEVERE: org.apache.tomcat.util.net.JIoEndpoint Socket accept failed
java.net.SocketException: Too many open files (Accept failed)
    at java.net.PlainSocketImpl.socketAccept(Native Method)
...
2019-03-19 13:49:45.729 -0300 WARNING: org.bonitasoft.console.common.server.page.PageServlet Error while trying to display a page or resource for key taskInstance/Request For Assistance/1.0.4/Confirmation
java.lang.NullPointerException
    at org.bonitasoft.console.common.server.page.CustomPageService.ensurePageFolderIsPresent(CustomPageService.java:102)
    at org.bonitasoft.console.common.server.page.PageRenderer.ensurePageFolderIsPresent(PageRenderer.java:70)
...
2019-03-19 13:49:45.994 -0300 WARNING: org.bonitasoft.forms.server.HomepageServlet Error while loading the file BonitaConsole.html in theme portal
java.io.FileNotFoundException: /opt/BonitaBPMSubscription-7.5.4-Tomcat-7.0.76/server/temp/bonita_portal_108793@ProdServer01.fr/tenants/1/theme/portal/.lastupdate (Too many open files)
    at java.io.FileInputStream.open0(Native Method)
    at java.io.FileInputStream.open(FileInputStream.java:195)
...
2019-03-19 13:50:26.771 -0300 INFO: com.support.assistance.servlet.Dummy INTEGRATION/DUMMY: Failure to call Assistance webservice. General Fail on the First Call.
javax.xml.ws.WebServiceException: Failed to access the WSDL at: http://webservice.support.fr/web-sso/support?wsdl. It failed with: 
    Too many open files.
...
2019-03-19 13:51:08.202 -0300 INFO: com.support.assistance.servlet.Dummy INTEGRATION/DUMMY: Failure to call Assistance webservice. General Fail on the First Call.
com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: java.net.SocketException: Too many open files
...


Why do you have this error?

The messages are generated because the java process has reached the maximum number of file descriptors allowed.

The application server involved usually has many web applications deployed: I.E. the bonita webapp is not the only one deployed.

By default on Linux OS, a user process can open a maximum of 1024 files (fd or file descriptors).

Log in to the system with the same user used to start the application server. Display the soft and hard limits respectively:

# ulimit -Sa | grep open
open files                      (-n) 1024

# ulimit -Ha | grep open
open files                      (-n) 4096

The -S option can be used to change the SOFT limit, which can range from 0-{HARD}.

Resolution

This procedure desribes how to raise up the SOFT and HARD, number of open files limit for all users' processes (except root).

1. Activate the PAM module to limit resources

Edit all the /etc/pam.d/common-session\* files
Add the line below, if not already present:

session required pam_limits.so
2. Raise up the number of open files limits for processes for all users (except root)

Edit the /etc/security/limits.conf file
Add these 2 lines below, or modify their values if already present:

* soft nofile 10240
* hard nofile 20480
3. Reboot the server
4. Validate the changes

Log in to the system with the same user used to start the application server:

# ulimit -Sa | grep open
open files                      (-n) 10240
5. Start the application server






Notifications