XA Excpetion/ARJUNA016027 (how to solve this error)

Hi!

We are running Bonita 7.9.0 in a docker environment, it is on a different virtual machine than the database, so the containers are not on the same VM however, the two VM-s are on the same machine. This error feels really random, sometimes it happens in a certain process and sometimes the process runs fine, it can happen with any process. If this happens the process will fail with only the error im posting.

This is the error:

2021-01-20 00:00:55.246 +0000 WARN: com.arjuna.ats.jta ARJUNA016027: Local XARecoveryModule.xaRecovery got XA exception XAException.XAER_RMFAIL
com.mysql.cj.jdbc.MysqlXAException: Undetermined error occurred in the underlying Connection - check your data for consistency
at com.mysql.cj.jdbc.MysqlXAConnection.mapXAExceptionFromSQLException(MysqlXAConnection.java:347)
at com.mysql.cj.jdbc.MysqlXAConnection.recover(MysqlXAConnection.java:189)
at com.mysql.cj.jdbc.MysqlXAConnection.recover(MysqlXAConnection.java:117)
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoveryFirstPass(XARecoveryModule.java:634)
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass(XARecoveryModule.java:226)
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass(XARecoveryModule.java:171)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:770)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:382)
Caused by: java.sql.SQLNonTransientConnectionException: No operations allowed after connection closed.
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:73)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:73)
at com.mysql.cj.jdbc.ConnectionImpl.createStatement(ConnectionImpl.java:1094)
at com.mysql.cj.jdbc.ConnectionImpl.createStatement(ConnectionImpl.java:1084)
at com.mysql.cj.jdbc.MysqlXAConnection.recover(MysqlXAConnection.java:166)
... 6 more
Caused by: com.mysql.cj.exceptions.ConnectionIsClosedException: No operations allowed after connection closed.
at jdk.internal.reflect.GeneratedConstructorAccessor7266.newInstance(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
at com.mysql.cj.NativeSession.checkClosed(NativeSession.java:1263)
at com.mysql.cj.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:575)
at com.mysql.cj.jdbc.ConnectionImpl.createStatement(ConnectionImpl.java)
... 8 more
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: The last packet successfully received from the server was 130,010 milliseconds ago. The last packet sent successfully to the server was 130,010 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
at com.mysql.cj.protocol.a.NativeProtocol.readMessage(NativeProtocol.java:562)
at com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:732)
at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:671)
at com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:986)
at com.mysql.cj.protocol.a.NativeProtocol.sendQueryString(NativeProtocol.java:921)
at com.mysql.cj.NativeSession.execSQL(NativeSession.java:1154)
at com.mysql.cj.jdbc.StatementImpl.executeQuery(StatementImpl.java:1186)
at com.mysql.cj.jdbc.MysqlXAConnection.recover(MysqlXAConnection.java:168)
... 6 more
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
at com.mysql.cj.protocol.FullReadInputStream.readFully(FullReadInputStream.java:67)
at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:63)
at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:45)
at com.mysql.cj.protocol.a.TimeTrackingPacketReader.readHeader(TimeTrackingPacketReader.java:52)
at com.mysql.cj.protocol.a.TimeTrackingPacketReader.readHeader(TimeTrackingPacketReader.java:41)
at com.mysql.cj.protocol.a.MultiPacketReader.readHeader(MultiPacketReader.java:54)
at com.mysql.cj.protocol.a.MultiPacketReader.readHeader(MultiPacketReader.java:44)
at com.mysql.cj.protocol.a.NativeProtocol.readMessage(NativeProtocol.java:556)
... 13 more

On the database we get this error:

{"log":"2020-12-31T11:36:28.455246Z 175845949 [Note] Aborted connection 175845949 to db: 'businessdb' user: 'user' host: 'appserv1.company.hu' (Got timeout reading communication packets)\n","stream":"stderr","time":"2020-12-31T11:36:28.455362175Z"}
{"log":"2020-12-31T11:36:28.455252Z 175845948 [Note] Aborted connection 175845949 to db: 'bonitadb' user: 'user' host: 'appserv1.company.hu' (Got timeout reading communication packets)\n","stream":"stderr","time":"2020-12-31T11:36:28.455362175Z"}

What could cause this? In my envirement where every container is on one VM everything works fine, i never saw this error. I think maybe the problem is with the VM-s communication.