What to do when there are XA Transaction Timeout?

Jerome.Jargot's picture
Jerome.Jargot
Blog Categories: 

Component: Bonita | Version: 7.9

What's wrong?

The messages in the log files are appended at the end. We can see some of these keywords:

  • ARJUNA016102: The transaction is not active!
  • HibernateException: Logical connection is closed
  • HibernateException: Current transaction is not in progress
  • ARJUNA012095: Abort of action id 0:...e invoked while multiple threads active within it

Answer

These messages in the log files show that there are XA transactions which time out.

XA Transaction

A XA transaction is a distributed transaction implementing the two-phase commit protocol (2PC, tupac): a type of atomic commitment protocol (ACP). It enables a transaction over more than one database. Note that a Transaction Manager (TM) at client side, partially manages the life cycle of a distributed transaction: it communicates with the Ressource Manager (RM) at database server side.

Explanation

A XA transaction can time out when:

  1. Database server: transactions are waiting for locks, when there are deadlocks...
  2. The transaction manager is using its defaults
  3. Sizing
    • There are shortage of connection in the pools associated to datasources.
  4. There are broken TCP/IP connections
  5. Tables' size: there is big data so that the transaction duration increased
  6. There are performance issues
    • A client application calls the Bonita engine API or the Bonita REST API which triggers resources consuming searches
    • The users are using the Bonita portal so that many resources consuming searches (via the Bonita REST API)
    • There are other processes which consuming resources on the server hosting the Tomcat
    • either an environmental problem impacts the CPU or Disk I/O
    • or an extra Garbage Collector's threads activity, triggered by a java memory shortage (-Xmx), leaves nothing to java application's threads.
  7. The current XA transaction timeout (TM configuration) limit is too small

1. Database server

Bonita requires specific database server settings.

If not implemented, there could be deadlocks, especially when MS Sql server is used.

Check that the settings listed in the Database creation and settings to work with Bonita documentation page have been implemented.

2. Simple TM configuration check with Tomcat

The preconfigured Tomcat with the Bonita product (downloadable from our site) is using Narayana: a transaction manager (TM) that was integrated into Tomcat in order to enable the use of the XA transactions.

Make sure that this option below is passed to the java command used to run the Tomcat, otherwise all the defaults will apply:

-Dcom.arjuna.ats.arjuna.common.propertiesFile=/a/path/to/jbossts-properties.xml

Make sure that the jbossts-properties.xml file exists.

3. Sizing

Analyze the configuration and better configure the connections' pools and threads' pools ; see the Sizing of the engine with examples.

Make sure that there will never be shortage of connections in the pools.

4. How to cope with broken TCP/IP connections

There is a page dedicated to this particular situation: Error: Bonita server not responding after an inactivity period

When the TCP/IP connections are broken: as there are not closed, the Tomcat's pools may provide to the http and bonita threads such broken connections. The threads send SQL queries and wait endlessly to an answer, until the XA transaction times out.

The pools of connections and the jdbc drivers can be configured to be more robust.

When BonitaSubscription-2021.1-0811/start-bonita.sh (.bat) script is used to start the Tomcat, edit BonitaSubscription-2021.1-0811/setup/tomcat-templates/bonita.xml file.

These first settings, will make the idle (unused) connections being droped from the pool. Make sure these properties below are used with bonitaDS, bonitaSequenceManagerDS, BusinessDataDS, and NotManagedBizDataDS resources

testWhileIdle="false"
softMinEvictableIdleTimeMillis="-1"
timeBetweenEvictionRunsMillis="60000"
minEvictableIdleTimeMillis="900000" 

These other settings will make the jdbc driver create the TCP/IP sockets with keep alive activated. Make sure this property below is used with both RawBonitaDS and RawBusinessDataDS resources.

tcpKeepAlive="true" 

On linux system, use the command-line below to display the settings:

$ sudo sysctl net.ipv4.tcp_keepalive_intvl net.ipv4.tcp_keepalive_probes net.ipv4.tcp_keepalive_time

It is wise to reduce the default value of the net.ipv4.tcp_keepalive_time variable to 600.

"Keep alive mechanism" will make the system find the broken connections, so that the pools will drop them in turn.

5. Tables' size

Analyze the tables' sizes used with the engine and plan to delete the oldest data, using the API: better do that with 7.7.5 version or greater.

In particular, you may want to delete archived process instances with the API call below:

6. Performance issue investigation

Monitor both the CPU and the disks I/O utilization, plus the CPU consumption of the java process running the Tomcat.

If there are performance problems when there are the XA transactions time out, fix them.

If the Java itself is consuming all the CPU:

  • if there is some RAM left (available memory), raise up the -Xmx value in order to provide more memory to the java process
  • when Business Data Model (BDM) is used, with multiple objects linked with EAGER type (or Always load related objects) relationship, then change the relationship to LAZY (Only load related objects when needed). The forms and the processes using the BDM need to be modified as well.

7. Raise up the XA default transaction timeout

At production sites, values from 300 to 2400 seconds are used.

Narayana is the transaction manager (TM) used with Bonita.

It enables the use of XA transactions.

Tomcat

Edit the BonitaSubscription-7.9-Tomcat-8.5.34/server/conf/jbossts-properties.xml.

Raise up the value, in seconds, of the xml element listed below:

<entry key="com.arjuna.ats.arjuna.coordinator.defaultTimeout">600</entry>

Restart the Tomcat to activate the changes.

Wildfly

Edit the standalone.xml file Add the coordinator-environment element in the subsystem section, as shown below:

<subsystem xmlns="urn:jboss:domain:transactions:3.0">
    <core-environment node-identifier="bonita">
        <process-id>
            <uuid />
        </process-id>
    </core-environment>
    <recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager" />
    <coordinator-environment default-timeout="600" />
</subsystem>        

Restart the Wildfly to activate the changes.

standalone.xml file location

start-bonita script

When the BonitaSubscription-7.9.0-wildfly-10.1.0.Final/start-bonita.sh (start-bonita.bat) script is used to start the application server, then edit the BonitaSubscription-7.9.0-wildfly-10.1.0.Final/setup/wildfly-templates/standalone.xml file.

Other situations

When Wildfly is started with the standard BonitaSubscription-7.9.0-wildfly-10.1.0.Final/server/bin/standalone.sh (standalone.bat) or as a Windows service, edit the BonitaSubscription-7.9.0-wildfly-10.1.0.Final/server/standalone/configuration/standalone.xml file.


Raw messages from the log files

Tomcat

There are below other examples from the server/logs/bonita*log file:

2018-08-17 16:05:37.780 +0200 SEVERE: org.bonitasoft.web.toolkit.server.servlet.ToolkitHttpServlet org.bonitasoft.web.toolkit.server.servlet.ToolkitHttpServlet catchAllExceptions SEVERE: org.bonitasoft.engine.exception.SearchException: USERNAME=kiki | org.bonitasoft.engine.persistence.SBonitaReadException: org.bonitasoft.engine.services.SPersistenceException: org.hibernate.HibernateException: Current transaction is not in progress org.bonitasoft.web.toolkit.client.common.exception.api.APIException: org.bonitasoft.engine.exception.SearchException: USERNAME=kiki | org.bonitasoft.engine.persistence.SBonitaReadException: org.bonitasoft.engine.services.SPersistenceException: org.hibernate.HibernateException: Current transaction is not in progress
... Caused by: org.bonitasoft.engine.services.SPersistenceException: org.hibernate.HibernateException: Current transaction is not in progress
... Caused by: org.hibernate.HibernateException: Current transaction is not in progress ... 
2018-09-10 07:40:22,470 SEVERE [org.restlet.Component.BonitaSPRestletApplication] (default task-29) Exception or error caught in server resource: org.bonitasoft.engine.exception.BonitaRuntimeException: USERNAME=kiki | org.bonitasoft.engine.persistence.SRetryableException: javax.persistence.PersistenceException: org.hibernate.HibernateException: Logical connection is closed at org.bonitasoft.engine.api.impl.ServerAPIImpl.wrapThrowable(ServerAPIImpl.java:148)
... Caused by: javax.persistence.PersistenceException: org.hibernate.HibernateException: Logical connection is closed ... at org.bonitasoft.engine.business.data.impl.JPABusinessDataRepositoryImpl.findList(JPABusinessDataRepositoryImpl.java:318) at org.bonitasoft.engine.business.data.impl.JPABusinessDataRepositoryImpl.findListByNamedQuery(JPABusinessDataRepositoryImpl.java:295) ... 149 more Caused by: org.hibernate.HibernateException: Logical connection is closed at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:164) ... 
2018-09-10 10:01:20,153 SEVERE [org.restlet.Component.BonitaSPRestletApplication] (default task-4) Exception or error caught in server resource: org.bonitasoft.engine.exception.BonitaRuntimeException: USERNAME=kiki | org.bonitasoft.engine.persistence.SRetryableException: javax.persistence.PersistenceException: org.hibernate.HibernateException: Logical connection is closed
... Caused by: org.bonitasoft.engine.persistence.SRetryableException: javax.persistence.PersistenceException: org.hibernate.HibernateException: Logical connection is closed at org.bonitasoft.engine.business.data.impl.JPABusinessDataRepositoryImpl.findById(JPABusinessDataRepositoryImpl.java:193)
... Caused by: javax.persistence.PersistenceException: org.hibernate.HibernateException: Logical connection is closed
... Caused by: org.hibernate.HibernateException: Logical connection is closed at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:164)
...
2020-09-28 00:08:00.537 -0400 FINE: org.bonitasoft.engine.transaction.JTATransactionServiceImpl THREAD_ID=106 | HOSTNAME=quantum-srv1 | Setting rollbackOnly on current transaction because callable 'org.bonitasoft.engine.execution.work.ExecuteConnectorWork$EvaluateParameterAndGetConnectorInstance@7570c2d1' has thrown an exception: javax.persistence.PersistenceException: org.hibernate.HibernateException: Logical connection is closed
org.bonitasoft.engine.commons.exceptions.SRetryableException: javax.persistence.PersistenceException: org.hibernate.HibernateException: Logical connection is closed
    at org.bonitasoft.engine.business.data.impl.JPABusinessDataRepositoryImpl.findById(JPABusinessDataRepositoryImpl.java:199)
    at org.bonitasoft.engine.business.data.BusinessDataRetriever.getSimpleBusinessData(BusinessDataRetriever.java:53)
    at org.bonitasoft.engine.business.data.BusinessDataRetriever.getBusinessData(BusinessDataRetriever.java:95)
    at org.bonitasoft.engine.expression.BusinessDataExpressionExecutorStrategy.evaluate(BusinessDataExpressionExecutorStrategy.java:68)
    at org.bonitasoft.engine.expression.BusinessDataExpressionExecutorStrategy.evaluate(BusinessDataExpressionExecutorStrategy.java:84)
    at org.bonitasoft.engine.expression.impl.ExpressionServiceImpl.evaluate(ExpressionServiceImpl.java:154)
    at org.bonitasoft.engine.core.expression.control.api.impl.ExpressionResolverServiceImpl.evaluateExpressionsOfKind(ExpressionResolverServiceImpl.java:225)
    at org.bonitasoft.engine.core.expression.control.api.impl.ExpressionResolverServiceImpl.evaluateAllExpressionsWithNoDependencies(ExpressionResolverServiceImpl.java:182)
    at org.bonitasoft.engine.core.expression.control.api.impl.ExpressionResolverServiceImpl.evaluateExpressionsFlatten(ExpressionResolverServiceImpl.java:115)
    at org.bonitasoft.engine.core.expression.control.api.impl.ExpressionResolverServiceImpl.evaluate(ExpressionResolverServiceImpl.java:83)
    at org.bonitasoft.engine.core.connector.impl.ConnectorServiceImpl.evaluateInputParameters(ConnectorServiceImpl.java:314)
    at org.bonitasoft.engine.connector.ConnectorServiceDecorator.evaluateInputParameters(ConnectorServiceDecorator.java:103)
    at org.bonitasoft.engine.execution.work.ExecuteConnectorWork$EvaluateParameterAndGetConnectorInstance.call(ExecuteConnectorWork.java:257)
    at org.bonitasoft.engine.execution.work.ExecuteConnectorWork$EvaluateParameterAndGetConnectorInstance.call(ExecuteConnectorWork.java:215)
    at org.bonitasoft.engine.transaction.JTATransactionServiceImpl.executeInTransaction(JTATransactionServiceImpl.java:274)
    at org.bonitasoft.engine.execution.work.ExecuteConnectorWork.work(ExecuteConnectorWork.java:148)
    at org.bonitasoft.engine.execution.work.failurewrapping.TxInHandleFailureWrappingWork.work(TxInHandleFailureWrappingWork.java:41)
    at org.bonitasoft.engine.execution.work.failurewrapping.TxInHandleFailureWrappingWork.work(TxInHandleFailureWrappingWork.java:41)
    at org.bonitasoft.engine.execution.work.failurewrapping.TxInHandleFailureWrappingWork.work(TxInHandleFailureWrappingWork.java:41)
    at org.bonitasoft.engine.execution.work.failurewrapping.TxInHandleFailureWrappingWork.work(TxInHandleFailureWrappingWork.java:41)
    at org.bonitasoft.engine.execution.work.InSessionBonitaWork.work(InSessionBonitaWork.java:59)
    at org.bonitasoft.engine.work.BonitaThreadPoolExecutor.lambda$submit$1(BonitaThreadPoolExecutor.java:132)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: javax.persistence.PersistenceException: org.hibernate.HibernateException: Logical connection is closed
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1763)
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1694)
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.find(AbstractEntityManagerImpl.java:1141)
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.find(AbstractEntityManagerImpl.java:1068)
    at org.bonitasoft.engine.business.data.impl.JPABusinessDataRepositoryImpl.findById(JPABusinessDataRepositoryImpl.java:196)
    ... 24 more
Caused by: org.hibernate.HibernateException: Logical connection is closed
    at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:164)
    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.connection(StatementPreparerImpl.java:63)
    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:162)
    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:186)
    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:160)
    at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.prepareQueryStatement(AbstractLoadPlanBasedLoader.java:257)
    at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeQueryStatement(AbstractLoadPlanBasedLoader.java:201)
    at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:137)
    at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:102)
    at org.hibernate.loader.entity.plan.AbstractLoadPlanBasedEntityLoader.load(AbstractLoadPlanBasedEntityLoader.java:186)
    at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:4126)
    at org.hibernate.event.internal.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:503)
    at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:468)
    at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:213)
    at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:275)
    at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:151)
    at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1106)
    at org.hibernate.internal.SessionImpl.access$2000(SessionImpl.java:176)
    at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.load(SessionImpl.java:2587)
    at org.hibernate.internal.SessionImpl.get(SessionImpl.java:991)
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.find(AbstractEntityManagerImpl.java:1110)
    ... 26 more

2020-09-28 00:08:00.537 -0400 FINEST: org.bonitasoft.engine.transaction.JTATransactionServiceImpl THREAD_ID=106 | HOSTNAME=quantum-srv1 | Completing transaction in thread 106 TransactionImple < ac, BasicAction: 0:ffff0afe3223:e16e:5f6b86b0:1a3e76 status: ActionStatus.ABORT_ONLY >
2020-09-28 00:08:00.537 -0400 FINEST: org.bonitasoft.engine.transaction.JTATransactionServiceImpl THREAD_ID=106 | HOSTNAME=quantum-srv1 | Rolling back transaction in thread 106 TransactionImple < ac, BasicAction: 0:ffff0afe3223:e16e:5f6b86b0:1a3e76 status: ActionStatus.ABORT_ONLY >
2020-09-28 00:08:00.538 -0400 FINE: com.bonitasoft.engine.work.DefaultExceptionRetryabilityEvaluator THREAD_ID=106 | HOSTNAME=quantum-srv1 | TENANT_ID=108 | Will retry, matching exception found: Exceptions:
 ↳org.bonitasoft.engine.commons.exceptions.SRetryableException: javax.persistence.PersistenceException: org.hibernate.HibernateException: Logical connection is closed ☚
 ↳javax.persistence.PersistenceException
 ↳org.hibernate.HibernateException

2020-09-28 00:08:00.538 -0400 WARNING: com.bonitasoft.engine.work.RetryingWorkExecutorService THREAD_ID=106 | HOSTNAME=quantum-srv1 | TENANT_ID=108 | Work WorkDescriptor[EXECUTE_ACTIVITY_CONNECTOR,parameters={processDefinitionId=4690108313069717636, processInstanceId=35238, connectorInstanceId=1987488, flowNodeDefinitionId=6178904343737415026, connectorDefinitionName=GetProductName, flowNodeInstanceId=3193478}] encountered an exception but it can be retried. Will retry. Attempt 1 on 10 with a delay of 1000 ms
2020-09-28 00:08:00.538 -0400 FINE: com.bonitasoft.engine.work.RetryingWorkExecutorService THREAD_ID=106 | HOSTNAME=quantum-srv1 | TENANT_ID=108 | Exception encountered during retry
org.bonitasoft.engine.commons.exceptions.SRetryableException: javax.persistence.PersistenceException: org.hibernate.HibernateException: Logical connection is closed
    at org.bonitasoft.engine.business.data.impl.JPABusinessDataRepositoryImpl.findById(JPABusinessDataRepositoryImpl.java:199)
    at org.bonitasoft.engine.business.data.BusinessDataRetriever.getSimpleBusinessData(BusinessDataRetriever.java:53)
    at org.bonitasoft.engine.business.data.BusinessDataRetriever.getBusinessData(BusinessDataRetriever.java:95)
    at org.bonitasoft.engine.expression.BusinessDataExpressionExecutorStrategy.evaluate(BusinessDataExpressionExecutorStrategy.java:68)
    at org.bonitasoft.engine.expression.BusinessDataExpressionExecutorStrategy.evaluate(BusinessDataExpressionExecutorStrategy.java:84)
    at org.bonitasoft.engine.expression.impl.ExpressionServiceImpl.evaluate(ExpressionServiceImpl.java:154)
    at org.bonitasoft.engine.core.expression.control.api.impl.ExpressionResolverServiceImpl.evaluateExpressionsOfKind(ExpressionResolverServiceImpl.java:225)
    at org.bonitasoft.engine.core.expression.control.api.impl.ExpressionResolverServiceImpl.evaluateAllExpressionsWithNoDependencies(ExpressionResolverServiceImpl.java:182)
    at org.bonitasoft.engine.core.expression.control.api.impl.ExpressionResolverServiceImpl.evaluateExpressionsFlatten(ExpressionResolverServiceImpl.java:115)
    at org.bonitasoft.engine.core.expression.control.api.impl.ExpressionResolverServiceImpl.evaluate(ExpressionResolverServiceImpl.java:83)
    at org.bonitasoft.engine.core.connector.impl.ConnectorServiceImpl.evaluateInputParameters(ConnectorServiceImpl.java:314)
    at org.bonitasoft.engine.connector.ConnectorServiceDecorator.evaluateInputParameters(ConnectorServiceDecorator.java:103)
    at org.bonitasoft.engine.execution.work.ExecuteConnectorWork$EvaluateParameterAndGetConnectorInstance.call(ExecuteConnectorWork.java:257)
    at org.bonitasoft.engine.execution.work.ExecuteConnectorWork$EvaluateParameterAndGetConnectorInstance.call(ExecuteConnectorWork.java:215)
    at org.bonitasoft.engine.transaction.JTATransactionServiceImpl.executeInTransaction(JTATransactionServiceImpl.java:274)
    at org.bonitasoft.engine.execution.work.ExecuteConnectorWork.work(ExecuteConnectorWork.java:148)
    at org.bonitasoft.engine.execution.work.failurewrapping.TxInHandleFailureWrappingWork.work(TxInHandleFailureWrappingWork.java:41)
    at org.bonitasoft.engine.execution.work.failurewrapping.TxInHandleFailureWrappingWork.work(TxInHandleFailureWrappingWork.java:41)
    at org.bonitasoft.engine.execution.work.failurewrapping.TxInHandleFailureWrappingWork.work(TxInHandleFailureWrappingWork.java:41)
    at org.bonitasoft.engine.execution.work.failurewrapping.TxInHandleFailureWrappingWork.work(TxInHandleFailureWrappingWork.java:41)
    at org.bonitasoft.engine.execution.work.InSessionBonitaWork.work(InSessionBonitaWork.java:59)
    at org.bonitasoft.engine.work.BonitaThreadPoolExecutor.lambda$submit$1(BonitaThreadPoolExecutor.java:132)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: javax.persistence.PersistenceException: org.hibernate.HibernateException: Logical connection is closed
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1763)
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1694)
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.find(AbstractEntityManagerImpl.java:1141)
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.find(AbstractEntityManagerImpl.java:1068)
    at org.bonitasoft.engine.business.data.impl.JPABusinessDataRepositoryImpl.findById(JPABusinessDataRepositoryImpl.java:196)
    ... 24 more
Caused by: org.hibernate.HibernateException: Logical connection is closed
    at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:164)
    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.connection(StatementPreparerImpl.java:63)
    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:162)
    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:186)
    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:160)
    at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.prepareQueryStatement(AbstractLoadPlanBasedLoader.java:257)
    at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeQueryStatement(AbstractLoadPlanBasedLoader.java:201)
    at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:137)
    at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:102)
    at org.hibernate.loader.entity.plan.AbstractLoadPlanBasedEntityLoader.load(AbstractLoadPlanBasedEntityLoader.java:186)
    at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:4126)
    at org.hibernate.event.internal.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:503)
    at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:468)
    at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:213)
    at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:275)
    at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:151)
    at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1106)
    at org.hibernate.internal.SessionImpl.access$2000(SessionImpl.java:176)
    at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.load(SessionImpl.java:2587)
    at org.hibernate.internal.SessionImpl.get(SessionImpl.java:991)
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.find(AbstractEntityManagerImpl.java:1110)
    ... 26 more

2020-10-01 16:43:58.026 -0400 WARN: com.arjuna.ats.arjuna ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0afe3223:d3cd:5f763ec9:1ffa in state  RUN
2020-10-01 16:43:58.026 -0400 WARN: com.arjuna.ats.arjuna ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0afe3223:d3cd:5f763ec9:1ffe in state  RUN
2020-10-01 16:43:58.026 -0400 WARN: com.arjuna.ats.arjuna ARJUNA012095: Abort of action id 0:ffff0afe3223:d3cd:5f763ec9:1ffa invoked while multiple threads active within it.
2020-10-01 16:43:58.026 -0400 WARN: com.arjuna.ats.arjuna ARJUNA012381: Action id 0:ffff0afe3223:d3cd:5f763ec9:1ffa completed with multiple threads - thread Bonita-Worker-108-04 was in progress with sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.FutureTask.awaitDone(FutureTask.java:426)
java.util.concurrent.FutureTask.get(FutureTask.java:204)
org.bonitasoft.engine.classloader.ClassLoaderUpdater.execute(ClassLoaderUpdater.java:64)
org.bonitasoft.engine.classloader.ClassLoaderUpdater.initializeClassLoader(ClassLoaderUpdater.java:47)
org.bonitasoft.engine.classloader.ClassLoaderServiceImpl.getLocalClassLoader(ClassLoaderServiceImpl.java:150)
org.bonitasoft.engine.classloader.ClassLoaderServiceImpl.getLocalClassLoader(ClassLoaderServiceImpl.java:140)
org.bonitasoft.engine.classloader.ClassLoaderServiceImpl.getLocalClassLoader(ClassLoaderServiceImpl.java:55)
org.bonitasoft.engine.execution.FlowNodeExecutorImpl.stepForward(FlowNodeExecutorImpl.java:137)
org.bonitasoft.engine.execution.FlowNodeExecutorImpl.executeFlowNode(FlowNodeExecutorImpl.java:301)
org.bonitasoft.engine.execution.work.ExecuteFlowNodeWork.work(ExecuteFlowNodeWork.java:73)
org.bonitasoft.engine.execution.work.TxBonitaWork.lambda$work$0(TxBonitaWork.java:42)
org.bonitasoft.engine.execution.work.TxBonitaWork$$Lambda$150/1691360812.call(Unknown Source)
org.bonitasoft.engine.transaction.JTATransactionServiceImpl.executeInTransaction(JTATransactionServiceImpl.java:274)
org.bonitasoft.engine.execution.work.TxBonitaWork.work(TxBonitaWork.java:42)
org.bonitasoft.engine.execution.work.LockProcessInstanceWork.work(LockProcessInstanceWork.java:63)
org.bonitasoft.engine.execution.work.failurewrapping.TxInHandleFailureWrappingWork.work(TxInHandleFailureWrappingWork.java:41)
org.bonitasoft.engine.execution.work.failurewrapping.TxInHandleFailureWrappingWork.work(TxInHandleFailureWrappingWork.java:41)
org.bonitasoft.engine.execution.work.failurewrapping.TxInHandleFailureWrappingWork.work(TxInHandleFailureWrappingWork.java:41)
org.bonitasoft.engine.execution.work.InSessionBonitaWork.work(InSessionBonitaWork.java:59)
org.bonitasoft.engine.work.BonitaThreadPoolExecutor.lambda$submit$1(BonitaThreadPoolExecutor.java:132)
org.bonitasoft.engine.work.BonitaThreadPoolExecutor$$Lambda$148/440663460.run(Unknown Source)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
-10-01 16:43:58.026 -0400 WARN: com.arjuna.ats.arjuna ARJUNA012108: CheckedAction::check - atomic action 0:ffff0afe3223:d3cd:5f763ec9:1ffa aborting with 1 threads active!
2020-10-01 16:43:58.026 -0400 WARN: com.arjuna.ats.arjuna ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff0afe3223:d3cd:5f763ec9:1ffa
2020-10-01 16:43:58.026 -0400 WARN: com.arjuna.ats.arjuna ARJUNA012095: Abort of action id 0:ffff0afe3223:d3cd:5f763ec9:1ffe invoked while multiple threads active within it.
2020-10-01 16:43:58.026 -0400 WARN: com.arjuna.ats.arjuna ARJUNA012381: Action id 0:ffff0afe3223:d3cd:5f763ec9:1ffe completed with multiple threads - thread BonitaTaskExecutor was in progress with java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
java.net.SocketInputStream.read(SocketInputStream.java:171)
java.net.SocketInputStream.read(SocketInputStream.java:141)
org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:214)
org.postgresql.core.PGStream.receive(PGStream.java:462)
org.postgresql.core.PGStream.receiveTupleV3(PGStream.java:422)
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2146)
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308)
org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:143)
org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:106)
sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.postgresql.ds.PGPooledConnection$StatementHandler.invoke(PGPooledConnection.java:428)
com.sun.proxy.$Proxy11.executeQuery(Unknown Source)
org.apache.tomcat.dbcp.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
org.apache.tomcat.dbcp.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:82)
org.hibernate.loader.Loader.getResultSet(Loader.java:2066)
org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1863)
org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1839)
org.hibernate.loader.Loader.doQuery(Loader.java:910)
org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:355)
org.hibernate.loader.Loader.doList(Loader.java:2554)
org.hibernate.loader.Loader.doList(Loader.java:2540)
org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2370)
org.hibernate.loader.Loader.list(Loader.java:2365)
org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:497)
org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:387)
org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:236)
org.hibernate.internal.SessionImpl.list(SessionImpl.java:1300)
org.hibernate.internal.QueryImpl.list(QueryImpl.java:103)
org.hibernate.internal.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:966)
org.bonitasoft.engine.persistence.AbstractHibernatePersistenceService.selectOne(AbstractHibernatePersistenceService.java:397)
org.bonitasoft.engine.persistence.AbstractHibernatePersistenceService.selectOne(AbstractHibernatePersistenceService.java:320)
org.bonitasoft.engine.dependency.impl.TenantDependencyService.getDependencyContentOnly(TenantDependencyService.java:232)
org.bonitasoft.engine.dependency.impl.AbstractDependencyService.lambda$getDependenciesResources$0(AbstractDependencyService.java:70)
org.bonitasoft.engine.dependency.impl.AbstractDependencyService$$Lambda$128/1896934036.apply(Unknown Source)
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
org.bonitasoft.engine.classloader.BonitaClassLoader.addResources(BonitaClassLoader.java:101)
org.bonitasoft.engine.classloader.BonitaClassLoader.<init>(BonitaClassLoader.java:70)
org.bonitasoft.engine.classloader.ClassLoaderUpdater.doInitializeClassLoader(ClassLoaderUpdater.java:56)
org.bonitasoft.engine.classloader.ClassLoaderUpdater.lambda$initializeClassLoader$1(ClassLoaderUpdater.java:48)
org.bonitasoft.engine.classloader.ClassLoaderUpdater$$Lambda$125/2010267676.call(Unknown Source)
org.bonitasoft.engine.transaction.JTATransactionServiceImpl.executeInTransaction(JTATransactionServiceImpl.java:274)
org.bonitasoft.engine.classloader.ClassLoaderUpdater.lambda$inTransaction$3(ClassLoaderUpdater.java:85)
org.bonitasoft.engine.classloader.ClassLoaderUpdater$$Lambda$126/1505813797.call(Unknown Source)
org.bonitasoft.engine.classloader.ClassLoaderUpdater.lambda$inSession$2(ClassLoaderUpdater.java:77)
org.bonitasoft.engine.classloader.ClassLoaderUpdater$$Lambda$127/1081293638.call(Unknown Source)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
2021-10-01 16:43:58.026 -0400 WARN: com.arjuna.ats.arjuna ARJUNA012108: CheckedAction::check - atomic action 0:ffff0afe3223:d3cd:5f763ec9:1ffe aborting with 1 threads active!
2020-10-01 16:43:58.526 -0400 WARN: com.arjuna.ats.arjuna ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0afe3223:d3cd:5f763ec9:1ffe in state  CANCEL
2020-10-01 16:43:58.526 -0400 WARN: com.arjuna.ats.arjuna ARJUNA012378: ReaperElement appears to be wedged: org.postgresql.core.QueryExecutorBase.getTransactionState(QueryExecutorBase.java:252)
org.postgresql.jdbc.PgConnection.rollback(PgConnection.java:798)
org.postgresql.xa.PGXAConnection.rollback(PGXAConnection.java:449)
com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelAbort(XAResourceRecord.java:362)
com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3023)
com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3002)
com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1674)
com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:124)
com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:215)
com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:381)
com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:78)

2021-08-27 13:46:15.421 +0000 WARN (Transaction Reaper) com.arjuna.ats.arjuna ARJUNA012117: TransactionReaper::check timeout for TX 0:ffffac140ec7:8051:611e75cd:dddc in state  RUN
2021-08-27 13:46:15.431 +0000 WARN (Transaction Reaper Worker 0) com.arjuna.ats.arjuna ARJUNA012095: Abort of action id 0:ffffac140ec7:8051:611e75cd:dddc invoked while multiple threads active within it.
2021-08-27 13:46:15.433 +0000 WARN (Transaction Reaper Worker 0) com.arjuna.ats.arjuna ARJUNA012381: Action id 0:ffffac140ec7:8051:611e75cd:dddc completed with multiple threads - thread http-nio-8080-exec-30 was in progress with java.base@11.0.11/java.net.SocketInputStream.socketRead0(Native Method)
java.base@11.0.11/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
java.base@11.0.11/java.net.SocketInputStream.read(SocketInputStream.java:168)
java.base@11.0.11/java.net.SocketInputStream.read(SocketInputStream.java:140)
org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:140)
org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:109)
org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:67)
org.postgresql.core.PGStream.receiveChar(PGStream.java:306)
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1952)
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308)
org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:143)
org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:106)
org.apache.tomcat.dbcp.dbcp2.PoolableConnection.validate(PoolableConnection.java:302)
org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:656)
org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.java:674)
org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:476)
org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:353)
org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:733)
org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator$ConnectionProviderJdbcConnectionAccess.obtainConnection(JdbcEnvironmentInitiator.java:180)
org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:68)
org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:35)
org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:101)
org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:263)
org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:237)
org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214)
org.hibernate.id.factory.internal.DefaultIdentifierGeneratorFactory.injectServices(DefaultIdentifierGeneratorFactory.java:152)
org.hibernate.service.internal.AbstractServiceRegistryImpl.injectDependencies(AbstractServiceRegistryImpl.java:286)
org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:243)
org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214)
org.hibernate.boot.internal.InFlightMetadataCollectorImpl.<init>(InFlightMetadataCollectorImpl.java:175)
org.hibernate.boot.model.process.spi.MetadataBuildingProcess.complete(MetadataBuildingProcess.java:118)
org.hibernate.boot.model.process.spi.MetadataBuildingProcess.build(MetadataBuildingProcess.java:83)
org.hibernate.boot.internal.MetadataBuilderImpl.build(MetadataBuilderImpl.java:473)
org.hibernate.boot.internal.MetadataBuilderImpl.build(MetadataBuilderImpl.java:84)
org.hibernate.boot.MetadataSources.buildMetadata(MetadataSources.java:185)
org.bonitasoft.engine.business.data.impl.SchemaManagerUpdate.buildConfiguration(SchemaManagerUpdate.java:62)
org.bonitasoft.engine.business.data.impl.SchemaManagerUpdate.update(SchemaManagerUpdate.java:94)
org.bonitasoft.engine.business.data.impl.BusinessDataModelRepositoryImpl.update(BusinessDataModelRepositoryImpl.java:186)
org.bonitasoft.engine.business.data.impl.BusinessDataModelRepositoryImpl.createAndDeployServerBDMJar(BusinessDataModelRepositoryImpl.java:178)
org.bonitasoft.engine.business.data.impl.BusinessDataModelRepositoryImpl.install(BusinessDataModelRepositoryImpl.java:163)
org.bonitasoft.engine.api.impl.TenantAdministrationAPIImpl.installBusinessDataModel(TenantAdministrationAPIImpl.java:156)
jdk.internal.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
java.base@11.0.11/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.base@11.0.11/java.lang.reflect.Method.invoke(Method.java:566)
org.bonitasoft.engine.api.impl.ServerAPIImpl.invokeAPI(ServerAPIImpl.java:427)
org.bonitasoft.engine.api.impl.ServerAPIImpl.lambda$invokeAPIInTransaction$1(ServerAPIImpl.java:395)
org.bonitasoft.engine.api.impl.ServerAPIImpl$$Lambda$475/0x0000000840c37840.call(Unknown Source)
org.bonitasoft.engine.transaction.JTATransactionServiceImpl.executeInTransaction(JTATransactionServiceImpl.java:269)
org.bonitasoft.engine.api.impl.ServerAPIImpl.invokeAPIInTransaction(ServerAPIImpl.java:391)
org.bonitasoft.engine.api.impl.ServerAPIImpl.invokeAPI(ServerAPIImpl.java:265)
org.bonitasoft.engine.api.impl.ServerAPIImpl.invokeMethod(ServerAPIImpl.java:121)
org.bonitasoft.engine.api.impl.ClientInterceptor.invoke(ClientInterceptor.java:79)
com.sun.proxy.$Proxy143.installBusinessDataModel(Unknown Source)
org.bonitasoft.web.rest.server.api.bdm.BusinessDataModelResource.addBDM(BusinessDataModelResource.java:65)
jdk.internal.reflect.GeneratedMethodAccessor161.invoke(Unknown Source)
java.base@11.0.11/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.base@11.0.11/java.lang.reflect.Method.invoke(Method.java:566)
org.restlet.resource.ServerResource.doHandle(ServerResource.java:508)
org.restlet.resource.ServerResource.post(ServerResource.java:1341)
org.restlet.resource.ServerResource.doHandle(ServerResource.java:606)
org.bonitasoft.web.rest.server.api.resource.CommonResource.doHandle(CommonResource.java:206)
org.restlet.resource.ServerResource.doNegotiatedHandle(ServerResource.java:662)
org.restlet.resource.ServerResource.doConditionalHandle(ServerResource.java:348)
org.restlet.resource.ServerResource.handle(ServerResource.java:1020)
org.restlet.resource.Finder.handle(Finder.java:236)
org.restlet.routing.Filter.doHandle(Filter.java:150)
org.restlet.routing.Filter.handle(Filter.java:197)
org.restlet.routing.Router.doHandle(Router.java:422)
org.restlet.routing.Router.handle(Router.java:641)
org.restlet.routing.Filter.doHandle(Filter.java:150)
org.restlet.routing.Filter.handle(Filter.java:197)
org.restlet.routing.Filter.doHandle(Filter.java:150)
org.restlet.routing.Filter.handle(Filter.java:197)
org.restlet.routing.Filter.doHandle(Filter.java:150)
org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:140)
org.restlet.routing.Filter.handle(Filter.java:197)
org.restlet.routing.Filter.doHandle(Filter.java:150)
org.restlet.routing.Filter.handle(Filter.java:197)
org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202)
org.restlet.engine.application.ApplicationHelper.handle(ApplicationHelper.java:77)
org.restlet.Application.handle(Application.java:385)
org.bonitasoft.web.rest.server.BonitaRestletApplication.handle(BonitaRestletApplication.java:191)
org.restlet.routing.Filter.doHandle(Filter.java:150)
org.restlet.routing.Filter.handle(Filter.java:197)
org.restlet.routing.Router.doHandle(Router.java:422)
org.restlet.routing.Router.handle(Router.java:641)
org.restlet.routing.Filter.doHandle(Filter.java:150)
org.restlet.routing.Filter.handle(Filter.java:197)
org.restlet.routing.Router.doHandle(Router.java:422)
org.restlet.routing.Router.handle(Router.java:641)
org.restlet.routing.Filter.doHandle(Filter.java:150)
org.restlet.routing.Filter.handle(Filter.java:197)
org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202)
org.restlet.Component.handle(Component.java:408)
org.restlet.Server.handle(Server.java:507)
org.restlet.engine.connector.ServerHelper.handle(ServerHelper.java:63)
org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:143)
org.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:1117)
javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:232)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:194)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)
org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)
org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:194)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)
org.bonitasoft.console.common.server.login.filter.AbstractAuthorizationFilter.doFilter(AbstractAuthorizationFilter.java:61)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:194)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)
org.bonitasoft.console.common.server.login.filter.AbstractAuthorizationFilter.doFilter(AbstractAuthorizationFilter.java:61)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:194)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)
org.bonitasoft.console.common.server.filter.NoCacheFilter.doFilter(NoCacheFilter.java:51)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:194)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:167)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:544)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:364)
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:624)
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831)
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1651)
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
java.base@11.0.11/java.lang.Thread.run(Thread.java:829)
Wildfly

These messages can be found in the server.log file:

2018-10-08 15:23:41,114 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0afe21b2:4672bcb6:5bbb73ed:6930 in state RUN 
2018-10-08 15:23:41,117 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012095: Abort of action id 0:ffff0afe21b2:4672bcb6:5bbb73ed:6930 invoked while multiple threads active within it. 
2018-10-08 15:23:41,137 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012381: Action id 0:ffff0afe21b2:4672bcb6:5bbb73ed:6930 completed with multiple threads - thread default task-23 was in progress with java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.socketRead(SocketInputStream.java:116) java.net.SocketInputStream.read(SocketInputStream.java:171) java.net.SocketInputStream.read(SocketInputStream.java:141) ... 
2018-10-08 15:23:41,138 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012108: CheckedAction::check - atomic action 0:ffff0afe21b2:4672bcb6:5bbb73ed:6930 aborting with 1 threads active! 
2018-10-08 15:23:41,615 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0afe21b2:4672bcb6:5bbb73ed:6930 in state CANCEL 
2018-10-08 15:23:41,616 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012378: ReaperElement appears to be wedged: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) ... 
2018-10-08 15:23:42,116 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0afe21b2:4672bcb6:5bbb73ed:6930 in state CANCEL_INTERRUPTED 
2018-10-08 15:23:42,117 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012120: TransactionReaper::check worker Thread[Transaction Reaper Worker 0,5,main] not responding to interrupt when cancelling TX 0:ffff0afe21b2:4672bcb6:5bbb73ed:6930 -- worker marked as zombie and TX scheduled for mark-as-rollback 
2018-10-08 15:23:42,117 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012110: TransactionReaper::check successfuly marked TX 0:ffff0afe21b2:4672bcb6:5bbb73ed:6930 as rollback only 
2018-11-29 17:18:17,459 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff7f000101:-201592e0:5c000d43:e772 in state RUN 
2018-11-29 17:18:17,478 DEBUG [org.hibernate.engine.jdbc.internal.LogicalConnectionImpl] (default task-36) Obtained JDBC connection 
2018-11-29 17:18:17,508 DEBUG [org.hibernate.engine.jdbc.internal.LogicalConnectionImpl] (default task-36) Releasing JDBC connection 
2018-11-29 17:18:17,508 DEBUG [org.hibernate.engine.jdbc.internal.LogicalConnectionImpl] (default task-36) Released JDBC connection 
2018-11-29 17:18:17,513 DEBUG [org.hibernate.engine.jdbc.internal.LogicalConnectionImpl] (default task-36) Obtaining JDBC connection 
2018-11-29 17:18:17,516 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012095: Abort of action id 0:ffff7f000101:-201592e0:5c000d43:e772 invoked while multiple threads active within it. 
2018-11-29 17:18:17,517 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012381: Action id 0:ffff7f000101:-201592e0:5c000d43:e772 completed with multiple threads - thread default task-36 was in progress with java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ... 
2018-11-29 17:18:17,517 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012108: CheckedAction::check - atomic action 0:ffff7f000101:-201592e0:5c000d43:e772 aborting with 1 threads active! 
2018-11-29 17:18:17,558 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff7f000101:-201592e0:5c000d43:e772 
2018-11-29 17:18:17,571 WARN [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default task-36) SQL Error: 0, SQLState: null 
2018-11-29 17:18:17,571 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default task-36) javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@63fbf68b[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@78744f3f connection handles=0 lastReturned=1543508297571 lastValidated=1543508198248 lastCheckedOut=1543508297513 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@3ee3e0ef mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@31f31673[pool=BusinessDataDS] xaResource=XAResourceWrapperImpl@1f3212c3[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@78744f3f pad=false overrideRmValue=null productName=Microsoft SQL Server productVersion=13.00.4001 jndiName=java:jboss/datasources/BusinessDataDS] txSync=null] 
2018-11-29 17:18:17,571 DEBUG [org.hibernate.engine.jdbc.internal.LogicalConnectionImpl] (default task-36) Aggressively releasing JDBC connection 
2018-11-29 17:18:17,571 DEBUG [org.hibernate.engine.jdbc.internal.LogicalConnectionImpl] (default task-36) Aggressively releasing JDBC connection 
2018-11-29 17:18:17,571 WARN [com.arjuna.ats.arjuna] (default task-36) ARJUNA012077: Abort called on already aborted atomic action 0:ffff7f000101:-201592e0:5c000d43:e772 
2018-11-29 17:18:17,571 DEBUG [com.bonitasoft.engine.api.impl.ServerAPIExt] (default task-36) THREAD_ID=233 | HOSTNAME=quantum-srv1 | TENANT_ID=1 | javax.transaction.RollbackException: ARJUNA016102: The transaction is not active! Uid is 0:ffff7f000101:-201592e0:5c000d43:e772: org.bonitasoft.engine.transaction.STransactionCommitException: javax.transaction.RollbackException: ARJUNA016102: The transaction is not active! Uid is 0:ffff7f000101:-201592e0:5c000d43:e772 ... Caused by: javax.transaction.RollbackException: ARJUNA016102: The transaction is not active! Uid is 0:ffff7f000101:-201592e0:5c000d43:e772 ... 
2018-11-29 17:18:17,571 TRACE [com.bonitasoft.engine.api.impl.ServerAPIExt] (default task-36) THREAD_ID=233 | HOSTNAME=quantum-srv1 | TENANT_ID=1 | End Server API call org.bonitasoft.engine.api.CommandAPI execute 
2018-11-29 17:18:17,572 SEVERE [org.restlet.Component.BonitaSPRestletApplication] (default task-36) Exception or error caught in server resource: org.bonitasoft.engine.exception.BonitaRuntimeException: USERNAME=walter.bates | org.bonitasoft.engine.transaction.STransactionCommitException: javax.transaction.RollbackException: ARJUNA016102: The transaction is not active! Uid is 0:ffff7f000101:-201592e0:5c000d43:e772 ... Caused by: org.bonitasoft.engine.transaction.STransactionCommitException: javax.transaction.RollbackException: ARJUNA016102: The transaction is not active! Uid is 0:ffff7f000101:-201592e0:5c000d43:e772 ... Caused by: javax.transaction.RollbackException: ARJUNA016102: The transaction is not active! Uid is 0:ffff7f000101:-201592e0:5c000d43:e772 ... 
2018-11-29 17:18:17,572 SEVERE [org.bonitasoft.web.rest.server.api.resource.CommonResource] (default task-36) USERNAME=walter.bates | org.bonitasoft.engine.transaction.STransactionCommitException: javax.transaction.RollbackException: ARJUNA016102: The transaction is not active! Uid is 0:ffff7f000101:-201592e0:5c000d43:e772: org.bonitasoft.engine.exception.BonitaRuntimeException: USERNAME=walter.bates | org.bonitasoft.engine.transaction.STransactionCommitException: javax.transaction.RollbackException: ARJUNA016102: The transaction is not active! Uid is 0:ffff7f000101:-201592e0:5c000d43:e772 ... Caused by: org.bonitasoft.engine.transaction.STransactionCommitException: javax.transaction.RollbackException: ARJUNA016102: The transaction is not active! Uid is 0:ffff7f000101:-201592e0:5c000d43:e772 ... Caused by: javax.transaction.RollbackException: ARJUNA016102: The transaction is not active! Uid is 0:ffff7f000101:-201592e0:5c000d43:e772 ... 
2018-11-29 17:18:18,129 INFO [org.hibernate.event.internal.DefaultLoadEventListener] (default task-36) HHH000327: Error performing load command : org.hibernate.HibernateException: Logical connection is closed 
2018-11-29 17:18:18,131 DEBUG [org.hibernate.engine.jdbc.internal.LogicalConnectionImpl] (default task-36) Aggressively releasing JDBC connection 
2018-11-29 17:18:18,131 DEBUG [org.hibernate.engine.jdbc.internal.LogicalConnectionImpl] (default task-36) Aggressively releasing JDBC connection 
2018-11-29 17:18:18,131 TRACE [org.hibernate.engine.jdbc.internal.LogicalConnectionImpl] (default task-36) Closing logical connection 
2018-11-29 17:18:18,131 TRACE [org.hibernate.engine.jdbc.internal.LogicalConnectionImpl] (default task-36) Logical connection closed 
2018-11-29 17:18:18,131 DEBUG [org.hibernate.engine.jdbc.internal.LogicalConnectionImpl] (default task-36) Aggressively releasing JDBC connection 
2018-11-29 17:18:18,131 TRACE [org.hibernate.engine.jdbc.internal.LogicalConnectionImpl] (default task-36) Closing logical connection 
2018-11-29 17:18:18,131 TRACE [org.hibernate.engine.jdbc.internal.LogicalConnectionImpl] (default task-36) Logical connection closed 
2018-11-29 17:18:18,131 DEBUG [com.bonitasoft.engine.api.impl.ServerAPIExt] (default task-36) THREAD_ID=233 | HOSTNAME=quantum-srv1 | TENANT_ID=1 | javax.persistence.PersistenceException: org.hibernate.HibernateException: Logical connection is closed: org.bonitasoft.engine.persistence.SRetryableException: javax.persistence.PersistenceException: org.hibernate.HibernateException: Logical connection is closed at org.bonitasoft.engine.business.data.impl.JPABusinessDataRepositoryImpl.findById(JPABusinessDataRepositoryImpl.java:193) ... Caused by: javax.persistence.PersistenceException: org.hibernate.HibernateException: Logical connection is closed ...

Comments

Submitted by the.shirini on Sat, 08/10/2019 - 20:45

Thats Cool, Thank you

Notifications