J'ai récemment reçu les messages suivants dans nos journaux, suivis d'une panne de la machine virtuelle Java (due à OOME). Je ne sais pas quoi en penser et j'apprécierais vraiment toute orientation.
2015-03-19 21:15:02,457 [Timer-0] WARN (ThreadPoolAsynchronousRunner.Java [run]:608) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2015-03-19 21:26:29,543 [Timer-0] WARN (ThreadPoolAsynchronousRunner.Java [run]:624) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@15da1b6b (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b35b08a (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@51cfdd17 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@19397937
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@5c7d3838
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7aea62dd
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@55622ff2
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@74004a8
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.Java:560)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
Java.net.SocketOutputStream.socketWrite0(Native Method)
Java.net.SocketOutputStream.socketWrite(SocketOutputStream.Java:109)
Java.net.SocketOutputStream.write(SocketOutputStream.Java:153)
Java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.Java:82)
Java.io.BufferedOutputStream.flush(BufferedOutputStream.Java:140)
com.mysql.jdbc.MysqlIO.send(MysqlIO.Java:3227)
com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.Java:1917)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.Java:2060)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.Java:2536)
com.mysql.jdbc.ConnectionImpl.configureClientCharacterSet(ConnectionImpl.Java:1751)
com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.Java:3425)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.Java:2196)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.Java:718)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.Java:46)
Sun.reflect.GeneratedConstructorAccessor306.newInstance(Unknown Source)
Sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.Java:45)
Java.lang.reflect.Constructor.newInstance(Constructor.Java:408)
com.mysql.jdbc.Util.handleNewInstance(Util.Java:406)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.Java:302)
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.Java:282)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.Java:135)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.Java:182)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.Java:171)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.Java:137)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.Java:1014)
com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.Java:32)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.Java:1810)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.Java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.Java:560)
2015-03-19 21:56:59,137 [Timer-0] WARN (ThreadPoolAsynchronousRunner.Java [run]:608) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2015-03-19 21:56:59,143 [eXistThread-18676] ERROR (XQueryServlet.Java [process]:566) - Java heap space
Java.lang.OutOfMemoryError: Java heap space
at org.exist.storage.btree.BTree$BTreeNode.read(BTree.Java:1269)
at org.exist.storage.btree.BTree$BTreeNode.access$16(BTree.Java:1239)
at org.exist.storage.btree.BTree.getBTreeNode(BTree.Java:460)
at org.exist.storage.btree.BTree.scanSequential(BTree.Java:413)
at org.exist.storage.btree.BTree$BTreeNode.scanNextPage(BTree.Java:2039)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.Java:1835)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.Java:1759)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.Java:1759)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.Java:1759)
at org.exist.storage.btree.BTree$BTreeNode.access$12(BTree.Java:1734)
at org.exist.storage.btree.BTree.query(BTree.Java:379)
at org.exist.storage.structural.NativeStructuralIndexWorker.scanByType(NativeStructuralIndexWorker.Java:259)
at org.exist.dom.VirtualNodeSet.getNodesFromIndex(VirtualNodeSet.Java:457)
at org.exist.dom.VirtualNodeSet.realize(VirtualNodeSet.Java:585)
at org.exist.dom.VirtualNodeSet.iterator(VirtualNodeSet.Java:860)
at org.exist.dom.AbstractNodeSet.iterator(AbstractNodeSet.Java:1)
at org.exist.storage.structural.NativeStructuralIndexWorker.findDescendantsByTagName(NativeStructuralIndexWorker.Java:162)
at org.exist.xquery.LocationStep.getAttributes(LocationStep.Java:645)
at org.exist.xquery.LocationStep.eval(LocationStep.Java:434)
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.Java:71)
at org.exist.xquery.PathExpr.eval(PathExpr.Java:264)
at org.exist.xquery.Predicate.selectByNodeSet(Predicate.Java:446)
at org.exist.xquery.Predicate.evalPredicate(Predicate.Java:326)
at org.exist.xquery.LocationStep.processPredicate(LocationStep.Java:251)
at org.exist.xquery.LocationStep.applyPredicate(LocationStep.Java:238)
at org.exist.xquery.LocationStep.eval(LocationStep.Java:462)
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.Java:71)
at org.exist.xquery.PathExpr.eval(PathExpr.Java:264)
at org.exist.xquery.LetExpr.eval(LetExpr.Java:142)
at org.exist.xquery.LetExpr.eval(LetExpr.Java:187)
at org.exist.xquery.LetExpr.eval(LetExpr.Java:187)
at org.exist.xquery.BindingExpression.eval(BindingExpression.Java:164)
2015-03-19 21:56:59,147 [Timer-0] WARN (ThreadPoolAsynchronousRunner.Java [run]:624) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@79180a12 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@243c6d0c (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@50191373 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3a9d08ca
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3ecdd11
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@44ff846d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@5ce5850a
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@eec1d04
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@6b8d4d9d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@53e9706d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@23d472cf
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@4dbe4f8c
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@4c5e0203
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@54ac79fd
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@546e2bad
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@6b13cc83
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@57e185f8
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@60357d68
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@45231180
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@3021aa73
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@6bb437ca
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@2021c9e9
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@7d53637c
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@409c2c97
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@adc5929
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@241ca71a
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@42b26866
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@636b1c33
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@b160466
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4af34669
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1b53e609
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@2062ebd4
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1b6cfe8a
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4b7c2380
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4f9be748
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@78108924
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@474b002
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@2ebee32f
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3e0fe017
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@42aa175b
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@637f5bac
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3a017b77
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7b4f2b78
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
Sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.Java:45)
Java.lang.reflect.Constructor.newInstance(Constructor.Java:408)
com.mysql.jdbc.Util.handleNewInstance(Util.Java:406)
com.mysql.jdbc.ResultSetImpl.getInstance(ResultSetImpl.Java:370)
com.mysql.jdbc.MysqlIO.buildResultSetWithRows(MysqlIO.Java:2532)
com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.Java:477)
com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.Java:2510)
com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.Java:1746)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.Java:2135)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.Java:2536)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.Java:2465)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.Java:1383)
com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.Java:4826)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.Java:50)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.Java:4804)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.Java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.Java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.Java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.Java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.Java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.Java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.Java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.Java:2596)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.Java:2465)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.Java:1383)
com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.Java:4826)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.Java:50)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.Java:4804)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.Java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.Java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.Java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.Java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.Java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.Java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.Java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
Java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.Java:1012)
Java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.Java:1006)
com.newrelic.agent.TransactionService.addTransaction(TransactionService.Java:142)
com.newrelic.agent.Transaction.getTransaction(Transaction.Java:1104)
com.newrelic.agent.Transaction.getTransaction(Transaction.Java:1087)
com.newrelic.agent.TracerService$TracerServiceImpl.getTracer(TracerService.Java:136)
com.newrelic.agent.TracerService.getTracer(TracerService.Java:41)
com.newrelic.agent.instrumentation.InvocationPoint.invoke(InvocationPoint.Java:55)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.Java)
com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.Java:4826)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.Java:50)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.Java:4804)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.Java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.Java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.Java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.Java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.Java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.Java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.Java:547)
Nous utilisons hiberanate et c3p0 avec la configuration c3p0 suivante:
Nous utilisons les artefacts Maven suivants pour hiberner et c3p0:
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>4.3.6.Final</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-c3p0</artifactId>
<version>4.3.6.Final</version>
</dependency>
avec la configuration c3p0 suivante:
configuration = new Configuration().setProperty("hibernate.dialect", "org.hibernate.dialect.MySQL5InnoDBDialect")
.setProperty("hibernate.connection.provider_class", "org.hibernate.connection.C3P0ConnectionProvider")
.setProperty("hibernate.c3p0.idle_test_period", "1000")
.setProperty("hibernate.c3p0.min_size", "20")
.setProperty("hibernate.c3p0.max_size", "50")
.setProperty("hibernate.c3p0.timeout", "1800")
.setProperty("hibernate.c3p0.max_statements", "50")
Le serveur est sous une charge très légère, environ 5 requêtes par seconde. (Java 8)
Le problème est donc directement que le pool de connexions essayait d’acquérir de nouvelles connexions, mais que les tâches qui tentaient d’acquérir ces tâches étaient figées pendant une longue période, si longtemps que c3p0 a décidé que les tâches devaient être bloquées, puis ignorées et remplacées par le thread. bassin. Plus tard, le pool de threads a été bloqué lors de tâches de test de connexion inactive.
Normalement, les tâches "bloquées" ont tendance à ressembler au deuxième thread situé sous le premier libellé "Traces de pile de threads de pool:" ci-dessus: réseau IO échoué. Votre situation est étrange en ce que deux des trois threads ne sont pas bloqués dans IO. Ils ont à peine commencé à faire quoi que ce soit, mais ils ne sont pas vivants. Ensuite, vous rencontrez une erreur OutOfMemoryError et vous obtenez un autre APPADENT DEADLOCK sur des tâches de test de connexion inactive qui semblent également devoir être en direct.
Peut-être que votre application est très proche d'une limite de ressources qui rend les choses très lentes? De manière directe, vous pouvez augmenter la quantité de mémoire disponible pour cette application (ou la modifier pour réduire l’empreinte mémoire). Vous rencontrez un OOME la deuxième fois que le pool de threads essaie de vider et de recréer des threads accrochés, ce qui n'est pas directement provoqué par cela, mais très probablement causé par l'empreinte croissante de Thread. (Dans vos journaux, existe-t-il déjà beaucoup de ces DEADLOCK APPARENTS? Si vous forcez un vidage de thread JVM, voyez-vous beaucoup d'instances com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
toujours suspendues à des tâches anciennes?
Quelques commentaires généraux: Si votre charge de travail n’est pas supérieure à 5 requêtes simultanées, pourquoi un tel pool? pourquoi pas min_size
5-ish, max 10-ish? Votre paramètre max_statements est beaucoup trop petit pour le nombre de connexions que vous autorisez dans le pool. Je l'omettrais complètement jusqu'à ce que tout se passe bien. Ensuite, pour obtenir de meilleures performances, vous pouvez définir le paramètre maxStatementsPerConnection
, plus simple à comprendre pour la raison, si vous le souhaitez.
La plupart du temps, vous devez maintenir l'empreinte de votre application (mémoire? Threads?) Bien en dessous des ressources qui lui sont allouées, soit en augmentant les ressources, en réduisant son empreinte, ou en corrigeant tout problème pouvant entraîner une augmentation de l'empreinte de votre ressource. Je commencerais par réduire la taille du pool, la mémoire disponible, puis par la configurer pour qu'il soit beaucoup plus petit.
Je suis tombé sur la même exception et la raison était un mot de passe erroné de l'utilisateur de base de données ...
c3p0 tu es un gars drôle
Problème d'authentification pour moi aussi. Je viens d'ajouter la partie domaine au nom du serveur de base de données et tout a fonctionné. Erreur très trompeuse.