(FAQ)线程死锁导致管理控制台宕机
原因分析： 1、在集群控制器目录下（/opt/kingdee/eas/server/cluster）产生了dump文件，打开后分析，主要内存占用为RPCConnection对象。 但问题是，该dump文件只反映了680M的占用，也可以说，内存占用在680M时oom宕机了，可最大堆内存为1536m 2、分析javacore，可知上述产生dump的原因并非jvm堆内存oom，原因是由于无法创建更多的线程而出现OOM。 Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError":"Failed to create a thread a、main线程对一个ArryList进行了锁定，导致3万多RPC线程以及一个健康度相关线程dynamicFactorRetriever等待。 2LKMONINUSE sys_mon_t:0x000000011343B8C0 infl_mon_t: 0x000000011343B910: 3LKMONOBJECT java/util/ArrayList@07000000017E9BD8/07000000017E9BF0: Flat locked by "main" (0x0000000111C2D600), entry count 1 3LKWAITERQ Waiting to enter: 3LKWAITER "dynamicFactorRetriever[tcp://192.168.2.204:11040]" (0x00000001139F7900) 3LKWAITER "RPC-1" (0x00000001143AA500) 3LKWAITER "RPC-2" (0x00000001143CBC00) 3LKWAITER "RPC-3" (0x00000001144CA900) …. 3LKWAITER "RPC-32725" (0x00000003A3DE5C00) 3LKWAITER "RPC-32726" (0x00000003A3E37500) b、dynamicFactorRetriever拥有一个Object的锁，阻塞了其他类似线程和main线程。 2LKMONINUSE sys_mon_t:0x0000000123141240 infl_mon_t: 0x0000000123141290: 3LKMONOBJECT java/lang/Object@070000000180B920/070000000180B938: owner "dynamicFactorRetriever[tcp://192.168.2.204:11040]" (0x00000001139F7900), entry count 2 3LKWAITERQ Waiting to enter: 3LKWAITER "main" (0x0000000111C2D600) 3LKWAITER "dynamicFactorRetriever[tcp://192.168.2.203:11029]" (0x000000011396AE00) 3LKWAITER "dynamicFactorRetriever[tcp://192.168.2.203:11035]" (0x0000000113977700) 3LKWAITER "dynamicFactorRetriever[tcp://192.168.2.203:11036]" (0x00000001139DA500) 3LKWAITER "dynamicFactorRetriever[tcp://192.168.2.203:11038]" (0x00000001139DEE00) 3LKWAITER "dynamicFactorRetriever[tcp://192.168.2.204:11041]" (0x00000001139E8700) 3LKWAITER "dynamicFactorRetriever[tcp://192.168.2.204:11039]" (0x00000001139EF000) 3LKWAITER "dynamicFactorRetriever[tcp://192.168.2.204:11037]" (0x0000000113A1FE00) 很明显，main和dynamicFactorRetriever发生了死锁。死锁导致所有RPC线程被阻塞，集群控制器在接收RPC连接后不断创建新的线程，但又被阻塞，最终导致故障。 解决方法： 存在互锁的情况，需要修改代码，发私包解决