版权所有©1993-2016金蝶国际软件集团有限公司④内部公开请勿外传WinDbg介绍崔智刚2WinDbg简介用途获取应用程序内存，CPU，线程使用情况，以及阻塞，异常信息等。使用方式附加进程调试抓取Dump分析文件版本匹配（非操作系统版本）32位的应用使用32位的WinDbg64位的应用使用64位的WinDbg3DUMP文件文件类型.mdmp—不完整的dump文件，只能看堆栈等信息，不能看对象引用关系等.hdmp—不完整的dump文件，可以看部分对象，引用关系.dmp--—完整的dump文件4第一步：抓取DumpWinDbg.dump/mad:\a.dmpWindows2008（文件有可能不可靠）任务管理器DebugDiagnosticsToolprocexp.exe5实时抓取Dump1：附加进程2：.dump/mad:\w3wp1.dmp3：分离进程注意：不能选择StopDebuggin，否则将导致进程关闭。6如何抓取内存泄漏和高CPU问题系统运行平稳后：每隔一分钟抓一次，一般抓3个。如果没有抓到，继续挂起，异常不退出应用程序，直接抓取附加进程抓取一般都需要FullDump非MiniDump7第二步：加载DUMP文件打开WinDbg选择【打开CashDump】选择抓取到的文件8设置Symbols符号表文件：一般以.PDB结尾的后缀文件匹配的符号表才能得到可靠的信息设置路径(ex)srv*d:\symbols*http://msdl.microsoft.com/download/symbolsoad/symbols加载符号表.reload/f--强制全部加载检查加载的符号表情况lmlmvmKingdee_BOS_Business_DynamicForm9设置Symbols.net符号表http://referencesource.microsoft.com程序符号表K/3Cloud的符号表文件，可以预先拷贝到D:\symbols（也可以是其它文件夹）K/3Cloud的符号表文件需要跟DLL的版本一致，否则看到堆栈将不匹配，影响问题定位10加载SOS.dll不同的.netframework加载命令存在区别查看操作系统版本vertarget2.0基本上不会碰到了.loadbysosmscorwks4.0.loadbysosclr如果安装目录跟运行环境不一致.load安装目录\sos11.net版本不一致处理方式执行命令出现类似提示执行.cordll-ve-u–l，提示当前版本4.8，需要4.6将抓取dump对应机器的.netframework拷贝到分析的机器，然后执行.cordll–lp路径，如.cordll-lpD:\4.6.1087.0012加载SOSEXSOS的扩展包，主要用于调试，分析当前线程对象，阻塞等等加载方式.loadsosex.unloadsosex参考http://www.stevestechspot.com/13加载psscor4增加查看时间，HTTPContext等.net4.0.loadpsscor4http://www.microsoft.com/en-us/download/details.aspx?id=21255.net2.0.loadpsscor2http://www.microsoft.com/en-us/download/details.aspx?id=107314线程类型常见的线程划分为以下几个种类：CLR线程W3WP线程RPC线程其他COM线程15闲置状态中的CLR工作线程（WorkerThread）11Id:fb8.268Suspend:0Teb:7ffaa000UnfrozenChildEBPRetAddrArgstoChild0199fdf87c82212477e6baa80000023400000000ntdll!KiFastSystemCallRet0199fdfc77e6baa800000234000000000199fe40ntdll!NtWaitForSingleObject+0xc0199fe6c77e6ba120000023400009c4000000000kernel32!WaitForSingleObjectEx+0xac0199fe80791d401f0000023400009c4000000000kernel32!WaitForSingleObject+0x120199fea4791fdacc000000000000000080a56bccmscorsvr!ThreadpoolMgr::WorkerThreadStart+0x3a0199ffb877e66063000b6da80000000000000000mscorsvr!ThreadpoolMgr::intermediateThreadProc+0x440199ffec00000000791fda8b000b6da800000000kernel32!BaseThreadStart+0x3416闲置状态中的CLRCompletionPort/IOThread9Id:fb8.ef8Suspend:0Teb:7ffac000UnfrozenChildEBPRetAddrArgstoChild0191fec07c821bf477e6611a0000023c0191ff1cntdll!KiFastSystemCallRet0191fec477e6611a0000023c0191ff1c0191ff08ntdll!NtRemoveIoCompletion+0xc0191fef0791fdb220000023c0191ff180191ff1ckernel32!GetQueuedCompletionStatus+0x290191ff24791fdacc00000001809a05fe7ffac000mscorsvr!ThreadpoolMgr::CompletionPortThreadStart+0x490191ffb877e66063000b6da80000000000000000mscorsvr!ThreadpoolMgr::intermediateThreadProc+0x440191ffec00000000791fda8b000b6da800000000kernel32!BaseThreadStart+0x3417闲置状态中的CLR（gate）线程10Id:fb8.bdcSuspend:0Teb:7ffab000UnfrozenChildEBPRetAddrArgstoChild0195fe507c82136477e42439000000000195fe94ntdll!KiFastSystemCallRet0195fe5477e42439000000000195fe9400000000ntdll!NtDelayExecution+0xc0195febc77e424b7000001f4000000000195ffb8kernel32!SleepEx+0x680195fecc791bf4f9000001f40b2646e600000939kernel32!Sleep+0xf0195ffb877e66063000000000000000000000000mscorsvr!ThreadpoolMgr::GateThreadStart+0x540195ffec00000000791bf4a50000000000000000kernel32!BaseThreadStart+0x3418闲置状态中的CLR等待（wait）线程25Id:230.ac0Suspend:0Teb:7ff4c000UnfrozenChildEBPRetAddrArgstoChild1b88ff6c7c573a4e000000011b88ff8400000000NTDLL!ZwDelayExecution+0xb1b88ff8c7923558cffffffff00000001198ba828KERNEL32!SleepEx+0x321b88ffb47c57438b00000000198ba828197f7498mscorsvr!ThreadpoolMgr::WaitThreadStart+0x451b88ffec000000007923556a1973b3d000000000KERNEL32!BaseThreadStart+0x5219闲置状态中的CLR定时器（Timer）线程18Id:fb8.914Suspend:0Teb:7ff5f000UnfrozenChildEBPRetAddrArgstoChild0224ff387c82136477e42439000000010224ff7cntdll!KiFastSystemCallRet0224ff3c77e42439000000010224ff7c000003e8ntdll!NtDelayExecution+0xc0224ffa4791cc578000003e80000000100000000kernel32!SleepEx+0x680224ffb877e66063000000000000000000000000mscorsvr!ThreadpoolMgr::TimerThreadStart+0x300224ffec00000000791cc5480000000000000000kernel32!BaseThreadStart+0x3420闲置状态中的CLR垃圾回收（GC）线程13Id:fb8.c10Suspend:0Teb:7ffa8000UnfrozenChildEBPRetAddrArgstoChild01d6fefc7c82212477e6baa8000002b400000000ntdll!KiFastSystemCallRet01d6ff0077e6baa8000002b40000000000000000ntdll!NtWaitForSingleObject+0xc01d6ff7077e6ba12000002b4ffffffff00000000kernel32!WaitForSingleObjectEx+0xac01d6ff84791fe6b0000002b4ffffffff00000000kernel32!WaitForSingleObject+0x1201d6ffac792356be0000000001d6ffec77e66063mscorsvr!gc_heap::gc_thread_function+0x2f01d6ffb877e66063000d2ed80000000000000000mscorsvr!gc_heap::gc_thread_stub+0x1e01d6ffec00000000792356a0000d2ed800000000kernel32!BaseThreadStart+0x3421闲置状态中的CLR终结器（Finalizer）线程17Id:fb8.7b0Suspend:0Teb:7ffa4000UnfrozenChildEBPRetAddrArgstoChild01e6fdf87c82211477e6711b0000000201e6fe48ntdll!KiFastSystemCallRet01e6fdfc77e6711b0000000201e6fe4800000001ntdll!NtWaitForMultipleObjects+0xc01e6fea477e6107500000002793eee0800000000kernel32!WaitForMultipleObjectsEx+0x11a01e6fec07927826b00000002793eee0800000000kernel32!WaitForMultipleObjects+0x1801e6fee0791fecf4000002dc00000000792376a4mscorsvr!WaitForFinalizerEvent+0x5a01e6ff247924568100000000809a05fe7ffa4000mscorsvr!GCHeap::FinalizerThreadStart+0x9601e6ffb877e66063000d55000000000000000000mscorsvr!Thread::intermediateThreadProc+0x4401e6ffec0000000079245640000d550000000000kernel32!BaseThreadStart+0x3422闲置状态中的CLR调试（Debugger）线程12Id:fb8.b30Suspend:0Teb:7ffa9000UnfrozenChildEBPRetAddrArgstoChild01adfe707c82211477e6711b0000000301adfec0ntdll!KiFastSystemCallRet01adfe7477e6711b0000000301adfec000000001ntdll!NtWaitForMultipleObjects+0xc01adff1c77e610750000000301adff5c00000000kernel32!WaitForMultipleObjectsEx+0x11a01adff3879238fd60000000301adff5c00000000kernel32!WaitForMultipleObjects+0x1801adffa079238f4d000000000000000000000000mscorsvr!DebuggerRCThread::MainLoop+0x9001adffb07923a71401adffec77e66063019d1eb0mscorsvr!DebuggerRCThread::ThreadProc+0x6801adffb877e66063019d1eb00000000000000000mscorsvr!DebuggerRCThread::ThreadProcStatic+0xb01adffec000000007923a709019d1eb000000000kernel32!BaseThreadStart+0x3423W3WP主（main）线程.0Id:fb8.fd8Suspend:0Teb:7ffdf000UnfrozenChildEBPRetAddrArgstoChild0006fc087c82212477e6baa80000018400000000ntdll!KiFastSystemCallRet0006fc0c77e6baa8000001840000000000000000ntdll!NtWaitForSingleObject+0xc0006fc7c77e6ba1200000184ffffffff00000000kernel32!WaitForSingleObjectEx+0xac0006fc905a36467a00000184ffffffff00000000kernel32!WaitForSingleObject+0x120006fca05a366e6300254ff85a3af41d00000000w3dt!WP_CONTEXT::RunMainThreadLoop+0x100006fca85a3af41d0000000064711da900000000w3dt!UlAtqStartListen+0x2d0006fcb85a3bc2590100141c010013e4010012d0w3core!W3_SERVER::StartListen+0xbd0006ff0c0100187c00000007002538e000000000w3core!UlW3Start+0x26e0006ff4401001a2300000007002538e0002543d0w3wp!wmain+0x22a0006ffc077e523cd00000000000000007ffd7000w3wp!wmainCRTStartup+0x12b0006fff000000000010018f80000000078746341kernel32!BaseProcessStart+0x2324闲置状态中的w3wp线程池（ThreadPool）线程2Id:fb8.c5cSuspend:0Teb:7ffd9000UnfrozenChildEBPRetAddrArgstoChild00c8ff247c821bf477e6611a0000017000c8ff80ntdll!KiFastSystemCallRet00c8ff2877e6611a0000017000c8ff8000c8ff6cntdll!NtRemoveIoCompletion+0xc00c8ff545a30249e0000017000c8ff7c00c8ff80kernel32!GetQueuedCompletionStatus+0x2900c8ff8c5a3026bc00000000002585805a300000w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x3300c8ffa05a301db9000001020000000000000000w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x2400c8ffb877e66063002585800000000000000000w3tp!THREAD_MANAGER::ThreadManagerThread+0x3900c8ffec000000005a301d800025858000000000kernel32!BaseThreadStart+0x3425闲置状态中的w3wp压缩（Compression）线程7Id:fb8.b20Suspend:0Teb:7ffaf000UnfrozenChildEBPRetAddrArgstoChild017dfa847c82212477e6baa8000001d000000000ntdll!KiFastSystemCallRet017dfa8877e6baa8000001d00000000000000000ntdll!NtWaitForSingleObject+0xc017dfaf877e6ba12000001d0ffffffff00000000kernel32!WaitForSingleObjectEx+0xac017dfb0c5a3b8147000001d0ffffffff00000000kernel32!WaitForSingleObject+0x12017dffb877e66063000000000000000000000000w3core!HTTP_COMPRESSION::CompressionThread+0x126017dffec000000005a3b80210000000000000000kernel32!BaseThreadStart+0x3426星空产品可以忽略的线程27第三步：开始目标高CPU？内存泄漏？异常？阻塞/死锁？所有现象都可能?28CPU相关命令.time!runaway!threads!threadpoolkb~29s!clrstack!dumpstack任何线程都是以ntdll!RtlUserThreadStart开始的29高CPU同一对象并发访问锁资源争用Linq查询死循环操作大对象30.timeUsertime31!runaway选择连续抓取的dump进行比较，确定线程的时间一直在累加，然后查看该线程正在做什么32!threads线程运行状态，两个参数live,special33ID：托管线程的ID,Thread.ManagedThreadId的值一致；OSID：操作系统线程ID；如果托管没有任何本机线程与其关联，旁边会显示一个0；~~[c68]sThreadOBJ：内存中的线程对象State：线程状态http://www.willydev.net/descargas/prev/DBG_download.pdf34PreEmptiveGC：表示线程是否可以为垃圾回收而中断GCAllocContext：报告垃圾回收器用于同步存储的同步对象，如果不为0，说明运行在一台多处理器的机器上。终结器线程，未启动线程，以及死线程，都为0Domain：应用程序域!dumpdomainLock：显示线程中锁的数量APT：显示COM组件模型；!comstateException：线程正在处理的异常35!threadpool线程池情况Cpuutilization36!clrstack查看所有托管代码堆栈~*e!clrstack查看调用方法参数对象!clrstack–p!clrstack-a3738内存!eeheap!dumpheap!do,!dso,!mdso!objsize!mdt!gcroot,!refs!gcwhere,!gcgen!gchandles!gchandleleaks39内存数量多尺寸大的对象大对象40!eeheap!eeheap-gcHeap:有多少CPU就有多少个heapGen起始地址短暂对象分布情况大对象分布情况大小查看某个段的对象情况!dumpheap000000cbfffd1000000000cc4a02460841!dumpheap-stat查看堆的统计信息-strings查看字符串，显示头63个字符-short只显示地址-mt方法表参数-min–max-type/DDMLSupport42!DumpHeap/d-mt00007ffd87cd13d0.foreach(mv{!dumpheap-min85000-short}){!do${mv};.echo*************}循环获取所有大对象!dumpheap–typeSystem.String43!dumpheap-typeKingdee.BOS.Core.Permission.BusinessObject44!do查看某个对象信息!dumpheap-mt00007ffd28d03a8845!mdt显示对象所有信息包括子对象!mdt–r显示对象到第二级!mdt–e:246!dso查看当前堆栈的所有对象47!objsize查看对象的实际大小值类型一般从使用dumpheap可以得到引用类型必须通过该命令获取如果查询返回负数48!gcroot查看对象的引用关系Pinned垃圾回收过程中不会移动的强句柄固定句柄是垃圾回收堆中已锁定到物理地址的内存地址。因为已经锁定，所以垃圾回收器在合并已释放内存时必须绕过它们。这将大大拖慢垃圾回收器工作速度如果对象A拥有类B的一个字段，则称实例Brooted于A，而Arooted于垃圾回收堆。拥有实例A的一个引用，意味着A和B两个对象都无法被回收。如果释放了拥有A引用的变量，A和B就都可以被回收了49对象的线程为1ef8，可以通过~~[1ef8]得到对应的线程50!refs逐层查看对象被引用和引用关系51!gchandles查看句柄，关注pinned.net内存在引用程序中，内部引用都被保存于来自System.Runtime.InteropServices命名空间的GCHandle结构中，是对一个存储内存地址IntPtr类的简单包装52关注Pinned53查看事件句柄情况!dumpheap-typeEventHandle!dumpheap-mt000007fef357dfe0!do00000001c5358ea0!do00000001c5354e40!gcroot00000001c5354e4054!GCHandleLeaks句柄泄漏情况，只显示强句柄和固定句柄!dopoi(address)55!gcwhere对象在哪一代!gcwhere000000cc1fefe728!gcgen000000cc1fefe72856异常!dumpheap–typeException!dumpheap–mt!pe如果异常是多次抛出，需要关注_innerException和_remoteStackTraceString信息，来获取真实出错的位置570:020>!pe00000001064ac370Exceptionobject:00000001064ac370Exceptiontype:System.Data.SqlClient.SqlExceptionMessage:事务(进程ID105)与另一个进程被死锁在锁资源上，并且已被选作死锁牺牲品。请重新运行该事务。InnerException:StackTrace(generated):SPIPFunction0000000008DD8B20000007FF01382D8DKingdee_BOS_ServiceHelper!Kingdee.BOS.ServiceHelper.BusinessDataServiceHelper.Save(Kingdee.BOS.Context,Kingdee.BOS.Core.Metadata.BusinessInfo,Kingdee.BOS.Orm.DataEntity.DynamicObject,Kingdee.BOS.Orm.OperateOption)+0x18d…Serverstacktrace:在System.Data.SqlClient.SqlConnection.OnError(SqlExceptionexception,BooleanbreakConnection)在System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()在System.Data.SqlClient.TdsParser.Run(RunBehaviorrunBehavior,SqlCommandcmdHandler,SqlDataReaderdataStream,BulkCopySimpleResultSetbulkCopyHandler,TdsParserStateObjectstateObj)在System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(StringmethodName,Booleanasync)在System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResultresult,StringmethodName,BooleansendToPipe)在System.Data.SqlClient.SqlCommand.ExecuteNonQuery()在Kingdee.BOS.App.Data.AbstractDatabase.ExecuteNonQuery(DbCommandcommand,IEnumerable`1listParam)在Kingdee.BOS.App.Data.DBUtils.Execute(Contextctx,StringstrSQL,IEnumerable`1paramList)在Kingdee.BOS.App.Core.CodeRuleSeqProject.GetBillFlowNo(BooleanupdateMaxnum)58SQLFRULED索引IFNOTEXISTS(SELECT1FROMT_BAS_BILLCODESWHERE(FRULEID='cbb1a5779ee7492e8f3034119bab44bf'ANDFBYVALUE=N'{{{{{0}}}'))BEGININSERTINTOT_BAS_BILLCODESSELECTISNULL(max(fcodeid),0)+1,'cbb1a5779ee7492e8f3034119bab44bf',N'{{{{{0}}}',1FROMT_BAS_BILLCODESENDELSEBEGINUPDATET_BAS_BILLCODESSETFNUMMAX=(FNUMMAX+1)WHERE(FRULEID='cbb1a5779ee7492e8f3034119bab44bf'ANDFBYVALUE=N'{{{{{0}}}')END;596061阻塞/死锁!syncblk!dlk6243号线程：取序号的操作47号线程：在进行IO的写操作，我们看一下执行的堆栈情况0:000:x86>!syncblkIndexSyncBlockMonitorHeldRecursionOwningThreadInfoSyncBlockOwner1110000000000f5ba1c11000000001ade6b30cd44705fca480System.IO.TextWriter+SyncTextWriter135000000001f2be6cc12000000001ade5710120430635bc44System.Data.SqlClient.SqlInternalConnectionTds0:047:x86>!clrstackOSThreadId:0xcd4(47)ChildSPIPCallSite000000002233e07c2221dfd8[GCFrame:000000002233e07c]000000002233e1242221dfd8[HelperMethodFrame:000000002233e124]System.Threading.Monitor.Enter(System.Object)000000002233e170704f7834System.IO.TextWriter+SyncTextWriter.WriteLine(System.String)***WARNING:Unabletoverifychecksumformscorlib.ni.dll000000002233e1801f9de024Kingdee.BOS.Core.DynamicForm.AbstractDynamicFormModel.UpdateFieldValue(Kingdee.BOS.Core.Metadata.FieldElement.Field,Kingdee.BOS.Orm.DataEntity.DynamicObject,System.Object,Int32)63codepublicboolDataChanged{get{return_bDataChanged;}set{if(value==true){Console.WriteLine("");}_bDataChanged=value;}}64IIS崩溃自动生成DUMP将会在D盘的dumps目录下，最多生成2个完全的DUMP文件，如果出现多次，前面的将被覆盖（可修改）WindowsRegistryEditorVersion5.00[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\WindowsErrorReporting\LocalDumps\w3wp.exe]"DumpFolder"=hex(2):64,00,3a,00,5c,00,64,00,75,00,6d,00,70,00,73,00,00,00"DumpCount"=dword:00000002"DumpType"=dword:0000000265Oracle.DataAccess引起崩溃自动产生DUMPTraceLevel=8表示出现非托管异常时，将生成DUMPTraceOption=1表示将产生多个文件由于Oracle.DataAccess宿主在IIS里面，实际生成的目录将为IIS崩溃配置文件的目录[HKEY_LOCAL_MACHINE\SOFTWARE\Oracle\ODP.NET\4.112.4.0]"TraceLevel"=“8""TraceOption"=“1"66IIS重启问题分析应用程序中有如下提示信息67IIS重启问题分析加载netexthttps://github.com/rodneyviana/netext/tree/master/Binaries.loadnetext68IIS重启问题分析!wruntime!wfrom-typeSystem.Web.FileMonitorwhere($enumname(_lastAction)=="Modified")$a("==================\nPath",DirectoryMonitor.Directory+"\\"+_fileNameLong),$a("ChangeTime:",$tickstodatetime(_utcLastCompletion.dateData)),$a("Action",$enumname(_lastAction))69Netext介绍可以对.net的mindump进行分析可以使用类似的SELECT语句的方式查询信息查询0x242afe8(HttpContext)对象的_request._url.m_String和_response._statusCode属性值查询状态不为200的HttpRequest!wselect_request._url.m_String,_response._statusCodefrom0x242afe8!wfrom-type*.HttpContextwhere(($contains(_request._url.m_String,"http:"))&&(_response._statuscode!=0n200))select$addr(),_request._url.m_String,_response._statusCode70Q&A
