Downloaded 71 times
Uploaded bySasha Goldshtein
.NET Debugging Workshop
This document summarizes a workshop on .NET debugging techniques. The workshop covers debugging production issues, analyzing system and application performance, and automating debugging processes. It discusses tools like Procdump, DebugDiag, and Windows debugging tools. It also covers debugging techniques like generating dump files, using Event Tracing for Windows (ETW) to collect traces, and analyzing traces with tools like PerfView. The goal is to help integrate automatic error analysis and triage into development processes.
.NET Debugging Workshop
- 1. .NET Debugging Workshop #devconnections
- 2. SESSION TITLE #devconnections Sasha Goldshtein CTO, Sela Group Microsoft C# MVP, Azure MRS @goldshtn blog.sashag.net #devconnections
- 3. .NET DEBUGGING WORKSHOP In This Workshop… • Debugging issues in production environments • Automating triage and analysis processes • Analyzing system and application performance #devconnections
- 4. .NET DEBUGGING WORKSHOP Production Debugging • Requirements – Obtain actionable information about crashes and errors – Obtain accurate performance information • Limitations – Can’t install Visual Studio – Can’t suspend production servers – Can’t run intrusive tools #devconnections
- 5. .NET DEBUGGING WORKSHOP In the DevOps Process… • Automatic build (CI) • Automatic deployment (CD) • Automatic monitoring • Automatic error triage and analysis • Automatic remediation #devconnections
- 6. .NET DEBUGGING WORKSHOP The Tools • Sysinternals Procdump • DebugDiag • Windows SDK – Debugging Tools for Windows – Windows Performance Toolkit • PerfView #devconnections
- 7.
- 8. .NET DEBUGGING WORKSHOP Dump Files • A user dump is a snapshot of a running process • A kernel dump is a snapshot of the entire system • Dump files are useful for post-mortem diagnostics and for production debugging – Anytime you can’t attach and start live debugging, a dump might help #devconnections
- 9. .NET DEBUGGING WORKSHOP Limitations of Dump Files • A dump file is a static snapshot – You can’t debug a dump, just analyze it – Sometimes a repro is required (or more than one repro) • Sometimes several dumps must be compared #devconnections
- 10. .NET DEBUGGING WORKSHOP Taxonomy of Dumps • Crash dumps are dumps generated when an application crashes • Hang dumps are dumps generated on-demand at a specific moment • These are just names; the contents of the dump files are the same! #devconnections
- 11. .NET DEBUGGING WORKSHOP Generating a Hang Dump • Task Manager, right-click and choose “Create Dump File” – Creates a dump in %LOCALAPPDATA %Temp #devconnections
- 12. .NET DEBUGGING WORKSHOP Procdump • Sysinternals utility for creating dumps • Examples: Procdump -ma app.exe app.dmp Procdump -ma -h app.exe hang.dmp Procdump -ma -e app.exe crash.dmp Procdump -ma -c 90 app.exe cpu.dmp Procdump -m 1000 -n 5 -s 600 -ma app.exe #devconnections
- 13. .NET DEBUGGING WORKSHOP Windows Error Reporting • WER can create dumps automatically – HKLMSoftwareMicrosoftWindows Windows Error ReportingLocalDumps – http://tinyurl.com/localdumps • Can be application-specific, not system-wide #devconnections
- 14. .NET DEBUGGING WORKSHOP DebugDiag • Microsoft tool for monitoring and dump generation – Very suitable for ASP.NET – Dump analysis component included #devconnections
- 15. .NET DEBUGGING WORKSHOP Debugging Symbols • Debugging symbols link runtime memory addresses to function names, source file names and line numbers – PDB files – Required for proper debugging and dump analysis #devconnections
- 16. .NET DEBUGGING WORKSHOP Symbols for Microsoft Binaries • Microsoft has a public symbol server with PDB files for Microsoft binaries • Configure _NT_SYMBOL_PATH environment variable setx _NT_SYMBOL_PATH srv*C:symbols*http://msdl.microsoft.com/download/symbols #devconnections
- 17. .NET DEBUGGING WORKSHOP Opening Dump Files • Visual Studio can open dump files – For .NET, CLR 4.0+ and VS2010+ required – VS2013 recommended #devconnections
- 18. .NET DEBUGGING WORKSHOP Opening Dump Files • WinDbg is a free lightweight debugger • No intrinsic .NET support, but has SOS debugging extension !analyze -v (CLR 4.0+) .loadby sos clr !printexception !clrstack #devconnections
- 19. .NET DEBUGGING WORKSHOP Configuring LocalDumps Obtaining and opening a dump file TRY IT OUT #devconnections
- 20. Automatic Dump Analysis #devconnections
- 21. .NET DEBUGGING WORKSHOP Basic Automation • Run WinDbg automatically on a bunch of files and log its output: @echo off for %%f in (.*.dmp) do ( echo Launching analysis of file %%f... start "Analyzing %%f" "C:Program Files (x86)Windows Kits8.1Debuggersx86cdb.exe" -z %%f -c ".logopen %%f.log; !analyze -v; .logclose; qd" ) #devconnections
- 22. .NET DEBUGGING WORKSHOP Basic Automation • Parse the results for interesting tokens: for %%f in (.*.dmp.log) do ( echo In file %%f: findstr "EXCEPTION_MESSAGE MANAGED_OBJECT_NAME" %%f ) #devconnections
- 23. .NET DEBUGGING WORKSHOP ClrMD • Text-based analysis of debugger command output is very fragile and limited • ClrMD is a .NET library for analyzing dump files (and running processes) – A managed API for interacting with the .NET debugging runtime (“SOS API”) – Distributed through NuGet (search “ClrMD”) #devconnections
- 24. .NET DEBUGGING WORKSHOP ClrMD Basic Classes #devconnections DDaattaaTTaarrggeett CCllrrRRuunnttiimmee CCllrrRRuunnttiimmee CCllrrHHeeaapp CCllrrTThhrreeaadd CCllrrTTyyppee CCllrrTTyyppee CCllrrTThhrreeaadd
- 25. .NET DEBUGGING WORKSHOP mscordacwks.dll • Managed dump analysis requires mscordacwks.dll matching the CLR version • It can be automatically downloaded from the Microsoft symbol server in most cases #devconnections
- 26. .NET DEBUGGING WORKSHOP Connecting to a Target #devconnections
- 27. .NET DEBUGGING WORKSHOP Basic Exception Triage #devconnections
- 28. .NET DEBUGGING WORKSHOP Getting stacks from a live process TRY IT OUT #devconnections
- 29. .NET DEBUGGING WORKSHOP Inspecting the Heap • Enumerate all heap objects and statistics • Find specific objects • Inspect GC information (roots, finalization queues, etc.) #devconnections ClrHeap EnumerateObjects GetObjectType EnumerateRoots ClrType GetSize EnumerateRefsOfObject GetFieldValue
- 30. .NET DEBUGGING WORKSHOP Wait Information • Threads have a list of blocking objects, which have owner threads • Wait analysis and deadlock detection is made possible ClrThread BlockingObjects BlockingObject Reason Object HasSingleOwner Owner/Owners Waiters #devconnections
- 31. ETW and PerfView #devconnections
- 32. .NET DEBUGGING WORKSHOP Event Tracing for Windows • High-performance facility for emitting 100K+ log events per second with rich payloads and stack trace support • Used widely across Windows, .NET, drivers, services, third party components #devconnections
- 33. .NET DEBUGGING WORKSHOP ETW Participants • A provider generates ETW events • A controller starts and stops ETW collection • A consumer logs, analyzes, or processes ETW events #devconnections
- 34. .NET DEBUGGING WORKSHOP ETW Scenarios • Profile an app in sampling mode • Perform wait-time analysis • Log disk accesses including stacks • Log GC and JIT events • Log memory allocation statistics (C++) • Custom application event log #devconnections
- 35. .NET DEBUGGING WORKSHOP Custom ETW Events #devconnections
- 36. .NET DEBUGGING WORKSHOP ETW Tools • xperf.exe: Command-line tool for ETW capturing and processing • wpr.exe: Command-line and GUI for end users • wpa.exe: Visual trace analysis tool • PerfView.exe: Visual tool for capturing and recording ETW events from managed providers and the CLR #devconnections
- 37. .NET DEBUGGING WORKSHOP Capturing a Trace • Xperf xperf -on DiagEasy ... xperf -d diag.etl • WPR #devconnections
- 38. .NET DEBUGGING WORKSHOP What’s In A Trace? • A trace is a huge list of events • Events have multiple columns (payload) • Useless without additional processing #devconnections
- 39. .NET DEBUGGING WORKSHOP Trace Processing with Xperf • I/O summary report per file xperf -i fileio.etl -o fileio.csv -a diskio -summary • Interactive profiling report (for a specific process) xperf -i cpu.etl -o cpu.html -symbols -a stacks -process app.exe -butterfly #devconnections
- 40. .NET DEBUGGING WORKSHOP Managed Stacks • To display managed stack traces correctly, additional CLR data is required • WPR & PerfView take care of this automatically • If using Xperf, see: http:// msdn.microsoft.com/en-us/library/windows/desktop/#devconnections
- 41. .NET DEBUGGING WORKSHOP Collecting file I/O information TRY IT OUT #devconnections
- 42. .NET DEBUGGING WORKSHOP Trace Analysis with WPA List of graphs List of graphs #devconnections Graph display Graph display Ungrouped Ungrouped columns Grouped columns columns Grouped columns Grouping Grouping bar bar
- 43. .NET DEBUGGING WORKSHOP Stack Summaries • Learn how to read stack summaries – Group by Stack column – Expand “hot path”, like in profiler • Stack resolution requires symbols (slow) #devconnections
- 44. .NET DEBUGGING WORKSHOP PerfView • ETW collection and analysis tool tailored for .NET applications (but not only) • Can be used as a sampling profiler • Can be used as an allocation profiler • Can be used for heap snapshot analysis #devconnections
- 45. .NET DEBUGGING WORKSHOP Collecting Data w/ PerfView • CLI PerfView run app.exe • GUI #devconnections
- 46. .NET DEBUGGING WORKSHOP PerfView Collection Options Profiling wall-clock Profiling wall-clock time time CPU sampling profiling #devconnections File/registry accesses File/registry accesses Allocation profiling Allocation profiling CPU sampling profiling
- 47. .NET DEBUGGING WORKSHOP PerfView Tables Grouping options Filtering Grouping options #devconnections Filtering options options CCaalll ls tsatacckk t rtereee In-trace activity highlighter In-trace activity highlighter
- 48. .NET DEBUGGING WORKSHOP Memory Leak Analysis • PerfView can generate heap snapshots (smaller than a dump), analyze, and compare them • Can also import dumps directly #devconnections
- 49. .NET DEBUGGING WORKSHOP Leak analysis with PerfView TRY IT OUT #devconnections
- 50. .NET DEBUGGING WORKSHOP Automatic ETW Analysis • The TraceEvent library provides an API for ETW analysis – Understands kernel and CLR events – Supports call stacks (incl. managed) #devconnections
- 51. .NET DEBUGGING WORKSHOP Example Analysis Scenarios • Monitor the system for CLR exceptions w/ stacks ExceptionTraceData • Get a profiling trace and look for regressions TraceLog SampledProfileTraceData TraceCallStack #devconnections
- 52. .NET DEBUGGING WORKSHOP Summary • Production debugging and performance investigation is here, and entirely possible thanks to dumps and ETW traces • Integrate automatic error analysis and triage into your devops process #devconnections
- 53. SESSION TITLE RateThis Session Now! #devconnections Rate with Mobile App: 1. Select the session from the Agenda or Speakers menus 2. Select the Actions tab 3. Click Rate Session Rate Using Our Website: 1. Register at www.devconnections.com/logintoratesession 2. Go to www.devconnections.com/ratesession 3. Select this session from the list and rate it Tell Us What You Thought of This Session Be Entered to WIN Prizes!