TROUBLESHOOTING NODE.JS what to do when things go wrong Igor Soarez igor@yld.io @igorsoarez HassyVeldstra h@yld.io @hveldstra WDCNZ, July 2015
HELLO HELLO • YLD! • Node.js Consultancy &Training • Digital transformation for enterprise clients
FIRES Someone has to put them out
BE PREPARED
LOG EVERYTHING Make it easy to grep & visualise
TEST TEST TEST • Functional tests first • Good coverage for unit tests • Randomised testing & fuzz testing
MONITORING & ALERTING • Resource monitoring • CPU • Memory • I/O • Application monitoring • Errors • Real-user monitoring • Latency • Availability
AUTOMATED DEPLOYMENTS
PAIR PROGRAMMING
TAKEYOURTIME WHEN BUILDINGYOUR APPLICATION Tight deadlines are the root of all evil
LOADTESTING
LOADTESTING • You should do it
LOADTESTING • You should do it • Find the bottlenecks
LOADTESTING • You should do it • Find the bottlenecks • Understand your system
LOADTESTING • You should do it • Find the bottlenecks • Understand your system
LOADTESTING
LOADTESTING
YOU SHOULD DO IT IN PRODUCTION
IN PRODUCTION??? I must have heard you wrong
WHY NOT? Why — Reason 1
STAGING IS NEVERTHE SAME Why — Reason 1I
GAIN EXTRA CONFIDENCE Why — Reason 1II
IFYOU DOTHIS • Analytics
IFYOU DOTHIS • Analytics • Custom header • Cookie
IFYOU DOTHIS • Analytics • Custom header • Cookie • Let Marketing know
IFYOU DOTHIS • Analytics • Custom header • Cookie • Let Marketing know • Start outside of peak times
IFYOU DOTHIS • Analytics • Custom header • Cookie • Let Marketing know • Start outside of peak times • Start slow & ramp up
LOADTESTINGTOOLS • JMeter
LOADTESTINGTOOLS • JMeter • Tsung
LOADTESTINGTOOLS • JMeter • Tsung • Minigun - https://artillery.io/minigun
AFTERTHE FIRE STARTS how to approach it
PERFORMANCE PROBLEMS
ATTACKTHE PROBLEM 1. Isolated test case * 2. Measure 3. Change /Tweak / Improve 4. Goto 2 * If possible
TYPES OF ISSUES • CPU-bound code • Memory leaks • GC pressure • I/O performance • Event loop lag (Node.js)
KNOWTHETOOLS AVAILABLE • OS-level • perf • sysstat, netstat, netmon, ps, top • sysdig
KNOWTHETOOLS AVAILABLE • Platform-specific (Node.js)
KNOWTHETOOLS AVAILABLE • Platform-specific (Node.js) • V8 profiler • find hot-path functions that cannot be optimized • trace GC activity
KNOWTHETOOLS AVAILABLE • Platform-specific (Node.js) • V8 profiler • find hot-path functions that cannot be optimized • trace GC activity • heapdump • find memory leaks
MEMORY • ps • top / htop • Heapdumps
KNOWTHETOOLS AVAILABLE • Platform-specific (Node.js) • V8 profiler • find hot-path functions that cannot be optimized • trace GC activity • heapdump • find memory leaks • flamegraphs • see what's hot on the CPU
CPU • ps • top / htop • Flamegraphs
KNOWTHETOOLS AVAILABLE • Platform-specific (Node.js) • V8 profiler • find hot-path functions that cannot be optimized • trace GC activity • heapdump • find memory leaks • flamegraphs • see what's hot on the CPU • node-toobusy-js • track the event loop lag
FLAMEGRAPHS
PERF aka Performance Counters
Collected stack frames h() c() h() c() c() z() c() b() f() b() b() f() b() y() b() b() a() a() a() m() a() m() x() m() a() main() main() main() main() main() main() main() main() main()
Collapsed stack frames h() c() h() c() z() b() f() b() y() a() m() x() main()
DEMO!
THANKYOU! @hveldstra @igorsoarez

Troubleshooting Node.js