Download as PDF, PPTX
![Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | poonam@poonam-VirtualBox:/media/sf_shared/native_leak_tests$ diff pmap.1 pmap.9 12a13,14 > 00007f4060000000 32276K rw--- [ anon ] > 00007f4061f85000 33260K ----- [ anon ] 56,57c58 < 00007f40a4000000 18952K rw--- [ anon ] < 00007f40a5282000 46584K ----- [ anon ] --- > 00007f40a4000000 65536K rw--- [ anon ] 146c147 < total 3222140K --- > total 3287676K poonam@poonam-VirtualBox:/media/sf_shared/native_leak_tests$ diff pmap.2 pmap.9 12a13,14 > 00007f4060000000 32276K rw--- [ anon ] > 00007f4061f85000 33260K ----- [ anon ] 56,57c58 < 00007f40a4000000 25600K rw--- [ anon ] < 00007f40a5900000 39936K ----- [ anon ] --- > 00007f40a4000000 65536K rw--- [ anon ] 146c147 < total 3222140K --- > total 3287676K](https://image.slidesharecdn.com/troubleshootingnativememoryleaks-181025181129/75/Troubleshooting-Native-Memory-Leaks-in-Java-Applications-13-2048.jpg)
![Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Memory Map 0000000100000000 8K r-x-- /sw/.es-base/sparc/pkg/jdk-1.7.0_60/bin/sparcv9/java 0000000100100000 8K rwx-- /sw/.es-base/sparc/pkg/jdk-1.7.0_60/bin/sparcv9/java 0000000100102000 56K rwx-- [ heap ] 0000000100110000 2624K rwx-- [ heap ] <--- native heap 00000001FB000000 24576K rw--- [ anon ] <--- Java Heap starts here 0000000200000000 1396736K rw--- [ anon ] 0000000600000000 700416K rw--- [ anon ] 19](https://image.slidesharecdn.com/troubleshootingnativememoryleaks-181025181129/75/Troubleshooting-Native-Memory-Leaks-in-Java-Applications-19-2048.jpg)
![Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | $ diff pmap.15767.1 pmap.15767.3 69,70c69,70 < 00007f6d68000000 17036K rw--- [ anon ] < 00007f6d690a3000 48500K ----- [ anon ] --- > 00007f6d68000000 63816K rw--- [ anon ] > 00007f6d6be52000 1720K ----- [ anon ] ------------------------------------------------------------- $ gdb `which java` core.15767 GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. ... (gdb) x/100s 0x00007f6d690a3000 0x7f6d690a3000: "mory Leak " 0x7f6d690a300c: "Alert: JNI Memory Leak " 0x7f6d690a3025: "Alert: JNI Memory Leak " 0x7f6d690a303e: "Alert: JNI Memory Leak " 0x7f6d690a3057: "Alert: JNI Memory Leak " 0x7f6d690a3070: "Alert: JNI Memory Leak "](https://image.slidesharecdn.com/troubleshootingnativememoryleaks-181025181129/75/Troubleshooting-Native-Memory-Leaks-in-Java-Applications-41-2048.jpg)
Uploaded byPoonam Bajaj Parhar
Troubleshooting Native Memory Leaks in Java Applications
This document discusses troubleshooting native memory leaks in Java applications. It begins with an overview of how memory is laid out from the JVM's perspective, including the Java heap, native memory used internally by the JVM, and memory allocated via JNI. Common issues that can cause native memory leaks are described such as JNI allocations, JAXB usage, and NIO byte buffers. Approaches for detecting and diagnosing native leaks include monitoring process memory usage, checking for OutOfMemoryErrors, using tools like the Native Memory Tracker, and examining a core dump if available.
Troubleshooting Native Memory Leaks in Java Applications
- 1.
- 2.
- 3. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | About me • Poonam Parhar • JVM Sustaining Engineer at Oracle •Co-author of the book ‘Java Performance Companion’ • JavaOne RockStar • Speaker at Java Conferences • Published articles • Blog: https://blogs.oracle.com/poonam/ 3
- 4. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Agenda • Layout of Memory From the Perspective of the JVM • Detect Native Memory Leaks •OutOfMemoryError: Native Memory • Common Issues and their Solutions • Approaches/Tools to Troubleshoot a Native Memory Leak
- 5.
- 6. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Memory allocated by a Java application Java Heap - Memory allocated for JVM’s internal functioning such as Java Heap memory management - Memory used by loaded jars and native libraries -Thread stacks and thread-local storage - … Memory Storage of classes and metadata, including dynamically generated classes through reflection (Metaspace) JIT bytecode compilation and storage of compiled code (codecache) - JNI/JVMTI code allocating memory - NIO allocations - Direct Bytebuffers
- 7.
- 8.
- 9.
- 10. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Monitor Native Memory • ps, top, pmap on Unix systems –watch -n 1 ps v <pid> – top –p <pid> – Examine pmap output • PerfMon or VMMap on Windows
- 11. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | bash-3.2$ ps v 27890 PID STAT TIME SLRE PAGEIN VSZ RSS LIM TSIZ %CPU %MEM COMMAND 27890 S+ 0:01.68 0 0 0 10068904 63652 - 0 3.4 0.4 java -Djava.library.pat 27890 S+ 0:02.09 0 0 0 10076072 64604 - 0 3.6 0.4 java -Djava.library.pat 27890 S+ 0:02.57 0 0 0 10084268 65996 - 0 4.6 0.4 java -Djava.library.pat 27890 S+ 0:03.29 0 0 0 10099628 71592 - 0 3.6 0.4 java -Djava.library.pat 27890 S+ 0:03.60 0 0 0 10104748 72100 - 0 3.6 0.4 java -Djava.library.pat 27890 S+ 0:03.96 0 0 0 10111916 72644 - 0 3.5 0.4 java -Djava.library.pat 27890 S+ 0:04.37 0 0 0 10121136 73328 - 0 3.4 0.4 java -Djava.library.pat 27890 S+ 0:04.93 0 0 0 10129928 83688 - 0 3.1 0.5 java -Djava.library.pat 27890 S+ 0:05.44 0 0 0 10137060 84004 - 0 3.6 0.5 java -Djava.library.pat 27890 S+ 0:05.59 0 0 0 10139108 84272 - 0 3.2 0.5 java -Djava.library.pat ...
- 12.
- 13. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | poonam@poonam-VirtualBox:/media/sf_shared/native_leak_tests$ diff pmap.1 pmap.9 12a13,14 > 00007f4060000000 32276K rw--- [ anon ] > 00007f4061f8500033260K ----- [ anon ] 56,57c58 < 00007f40a4000000 18952K rw--- [ anon ] < 00007f40a5282000 46584K ----- [ anon ] --- > 00007f40a4000000 65536K rw--- [ anon ] 146c147 < total 3222140K --- > total 3287676K poonam@poonam-VirtualBox:/media/sf_shared/native_leak_tests$ diff pmap.2 pmap.9 12a13,14 > 00007f4060000000 32276K rw--- [ anon ] > 00007f4061f85000 33260K ----- [ anon ] 56,57c58 < 00007f40a4000000 25600K rw--- [ anon ] < 00007f40a5900000 39936K ----- [ anon ] --- > 00007f40a4000000 65536K rw--- [ anon ] 146c147 < total 3222140K --- > total 3287676K
- 14.
- 15. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | # A fatalerror has been detected by the Java Runtime Environment: # # java.lang.OutOfMemoryError : unable to create new native Thread # A fatal error has been detected by the Java Runtime Environment: # # java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space? # # Internal Error (allocation.cpp:166), pid=2290, tid=27 # Error: ChunkPool::allocate 15 Native OutOfMemoryError
- 16. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Native OutOfMemoryError • JVM is not able to allocate from native memory – Not managed by the JVM •This process or other processes on the system are eating up native memory • Can make more room for native allocations by: – Reducing the Java Heap, PermGen/Metaspace, number of threads and their stack sizes etc. – Reducing the number of processes running on the system • If the above don’t help, we might be facing a native memory leak – Example: JNI code allocating native buffers 16
- 17.
- 18. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Native Heap OutOfMemoryError with 64-bit JVM • Running with 32-bit JVM puts a maximum limit of 4GB on the process size –So more likely you’ll run out of native memory with 32-bit Java processes • Running with a 64-bit JVM gets us access to unlimited memory, so we would expect never to run out of native memory • However, we might see OutOfMemoryErrors occurring in a 64-bit JVM too • CompressedOops feature implementation determines where the Java heap should be placed in the address space Ø Below 4GB address boundary – no scaling, no offset addition ØBelow 32GB address boundary – no offset addition • The position of the Java heap can put a cap on the maximum capacity of the native heap. 18
- 19. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Memory Map 0000000100000000 8K r-x-- /sw/.es-base/sparc/pkg/jdk-1.7.0_60/bin/sparcv9/java 0000000100100000 8K rwx-- /sw/.es-base/sparc/pkg/jdk-1.7.0_60/bin/sparcv9/java 0000000100102000 56K rwx--[ heap ] 0000000100110000 2624K rwx-- [ heap ] <--- native heap 00000001FB000000 24576K rw--- [ anon ] <--- Java Heap starts here 0000000200000000 1396736K rw--- [ anon ] 0000000600000000 700416K rw--- [ anon ] 19
- 20. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Solution for OutOfMemoryError with 64-bit JVM • Can be resolved by using option -XX:HeapBaseMinAddress=n to specify the address the Java heap should be based at •Setting it to a higher address would leave more room for native heap 20
- 21. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | JAXB Issues • JAXB internally uses Inflater/Deflater to compress/uncompress files –Inflater/Deflater use native memory to hold their data – Depend on Finalizer to deallocate the java objects and the associated native memory data – Delay in running Finalizer can exhaust native memory • JAXBContext.newInstance() called for every new request – Classes from the context get reloaded again – Increases the native memory usage • Environment upgrade fails to upgrade all the JAXB jar files – Classes linking errors leading to re-loading of classes 21
- 22. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | OutOfMemoryError: Direct buffer memory • ByteBuffer.allocateDirect(SIZE_OF_BUFFER) • DirectByteBuffersare garbage collected by using a phantom reference and a reference queue • Maximum direct memory is unbounded, can be limited by using the JVM option -XX:MaxDirectMemorySize=n • We can explicitly call: ((DirectBuffer)buffer).cleaner().clean(); • Make GCs more frequent allowing DirectByteBuffers to be GC’ed 22
- 23. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | NIO ByteBuffers • Java NIO APIs use ByteBuffers as the source and destination of I/O calls •Java Heap ByteBuffers and Native Heap ByteBuffers • Java Heap ByteBuffer for I/O use a temporary direct ByteBuffer per thread • If large native ByteBuffers from multiple threads are used for I/O calls ØNative Memory Exhaustion • -Djdk.nio.maxCachedBufferSize=m (1.8.0_102, 9 and above) 23
- 24.
- 25. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Native Memory: Diagnostic Data • Native memory leaks in the JVM • Native Memory Tracker (NMT) output •Native Memory Leaks outside the JVM – Tools like jemalloc, valgrind, libumem, dbx, purify, UMDH • Process map output with tools like pmap • Core file 25
- 26. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Native Memory Tracker • Native Memory Tracker (NMT) that can be used to track native memory that is used internally by the JVM • It cannot track memory allocated outside the JVM or by native libraries •https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/to oldescr007.html 26
- 27. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | NMT • Start the process with NMT enabled using NativeMemoryTracking • The output level can be set to a ‘summary’ or ‘detail’ level: •-XX:NativeMemoryTracking=summary • -XX:NativeMemoryTracking=detail • Use jcmd to get the native memory usage details: • jcmd <pid> VM.native_memory • jcmd <pid> VM.native_memory baseline • jcmd <pid> VM.native_memory detail.diff/summary.diff 27
- 28.
- 29.
- 30. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 30 • bash-3.2$jcmd 39057 VM.native_memory summary.diff • 39057: • Native Memory Tracking: • Total: reserved=5761678KB +52943KB, committed=472350KB +104143KB • - Java Heap (reserved=4194304KB, committed=163328KB +7680KB) • (mmap: reserved=4194304KB, committed=163328KB +7680KB) • • - Class (reserved=1118333KB +47579KB, committed=117949KB +89963KB) • (classes #68532 +58527) • (malloc=8317KB +2523KB #5461 +3371) • (mmap: reserved=1110016KB +45056KB, committed=109632KB +87440KB) • • - Thread (reserved=21594KB -2057KB, committed=21594KB -2057KB) • (thread #22 -2) • (stack: reserved=21504KB -2048KB, committed=21504KB -2048KB) • (malloc=65KB -6KB #111 -10) • (arena=25KB -2 #42 -4) • • - Code (reserved=250400KB +244KB, committed=5612KB +1348KB) • (malloc=800KB +244KB #1498 +234) • (mmap: reserved=249600KB, committed=4812KB +1104KB) • • - GC (reserved=159039KB +18KB, committed=145859KB +50KB) • (malloc=5795KB +18KB #856 +590) • (mmap: reserved=153244KB, committed=140064KB +32KB) • • - Compiler (reserved=153KB, committed=153KB) • (malloc=22KB #72 -2) • (arena=131KB #3) • • - Internal (reserved=13537KB +6949KB, committed=13537KB +6949KB) • (malloc=13505KB +6949KB #70630 +59119) • (mmap: reserved=32KB, committed=32KB) • • - Symbol (reserved=2715KB +9KB, committed=2715KB +9KB) • (malloc=1461KB +9KB #702 +29) • (arena=1255KB #1) • • - Native Memory Tracking (reserved=1416KB +1031KB, committed=1416KB +1031KB) • (malloc=140KB +34KB #2197 +518) • (tracking overhead=1275KB +997KB) • • - Arena Chunk (reserved=186KB -832KB, committed=186KB -832KB) • (malloc=186KB -832KB)
- 31. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Native Memory Leaks Outside the JVM • For the native memory leaks stemming from outside the JVM, we need to rely on the native memory leak tools for their detection and troubleshooting • Native Memory Leak Detection Tools –jemalloc – valgrind – libumem – dbx – Purify – User-Mode Dump Heap (UMDH) 31
- 32. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | jemalloc • Memory allocator and profiler • FreeBSD –Scalability in terms of multithreading by avoiding lock contentions – Heap profiling capabilities • Very little performance impact (heavy malloc use, ~2%) – Can be used in production systems • Download it from https://github.com/jemalloc/jemalloc • ./configure --prefix=/usr/local --enable-prof • make
- 33.
- 34.
- 35.
- 36. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | • jeprof --show_bytes--gif `which java` jeprof.23299.0.f.heap > leak.gif
- 37. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Valgrind • valgrind --leak-check=yes --log-file=valgrind_full.log-v java <options and class file> • --leak-check=summary | yes | full • May not be effective for Java applications – Reports allocations done by the JVM for its internal functioning • Impacts application performance significantly – 20 to 30 times slower • Can be useful in tracking down JVM memory leaks but we have NMT for that
- 38.
- 39. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | UMDH • Install Debugging Tools for Windows • Use ’gflags’ to set the ‘Create user mode stack trace database flag’ for the process –gflags /i java.exe +ust • https://docs.microsoft.com/en-us/windows- hardware/drivers/debugger/umdh • umdh -p:<pid> -f:first.log , umdh -p:<pid> -f:second.log, … • umdh -d -v -l first.log second.log > report.log
- 40.
- 41. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | $ diff pmap.15767.1 pmap.15767.3 69,70c69,70 < 00007f6d68000000 17036K rw--- [ anon ] < 00007f6d690a300048500K ----- [ anon ] --- > 00007f6d68000000 63816K rw--- [ anon ] > 00007f6d6be52000 1720K ----- [ anon ] ------------------------------------------------------------- $ gdb `which java` core.15767 GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. ... (gdb) x/100s 0x00007f6d690a3000 0x7f6d690a3000: "mory Leak " 0x7f6d690a300c: "Alert: JNI Memory Leak " 0x7f6d690a3025: "Alert: JNI Memory Leak " 0x7f6d690a303e: "Alert: JNI Memory Leak " 0x7f6d690a3057: "Alert: JNI Memory Leak " 0x7f6d690a3070: "Alert: JNI Memory Leak "
- 42. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | #include "JNINativeLeak.h" #include <stdlib.h> #include<memory.h> JNIEXPORT void JNICALL Java_JNINativeLeak_allocateMemory (JNIEnv *env, jobject obj, jint size) { char* bytes = (char*) malloc(size); printf("Allocated %d bytes at %p n", size, (void*)bytes); for (int i=0; i<40; i++) { strcpy(bytes+i*25, "Alert: JNI Memory Leak "); } }
- 43. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Summary • Identify native memory leak • Monitor using j* tools and native tools such as ps, top, pmap •Rule out common issues • Leaks from the JVM – NMT • Leaks outside the JVM – Native tools: jemalloc, valgrind, libumem, purify, UMDH, pmap and core file • https://github.com/poonamparhar/native_leaks
- 44.