Oracle WebLogic Diagnostics & Perfomance tuning

Oracle WebLogic Troubleshoot, Diagnostics & Performance tuning Michel Schildmeijer, 13 juni 2013

Agenda 17:10-18:00 • What does Performance include? • Performance hints, tips & tricks • Questions

My personalia • Michel Schildmeijer, 46 years, living in Amsterdam • Oracle Fusion Middleware specialist WebLogic 6 to 12, OSB, Tuxedo, SOA • Since April 2012 part of Qualogy • Part of Qualogy’s Exalogic Squad team

WebLogic Troubleshoot overview • System errors, messages lost, bad performance • Where do I begin?? • What method should I follow?

WebLogic Troubleshoot overview • Using these troubleshooting methods helps you to: • A structured way of detecting issues • Get behind the cause of the problems that occur • Methods or patterns to follow: • Use of Operating system tools • Use of JDK tools • Use of WebLogic Server tools • Most likely you will often use a combi-nation

WebLogic Troubleshoot overview • O/S tools: • sar: system activity • mpstat: per-processor statistics • vmstat: virtual memory statistics • netstat: network statistics • iostat: Input/Output statistics • top

WebLogic Troubleshoot overview • Start with a simple check – it seems so obvious but: • Is there enough diskspace left • Is there enough RAM available • Does the system do a lot of paging

WebLogic Troubleshoot overview • On Operating System: • See if WebLogic's operating system Java process is running • Unix: ps -ef | grep -v ' weblogic.Name' | grep weblogic.Name – color to see which server instances are running on that host • Simple O/S tools like top, nmon, sar can help you on the way • Check the contents of the process's system-out/err logs • Usually in the WebLogic Server log location or redirected to the Server Logs • Check the contents of the WebLogic server and domain logs • Located at: domain/servers/ server/logs

Tools to troubleshoot • WebLogic Server generic: • WebLogic Diagnostics Framework • Logs • WLST • JVM specific : • JRockit Mission Control, • Jstat, • JConsole

Tools to investigate Oracle Fusion Middleware basis • Operating system basic HotSpot JDK tools: • FMW basis: de Applicatie Server • jps -l • jinfo pid • jmap pid • jstat -optionpid • Wat is een Applicatie Server? • Wat doet een Applicatie Server? • pid is gathered using ps –ef …. Or using jps • Welke Applicatie Server gebruikt Oracle?

WebLogic Troubleshoot overview • jps –l

Tools to investigate Oracle Fusion Middleware basis • Operating system basic JRockit JDK tools: • jps -l • jrcmd pid --MissonControl Command line • FMW basis: de Applicatie Server • Wat is een Applicatie Server? • pid is gathered using ps –ef …. Or using jps • Wat doet een Applicatie Server? • Welke Applicatie Server gebruikt Oracle?

Tools to investigate Oracle Fusion Middleware basis • jrcmd • FMW basis: de Applicatie Server • Wat is een Applicatie Server? • Wat jrcmd doet pid -l een | grep Applicatie jrockit.gc.Server? latest.heapSize • Welke Applicatie Server gebruikt Oracle?

Tools to investigate Oracle Fusion Middleware basis • Force a dump by: • FMW basis: de Applicatie Server • kill -3 pid … stack in STDOUT • Usually in domain/servers/logs/serverinstance/serverinstance.out • Wat is een Applicatie Server? • Wat doet een Applicatie Server? • Welke Applicatie Server gebruikt Oracle?

WebLogic troubleshoot methods • Keep it as simple as it is : • In the AdminConsole • WLST • connect('weblogic', 'weblogic', 't3://localhost:7001') serverRuntime() print cmo.getHealthState() • Or in clustered mode domainConfig() serverList=cmo.getServers(); domainRuntime() cd('/ServerLifeCycleRuntimes/server') cmo.getState()

WebLogic troubleshoot methods • Just a simple example, but parameterize your scripts

WebLogic taking threaddumps • A threaddump is a snapshot of the state of a WebLogic Server Instance • Different ways to take a threaddump: • kill -3 pid the O/S way • Using the JDK tools • WLST(WebLogic Scripting Tool) cd ('Servers') ls() cd ('AdminServer') ls() threadDump()

WebLogic taking threaddumps • In the AdminConsole

The use of taking threaddumps • Thread Dumps can give you: • View of the state of application server threads at that instant in time • Information about potential issues such as • Hot spots within code which seem to be called often • Portions of code where the application seems to be hung • Locking and thread synchronization issues in an application

The use of taking threaddumps • Take several dumps to investigate various states • Threads high in stack can be treated as idle, mostly waiting for some work to do • Blocked or Wait on socketAccept() listener thread waiting for socket connections

Understanding Java threads • JVM process consists Java and JVM internal threads • JVM threads: Garbage Collect, Optimizer and Finalizer • Each thread has a stack to store runtime data • Overflow will happen if this stack will store more than its size • Each O/S has its default stacksize • Can be extended with -Xss

Understanding Locks • Are meant to synchronize between threads • Lock contention is a thread waiting for another to be released • Fat locks: Have a history of contention (several threads try to take the lock simultaneously), or waiting for notification • Thin locks: Without any contention. • Recursive locks: A lock on a thread several times without having been released.

The use of taking threaddumps • WebLogic specific: • Ignore threads in ExecuteThread.waitForRequest() • Muxer and processSocket threads are mainly for houskeeping • A Muxer which is blocked should not be waiting for a lock by a non Muxer

The use of taking threaddumps • Hanging threads will be diagnosed as STUCK • Default time to get STUCK is 600 secs • Thread has not completed it’s work

WebLogic Log and Debug options • Logging services are important for diagnostics • WebLogic produces output by using a Message Catalog • Using the standard java.util.logging • Can be extended using log4j modules • All kinds of events, like startup/shutdown, appl.deployments, subsystem failures are written to logs

WebLogic Log and Debug options

WebLogic Log and debug methods

WebLogic Log and Debug options • Several severity levels like: • TRACE, DEBUG,INFO .. • Important to set the proper level • For FMW specific components, ODL-HANDLER can be used

WebLogic Log and Debug options • Subsystem logs can be from: • HTTP • Transaction Manager • JDBC • JMS server • WebLogic Auditing provider

WebLogic debug options • Several Subsystem can be debugged : • HTTP • JTA • JDBC • JMS server • Security/SSL Seems obvious, but only use it when it’s neccessary

WebLogic debug options • JDBC debug • Setting debug options in the AdminConsole

WebLogic Log4j • OpenSource Log engine from Apache • Enable the LogMBean.isLog4jLoggingEnabled attribute • In WLST: cmo.setLog4jLoggingEnabled(true) • Add WebLogic-specific Log4j classes, WL_HOME/server/lib/wllog4j.jar and the log4j.jar file to the server CLASSPATH

WebLogic Log4j • Enable log4j in AdminConsole

WebLogic debug options • Or use –D options in startup script -Dweblogic.debug.DebugJDBCSQL=true Prints information about all JDBC methods invoked, arguments and return values, and thrown exceptions Depending what item you want to debug: connection leaks, queries

WebLogic debug options • Or use WLST serverConfig() or domainConfig() debug = getMBean('/Servers/mserver/ServerDebug/mserver') debug.setDebugJDBCInternal(true)

Oracle diagnostic Logging (ODL) • As an extension on the default logging handlers • Comes with FMW installation, not WebLogic default • Logging view and config in Fusion Middleware Control • Can be set to several trace levels • Output in servername-diagnostic.log

WebLogic JDBC common errors • Wrong or missing drivers in Classpath • Connection errors by: • Usernames/passwords • Connection leaks by bad apps • Set Inactive timeout to a value (def is 0) • Insufficient connection parameters • Max Capacity, Reserve Time Out(-1) • Invalid JDBC URLS • weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception wa…

WebLogic and JVM diagnostics based on JRockit JDK

The JVM • JVM is a byte code interpreter • JVM heap is the area of memory used by the JVM

JVM from code to machine • A Java application runs a piece of code • Code enters the JVM ( JRockit, HotSpot) • It enters several stages: • Operations • Data structure • Transformation ( to bytecode) Java code JJVVMM Operatons Data Structure Transformation 110110100

JVM from code to machine • Crash can occur during • Compile process • Cause: invalid native code • Internal Code – Garbage Collect ( Segmentation Fault) • Cause: GC Fault leading into bad memory access • Long running GC can result in paging an CPU overload • Application code • Like 3rd Party drivers • (JDBC, MQ libraries) • Software modules

JVM from code to machine • Runtime behaviour: o JVM runs as an OS process o Execute work through threading model (concurrency) o Garbage Collector • Each Weblogic Server (instance) runs inside a separate JVM (instance).

JVM troubleshoot • What to look for in java thread dumps: • Low Memory Detector • (Daemon thread to detect and report low memory conditions) • CompilerThreadn • (Daemon thread/s to compile optimize byte code) • Signal Dispatcher • (Daemon thread to respond OS signals)

JVM troubleshoot • Finalizer • Daemon thread to handle objects in finalizer queue • Reference Handler • Daemon thread to handle objects in reference queue • VM Thread • Main thread from JVM • GC task thread#n • Thread/s to perform garbage collection

• JRockit • JRockit Mission Control • Grinder • HotSpot • JConsole • JVisualVM JVM tools

JRockit Mission Control • JRockit Mission Control provides: • Console • Live JVM statistics • Graphs and metrics about local or remote systems • Basic JMX interaction with MBeans • Runtime data • An in-depth look at live metrics, like memory, CPU and stack trace on specific threads • Profile methods or exceptions and execute commands

JRockit Mission Control • Memory Leak Detector • Shows overall trend • The growth rate of each type (class) is shown, rather than just the percentage of the heap they take up. • No common classes from showing at the top (like char[] arrays) • Visual representation of hot objects in the heap • By type • See how each class is referenced by other classes • By instance • Can also see how each instance of each class is referenced • Shows where objects are allocated • See stack trace of each method holding an instance and what percentage of instances are held by each method.

WebLogic Diagnostics Framework

WLDF • Framework for monitoring and diagnostics • Consolidated access for MBean data, instrumentation, logging, debugging,

WLDF • Source for monitoring data • Server application instrumentation notifications based on the output : log, email, SNMP, JMX, JMS • Request dyeing and tracing • Data archiving for later access • WLDF Dashboard

• Key Features • Gathering input from JMX MBeans • WebLogic subsystems and application code can be instrumented to provide events and watches • Notifications can be defined for events and watches • Console extension for monitoring of current and historical data • Data archiving WLDF

WLDF • Can be configured on filesystem or Database • Configure with Data retirement

WLDF • Create Diagnostic Module • Collect Metrics. A collection includes: • The MBean type to query • The specific MBean instance names to query (all instances, • by default) • The MBean attributes to collect (all attributes, by default) • How often to gather data • Create Watches Notifications

Performance Terms and Definitions • Performance : How system’s response time and throughput are affected by adding load. • Capacity : Maximum threshold a system is under a given set of conditions. • Scalability : how does a system responds to increasing load by adding additional resources. • Key indicators are : • Response time • Throughput

Performance objectives • Locate and minimize bottlenecks in Your System • Gather info about Application(s) • Concurrent number of users and interfaces. • Number and size of app requests. • Amount of data and its consistency. • Determining your target CPU utilization • Tune your Database(s) ( client or repos) • Tune WebLogic Server Performance Parameters • Tune Your JVM • Tune the Operating System • Tuning WebLogic Persistent Store

Tune WebLogic Server • Investigate and analyze! • Look at the following

Native I/O and muxers • Enable by default • Will use a O/S threadpool instead from “borrowing” from WebLogic def exec.Q • Native muxers use platform-specific native binaries to read data from sockets • Non native or Java sockets : • Uses pure Java to read data from sockets. • Blocks until there is data to be read from a socket(can cause issues on server level)

WebLogic Thread Count • Represents the number of simultaneous operations performed by apps that use the execute queue. Application WebLogic Socket Muxer WebLogic Kernel assigns Execute Req to ExecuteThread 1 Execute Request ExecuteThread 1 handles the request (for example executes a JDBC request) WebLogic Kernel handles self tuning ThreadPool, Workmanagers, allocations The app now handles the thread itself and gives back result

WebLogic Thread Count • Do not set this too high; can impact you performance • Default contains 15 threads • Can be changed in config.xml • Leave this default

Applications and Execute Queues • Default All apps use all execute queues • Configuring multiple execute queues can provide additional control for applications. • By using multiple execute queues, selected applications have access to a fixed number of execute threads, regardless of the load on WebLogic Server.

How to enable • To use user-defined execute queues include the use81-style-execute-queues sub-element in the config.xml file • Validation of config.xml will fail, set • setDomainEnv.sh -Dweblogic.configuration.schemaValidationEnabled=false • config.xml server nameWLS12cExaSvr1/name …….. use81-style-execute-queuestrue/use81-style-execute-queues

Threads as SocketReaders • ThreadPoolPercentSocketReaders • Default set to 33 ( range is 1-99) • Allocating threads as readers can increase accepting requests

WebLogic ChunkSize • Piece of memory for read and write data to sockets • A pool of chunks is maintained • For applications with large amounts of data • Parameters are: • weblogic.Chunksize—Size of a chunk (in bytes). Set to the network's maximum transfer unit (MTU), after subtracting from the value any Ethernet or TCP header sizes • weblogic.utils.io.chunkpoolsize—Sets the maximum size of the chunk pool. Monitor the CPU profile or use a memory/ heap profiler for call stacks invoking the constructor weblogic.utils.io.Chunk. • weblogic.PartitionSize—Sets the number of pool partitions used (default is 4). Partitioning the thread pool spreads the potential for contention over more than one partition To be set in the setDomainEnv.sh

WebLogic Backlog • Specifies how many TCP connections can be buffered. These are on the TCP stack but not yet accepted by an app. • Default is 50 • Maximum depends on O/S

WebLogic Backlog • When to tune? • When getting “connection refused” in high throughput apps ( SOA composites) • If tuning, increase by 25% of it’s default until errors dissapear

JDBC Connectionpool tuning • Prepared Statement Cache • Keeps compiled SQL statements in memory • Set to LRU • A data source with 10 connections deployed on 2 servers, with the Statement Cache to 10 (default), 200 cursors can be used on the database for these cached statements • MaxCapacity • Max of physical connections • Db drivers can limit connections • Number should equal number of client sessions

Tuning Startup • In some cases it takes ages after startup to continue • Generate a stackup

Tuning Startup • This happens when using default seed generator: • Use of dev/urandom during startup in stead of random • Specifiy in startup properties - Djava.security.egd=file:///dev/urandom • Or in $JAVA_HOME/jre/lib/security/java.security securerandom.source=file:/dev/./urandom

Tuning the JVM • Most aspects of JVM tuning relate to: • Sufficient memory heap • An efficient garbage collection scheme • Choice between two Unix threading models: green and native threads

Tuning the JVM • Most important to focus on: • Garbage collection scheme • Memory heap • The Java heap is a runtime repository for live objects, dead objects and free memory. • The JVM heap size determines how often and how long the VM spends collecting garbage • Large heap size : • full garbage collection slower and less frequent • Small heap size • full garbage collection is faster and frequency is more.

• Goal is: Tuning the JVM • Minimize Garbage Collect • Set heapsize that GC does no happen all the time

Tuning the JVM • Investigate with –verbosegc option • How often does GC run • How long does it take? • Full garbage collection should not take longer than 3 to 5 seconds. Lower heap if major GC time is greater.

Tuning the JVM • Set Xms and Xmx parameters to specify initial and Max HeapSize • Xmx is the Java Object Heap, not the System Heap – which is larger • Setting these values equal will cause no overhead of heapmanagement • PermSize and MaxPermSize For stored classes, methods

Oracle Fusion Middleware basis Let’s keep in touch!! • FMW basis: de Applicatie Server @MNEMONIC01 @Qualogy_news @QAFE • Wat is een Applicatie Server? Enterprise Management@@@ • Wat doet een Applicatie Server? • Welke Applicatie Server gebruikt Oracle?

Oracle Fusion Middleware basis Let’s keep in touch!! • FMW basis: de Applicatie Server nl.linkedin.com/in/mschldmr • Wat is een Applicatie Server? Enterprise Management@@@ • Wat doet een Applicatie Server? • Welke Applicatie Server gebruikt Oracle?

Oracle Fusion Middleware basis Let’s keep in touch!! • FMW basis: de Applicatie Server http://www.qualogy.com • Wat is een Applicatie Server? Enterprise Management@@@h • Wat doet een Applicatie Server? • Welke Applicatie Server gebruikt Oracle?

Oracle WebLogic Diagnostics & Perfomance tuning

In this document

More Related Content

What's hot

Viewers also liked

Similar to Oracle WebLogic Diagnostics & Perfomance tuning

More from Michel Schildmeijer

Recently uploaded

Oracle WebLogic Diagnostics & Perfomance tuning