Ways for Java CPU Profiling
In order to profile Java processes and get cpu usage hot points(better shown with flame graph), there’re 3 popular ways:
perf + perf-map-agent + FlameGraph
Linux perf tool and Gregg’s FlameGraph can do the job well, but as Java has a JVM virtual machine and JIT in-line feature, perf can’t get Java’s stack trace directly, here perf-map-agent comes to help.
perf-map-agent aims to provide a map file for symbols of in-line code.
XX:+PreserveFramePointershould be added when launce jvm as well.
There’re already many good tutorials and blogs about this method, so we won’t go deeper here.
eBPF + perf-map-agent + FlameGraph
Use eBPF instead of perf tool can alse work well, and it cost lower overhead theoretically. I’ve written another blog to share the detail about this (How To Profile Java Program With eBPF/bcc Tool).
jvm-profiling-tools/async-profiler is an open-source low overhead sampling profiler for Java.
For CPU profiling, it uses perf with
AsyncGetCallTrace, for visualization it has flame graph support out of box.
async-profiler already support profiling Java in a container, but I still met some obstacles while trying to use it.
It’s possible to profile both within the Docker container or from the host system, I tried both and both works well. Here I introduce profile within the container.
Easy to Use
Just download the async-profiler in place, it’s quite easy to use:
For CPU profiling, you may get some flame graphs like this.
When I use async-profiler the first time inside the contianer, I got some permission errors.
And in my error logs, I found this:
perf_event_paranoid? It’s a
sysctl file used to control the
permission of unprivileged user to use perf_event.
And you can’t modify the file directly in the container, because it’s read-only:
You have to do this command on the host system, in which case will definitly affect all containers on this host, which leverages some kind of security risks.
In addition, you may need to modify seccomp profile
or disable it altogether with
As my containers are running in Kubernetes cluster, which disables
default, I no longer need to do this.
For privileged user
As my Java process runs as unprivileged, I have to deal with the
perf_event_paranoid issue, however, if you run Java process as root user or
init process, you only need to add
SYS_ADMIN capabilities to your container.
(This blog may
help in this case.)