We run all our services in docker containers in a k8s cluster, as I suspect quite a few Datadog users do, and as such we have our standard boilerplate set to emit all logs from all sources (channeled through slf4j) as properly formatted single line JSON. This allows for the Datadog agent to pick up and properly index logs and deal with things like stack traces in a sane fashion.

For the most part, this works quite well, except for log messages printed to standard out by the datadog java agent. Since the slf4j infrastructure at that level is loaded and configured in a different classloader, we don't really have any way to influence it's output - it appears to just be using the stock simple slf4j logger implementation, which only really allows for coarse grained configuration of log levels.

Is there a mechanism by which we could have the java agent use logback or some other slf4j binding that outputs JSON instead? We have people getting woken up at all hours of the night due to errors generating alerts because the DD log infrastructure interprets anything that can't be parsed as JSON at the ERROR level. We've mitigated a bunch of this by adding -Ddatadog.slf4j.simpleLogger.defaultLogLevel=WARN to our JVM launch parameters, but there's still a bunch of stuff that occasionally gets through.