What could cause `{removed_failing_handler,'Elixir.Logger'}`

Questions / Help
,
1

In production we get {removed_failing_handler,'Elixir.Logger'} across multiple Elixir projects. I don’t get a stacktrace or anymore information than that. After that message we don’t see anymore logs until the app is redeployed. I’ve tried turning up our logger level to :debug because it looks like the erlang logger only includes more information at that level. Despite increasing the log level I still don’t see any more information or stacktrace. Any ideas what might cause this or how to get more information?

We are on elixir 1.14.1-otp-25 and erlang 25.3

We are also using LoggerJSON v5.1.2 with a formatter of LoggerJSON.Formatters.DatadogLogger

1 Like
2

The cause of this is fairly straightforward: if a log handler crashes, the system will stop sending events to it to try to stabilize things.

What kind of metadata / context are you setting in the logger config? I could see this happening if the formatter tries to encode a term to JSON that doesn’t support it…

1 Like
3

Thanks for looking at this.

My prod logger config is like this:

config :logger, :console, format: "$time $metadata[$level] $message\n", metadata: [:request_id] config :logger_json, :backend, metadata: :all, json_encoder: Jason, formatter: LoggerJSON.Formatters.DatadogLogger config :logger, backends: [LoggerJSON], level: :info, handle_sasl_reports: true

The JSON payload includes span_id, trace_Id, domain, erl_level, logger filename, logger line, logger method_name, logger thread_name, message, request_id, syslog hostname, syslog severity, syslog timestamp