We revamped ‘Alerts’ from the ground up in the Observability module of our dashboard because of one haunting question: 'How long has it been broken?'
The new monitoring system now tracks 11 different metric types including errors, performance, token usage, and custom evaluations across projects. It has-
both static and dynamic percentage-change based alerting
multi-channel notifications via email and Slack
real-time data visualization with alert status indicators
We started with a blank canvas, exploring a range of observability platforms both classic software and LLM-native to learn what actually works for us. Sentry’s robust, signal-first approach inspired us, so we riffed on that and kept asking: what would genuinely help teams running AI apps every day?
That question guided us as we sketched flows, brainstormed ideas, and spent a few intense days planning, designing, and building the feature.
Last week, we rolled it out to all our users. And already, it has proactively prevented catastrophic issues turning potential disasters into opportunities for improvement, growth, and stronger customer trust.
Check how it observes your AI agent-
Docs- https://shorturl.at/13Tfj
Dashboard- https://shorturl.at/JkqlP
Top comments (0)