Log user feedback

Logging user feedback and scoring traces is a crucial aspect of evaluating and improving your agent. By systematically recording qualitative or quantitative feedback on specific interactions or entire conversation flows, you can:

1. Track performance over time
2. Identify areas for improvement
3. Compare different model versions or prompts
4. Gather data for fine-tuning or retraining
5. Provide stakeholders with concrete metrics on system effectiveness

Logging user feedback using the SDK

You can use the SDKs to log user feedback and score traces:

1
import { Opik } from "opik";
2
3
const client = new Opik();
4
5
// Create a new trace
6
const trace = client.trace({
7
name: "my_trace",
8
input: { "input": "Hi!" },
9
output: { "output": "Hello!" },
10
});
11
12
// Log feedback scores to an existing trace
13
await client.logTracesFeedbackScores([
14
{ id: trace.id, name: "overall_quality", value: 0.9, reason: "Good answer" },
15
{ id: trace.id, name: "coherence", value: 0.8 }
16
]);

Annotating Traces through the UI

To annotate traces through the UI, you can navigate to the trace you want to annotate in the traces page and click on the Annotate button. This will open a sidebar where you can add annotations to the trace.

You can annotate both traces and spans through the UI, make sure you have selected the correct span in the sidebar.

Once a feedback scores has been provided, you can also add a reason to explain why this particular score was provided. This is useful to add additional context to the score.

If multiple team members are annotating the same trace, you can see the annotations of each team member in the UI in the Feedback scores section. The average score will be displayed at a trace and trace level.

Online evaluation

You don’t need to manually annotate each trace to measure the performance of your agents! By using Opik’s online evaluation feature, you can define LLM as a Judge metrics that will automatically score all, or a subset, of your production traces.

Next steps

You can go one step further and:

1. Create an offline evaluation to evaluate your agent before it is deployed to production
2. Score your agent in production to track and catch specific issues with your agent
3. Use annotation queues to organize your traces for review and labeling by your team of experts
4. Checkout our LLM as a Judge metrics