Developing Training Metrics

Over the last year, I’ve seen many people fall into the same trap: They launch an AI-powered agent (chatbot, assistant, support tool, etc.)… But only track surface-level KPIs — like response time or number of users. That’s not enough. To create AI systems that actually deliver value, we need 𝗵𝗼𝗹𝗶𝘀𝘁𝗶𝗰, 𝗵𝘂𝗺𝗮𝗻-𝗰𝗲𝗻𝘁𝗿𝗶𝗰 𝗺𝗲𝘁𝗿𝗶𝗰𝘀 that reflect: • User trust • Task success • Business impact • Experience quality    This infographic highlights 15 𝘦𝘴𝘴𝘦𝘯𝘵𝘪𝘢𝘭 dimensions to consider: ↳ 𝗥𝗲𝘀𝗽𝗼𝗻𝘀𝗲 𝗔𝗰𝗰𝘂𝗿𝗮𝗰𝘆 — Are your AI answers actually useful and correct? ↳ 𝗧𝗮𝘀𝗸 𝗖𝗼𝗺𝗽𝗹𝗲𝘁𝗶𝗼𝗻 𝗥𝗮𝘁𝗲 — Can the agent complete full workflows, not just answer trivia? ↳ 𝗟𝗮𝘁𝗲𝗻𝗰𝘆 — Response speed still matters, especially in production. ↳ 𝗨𝘀𝗲𝗿 𝗘𝗻𝗴𝗮𝗴𝗲𝗺𝗲𝗻𝘁 — How often are users returning or interacting meaningfully? ↳ 𝗦𝘂𝗰𝗰𝗲𝘀𝘀 𝗥𝗮𝘁𝗲 — Did the user achieve their goal? This is your north star. ↳ 𝗘𝗿𝗿𝗼𝗿 𝗥𝗮𝘁𝗲 — Irrelevant or wrong responses? That’s friction. ↳ 𝗦𝗲𝘀𝘀𝗶𝗼𝗻 𝗗𝘂𝗿𝗮𝘁𝗶𝗼𝗻 — Longer isn’t always better — it depends on the goal. ↳ 𝗨𝘀𝗲𝗿 𝗥𝗲𝘁𝗲𝗻𝘁𝗶𝗼𝗻 — Are users coming back 𝘢𝘧𝘵𝘦𝘳 the first experience? ↳ 𝗖𝗼𝘀𝘁 𝗽𝗲𝗿 𝗜𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻 — Especially critical at scale. Budget-wise agents win. ↳ 𝗖𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻 𝗗𝗲𝗽𝘁𝗵 — Can the agent handle follow-ups and multi-turn dialogue? ↳ 𝗨𝘀𝗲𝗿 𝗦𝗮𝘁𝗶𝘀𝗳𝗮𝗰𝘁𝗶𝗼𝗻 𝗦𝗰𝗼𝗿𝗲 — Feedback from actual users is gold. ↳ 𝗖𝗼𝗻𝘁𝗲𝘅𝘁𝘂𝗮𝗹 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 — Can your AI 𝘳𝘦𝘮𝘦𝘮𝘣𝘦𝘳 𝘢𝘯𝘥 𝘳𝘦𝘧𝘦𝘳 to earlier inputs? ↳ 𝗦𝗰𝗮𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆 — Can it handle volume 𝘸𝘪𝘵𝘩𝘰𝘶𝘵 degrading performance? ↳ 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆 — This is key for RAG-based agents. ↳ 𝗔𝗱𝗮𝗽𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗦𝗰𝗼𝗿𝗲 — Is your AI learning and improving over time? If you're building or managing AI agents — bookmark this. Whether it's a support bot, GenAI assistant, or a multi-agent system — these are the metrics that will shape real-world success. 𝗗𝗶𝗱 𝗜 𝗺𝗶𝘀𝘀 𝗮𝗻𝘆 𝗰𝗿𝗶𝘁𝗶𝗰𝗮𝗹 𝗼𝗻𝗲𝘀 𝘆𝗼𝘂 𝘂𝘀𝗲 𝗶𝗻 𝘆𝗼𝘂𝗿 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀? Let’s make this list even stronger — drop your thoughts 👇

✨ New resource: a PM Performance Evaluation template Throughout my 15+ years as a PM, I’ve consistently felt that ladder-based PM performance evaluations seem broken, but I couldn’t quite find the words to describe why. Early on in my PM career, I was actually part of the problem — I happily created or co-created elaborate PM ladders in spreadsheets, calling out all sorts of nuances between what “Product Quality focus” looks like at the PM3 level vs. at the Sr. PM level. (looking back, it was a non-trivial amount of nonsense — and having seen several dozens of ladder spreadsheets at this point, I can confidently say this is the case for >90% of such ladder spreadsheets) So that led me to develop the Insight-Execution-Impact framework for PM Performance Evaluations, which you can see in the picture below. I then used this framework informally to guide performance conversations and performance feedback for PMs on my team at Stripe — and I have also shared this with a dozen founders who’ve adapted it for their own performance evaluations as they have established more formal performance systems at their startups. And now, you can access this framework as an easy to update & copy Coda doc (link in the comments). How to use this template as a manager? In a small company that hasn’t yet created the standard mess of elaborate spreadsheet-based career ladders, you might consider adopting this template as your standard way of evaluating and communication PM performance (and you can marry it with other sane frameworks such as PSHE by Shishir Mehrotra to decide when to promote a given PM to the next level e.g. GPM vs. Director vs. VP). In a larger company that already has a lot of legacy, habits, and tools around career ladders & perf, you might not be able to wholesale replace your existing system & tools like Workday. That is fine. If this framework resonates with you, I’d still recommend that you use it to actually have meaningful conversations with your team members around planning what to expect over the next 3 / 6 / 9 months and also to provide more meaningful context on their performance & rating. When I was at Stripe, we used Workday as our performance review tool, but I first wrote my feedback in the form of Insight - Execution - Impact (privately) and then pasted the relevant parts of my write-up into Workday. So that’s it from me. Again, the link to the template is in the comments. And if you want more of your colleagues to see the light, there’s even a video in that doc, in which I explain the problem and the core framework in more detail. I hope this is useful.

Over the last few years, you have seen me posting about Data Centric AI, why it is important, and how to implement it in your ML pipeline. I shared resources on a key step: building a Data Validation module, for which there are several libraries. Two drawbacks I observed in many libraries are: (i) the data validation/quality checks need to be manually developed, (ii) the quality checks do not support different data modalities. While investigating, I discovered a standard open-source library for Data-Centric AI called Cleanlab. Curious to learn more, I got on a call where their scientists, Jonas Mueller shared research on Confident Learning, an algorithm for *automated data validation* in a general-purpose way that works for all data modalities (including tabular, text, image, audio, etc). This blew my mind! The library has been updated with all sorts of automated data improvement capabilities, and I am excited to share what I tried it out for. Let me first explain Confident Learning (CL) - CL is a novel probabilistic approach that uses a ML model to estimate which data/labels are not trustworthy in noisy real-world datasets (see blogpost linked below for more theory). In essence, CL uses probabilistic predictions from any ML model you trained to perform the following steps: 📊 Estimate joint distribution of given, noisy labels and latent (unknown) true labels to fully characterize class-conditional label noise. ✂️ Find and prune noisy examples with label issues. 📉 Train a more reliable ML model on filtered dataset, re-weighting the data by the estimated latent prior. This data-centric approach helps you turn unreliable data into reliable models, regardless what type of ML model you are using. What can you do with Cleanlab: 📌 Detect common data issues (outliers, near duplicates, label errors, drift, etc) with a single line of code 📌 Train robust models by integrating Cleanlab in your MLOps/DataOps pipeline 📌 Infer consensus + annotator-quality for data labeled by multiple annotators 📌 Suggest which data to (re)label next via ActiveLab - a practical Active Learning algorithm to collect a dataset with the fewest total annotations needed to train an accurate model. To reduce data annotation costs, ActiveLab automatically estimates when it is more informative to re-label examples vs. labeling entirely new ones. Try improving your own dataset with this open-source library via the 5-minute tutorials linked on their github: https://lnkd.in/gWtgPUXw  (⭐ it to support free open-source software!) More resources: 👩🏻💻 Cleanlab website: https://cleanlab.ai/  👩🏻💻 Confident Learning blogpost: https://lnkd.in/gDKccShh  👩🏻💻 ActiveLab blogpost: https://lnkd.in/giXHaPBF PS: Did you know Google also uses Cleanlab to find and fix errors in their big speech dataset in a scalable manner. #ml #datascience #ai #data #datacentricai

Evaluating LLMs is hard. Evaluating agents is even harder. This is one of the most common challenges I see when teams move from using LLMs in isolation to deploying agents that act over time, use tools, interact with APIs, and coordinate across roles. These systems make a series of decisions, not just a single prediction. As a result, success or failure depends on more than whether the final answer is correct. Despite this, many teams still rely on basic task success metrics or manual reviews. Some build internal evaluation dashboards, but most of these efforts are narrowly scoped and miss the bigger picture. Observability tools exist, but they are not enough on their own. Google’s ADK telemetry provides traces of tool use and reasoning chains. LangSmith gives structured logging for LangChain-based workflows. Frameworks like CrewAI, AutoGen, and OpenAgents expose role-specific actions and memory updates. These are helpful for debugging, but they do not tell you how well the agent performed across dimensions like coordination, learning, or adaptability. Two recent research directions offer much-needed structure. One proposes breaking down agent evaluation into behavioral components like plan quality, adaptability, and inter-agent coordination. Another argues for longitudinal tracking, focusing on how agents evolve over time, whether they drift or stabilize, and whether they generalize or forget. If you are evaluating agents today, here are the most important criteria to measure: • 𝗧𝗮𝘀𝗸 𝘀𝘂𝗰𝗰𝗲𝘀𝘀: Did the agent complete the task, and was the outcome verifiable? • 𝗣𝗹𝗮𝗻 𝗾𝘂𝗮𝗹𝗶𝘁𝘆: Was the initial strategy reasonable and efficient? • 𝗔𝗱𝗮𝗽𝘁𝗮𝘁𝗶𝗼𝗻: Did the agent handle tool failures, retry intelligently, or escalate when needed? • 𝗠𝗲𝗺𝗼𝗿𝘆 𝘂𝘀𝗮𝗴𝗲: Was memory referenced meaningfully, or ignored? • 𝗖𝗼𝗼𝗿𝗱𝗶𝗻𝗮𝘁𝗶𝗼𝗻 (𝗳𝗼𝗿 𝗺𝘂𝗹𝘁𝗶-𝗮𝗴𝗲𝗻𝘁 𝘀𝘆𝘀𝘁𝗲𝗺𝘀): Did agents delegate, share information, and avoid redundancy? • 𝗦𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗼𝘃𝗲𝗿 𝘁𝗶𝗺𝗲: Did behavior remain consistent across runs or drift unpredictably? For adaptive agents or those in production, this becomes even more critical. Evaluation systems should be time-aware, tracking changes in behavior, error rates, and success patterns over time. Static accuracy alone will not explain why an agent performs well one day and fails the next. Structured evaluation is not just about dashboards. It is the foundation for improving agent design. Without clear signals, you cannot diagnose whether failure came from the LLM, the plan, the tool, or the orchestration logic. If your agents are planning, adapting, or coordinating across steps or roles, now is the time to move past simple correctness checks and build a robust, multi-dimensional evaluation framework. It is the only way to scale intelligent behavior with confidence.

Many teams overlook critical data issues and, in turn, waste precious time tweaking hyper-parameters and adjusting model architectures that don't address the root cause. Hidden problems within datasets are often the silent saboteurs, undermining model performance. To counter these inefficiencies, a systematic data-centric approach is needed. By systematically identifying quality issues, you can shift from guessing what's wrong with your data to taking informed, strategic actions. Creating a continuous feedback loop between your dataset and your model performance allows you to spend more time analyzing your data. This proactive approach helps detect and correct problems before they escalate into significant model failures. Here's a comprehensive four-step data quality feedback loop that you can adopt: Step One: Understand Your Model's Struggles Start by identifying where your model encounters challenges. Focus on hard samples in your dataset that consistently lead to errors. Step Two: Interpret Evaluation Results Analyze your evaluation results to discover patterns in errors and weaknesses in model performance. This step is vital for understanding where model improvement is most needed. Step Three: Identify Data Quality Issues Examine your data closely for quality issues such as labeling errors, class imbalances, and other biases influencing model performance. Step Four: Enhance Your Dataset Based on the insights gained from your exploration, begin cleaning, correcting, and enhancing your dataset. This improvement process is crucial for refining your model's accuracy and reliability. Further Learning: Dive Deeper into Data-Centric AI For those eager to delve deeper into this systematic approach, my Coursera course offers an opportunity to get hands-on with data-centric visual AI. You can audit the course for free and learn my process for building and curating better datasets. There's a link in the comments below—check it out and start transforming your data evaluation and improvement processes today. By adopting these steps and focusing on data quality, you can unlock your models' full potential and ensure they perform at their best. Remember, your model's power rests not just in its architecture but also in the quality of the data it learns from. #data #deeplearning #computervision #artificialintelligence

Measuring Success: How Competency-Based Assessments Can Accelerate Your Leadership If it’s you who feels stuck in your career despite putting in the effort. To help you gain measurable progress, one can use competency-based assessments to track skills development over time. 💢Why Competency-Based Assessments Matter: They provide measurable insights into where you stand, which areas you need improvement, and how to create a focused growth plan. This clarity can break through #career stagnation and ensure continuous development. 💡 Key Action Points: ⚜️Take Competency-Based Assessments: Track your skills and performance against defined standards. ⚜️Review Metrics Regularly: Ensure you’re making continuous progress in key areas. ⚜️Act on Feedback: Focus on areas that need development and take actionable steps for growth. 💢Recommended Assessments for Leadership Growth: For leaders looking to transition from Team Leader (TL) to Assistant Manager (AM) roles, here are some assessments that can help: 💥Hogan Leadership Assessment – Measures leadership potential, strengths, and areas for development. 💥Emotional Intelligence (EQ-i 2.0) – Evaluates emotional intelligence, crucial for leadership and collaboration. 💥DISC Personality Assessment – Focuses on behavior and communication styles, helping leaders understand team dynamics and improve collaboration. 💥Gallup CliftonStrengths – Identifies your top strengths and how to leverage them for leadership growth. 💥360-Degree Feedback Assessment – A holistic approach that gathers feedback from peers, managers, and subordinates to give you a well-rounded view of your leadership abilities. By using these tools, leaders can see where they excel and where they need development, providing a clear path toward promotion and career growth. Start tracking your progress with these competency-based assessments and unlock your full potential. #CompetencyAssessment #LeadershipGrowth #CareerDevelopment #LeadershipSkills

What's the best way to evaluate the effectiveness of leader development initiatives like coaching, mentoring and training? If we take a common framework like the Kirkpatrick Model, it clearly guides us to measure: 👉 Reaction: Did participants find the experience valuable or engaging? 👉 Learning: Did they acquire new knowledge, skills, or insights? 👉 Behavior: Did their actions or habits change as a result? 👉 Results: Did these changes lead to measurable organizational outcomes? The visual below provides us with a few more evaluation ideas and methods, which are helpful! I particularly like the focus on measuring success with objectives set at the start of the coaching programme (because it guides us to make sure the objectives are clear and realistic). The one I struggle with is "Impact on business performance In my experience, evaluating the direct link between leader development and business results (e.g., profits, savings, or productivity) is difficult and often misaligned with the true purpose of these initiatives. Leader development fosters long-term growth, enhances team dynamics, and shapes organizational culture—outcomes that don’t always translate into immediate business metrics. It’s also essential to manage expectations. If the primary goal of leader development is to see immediate improvements in business performance, it’s worth asking if those expectations are realistic. Initiatives like coaching and mentoring often result in intangible but powerful outcomes, such as: ✔️ Increased self-awareness ✔️ Improved team communication ✔️ Strengthened confidence and competency While these outcomes may not directly show up in quarterly metrics, they lay the foundation for sustained organizational success. This is why setting clear, measurable objectives at the start is so important. If the intended outcomes include changes like better communication or a shift in culture, these should be the focus of evaluation—not solely traditional business performance indicators. Leadership development IS NOT a quick fix for the bottom line. It IS an investment in the people and culture that drive long-term success. What methods or frameworks have you found helpful for evaluating #leadershipdevelopment? Leave your comments below 🙏 Image Source;: Jarvis J (2004) Research Gate

❗ Only 12% of employees apply new skills learned in L&D programs to their jobs (HBR).  ❗ Are you confident that your Learning and Development initiatives are part of that 12%? And do you have the data to back it up?  ❗ L&D professionals who can track the business results of their programs report having a higher satisfaction with their services, more executive support and continued and increased resources for L&D investments.    Learning is always specific to each employee and requires personal context. Evaluating training effectiveness shows you how useful your current training offerings are and how you can improve them in the future. What’s more, effective training leads to higher employee performance and satisfaction, boosts team morale, and increases your return on investment (ROI). As a business, you’re investing valuable resources in your training programs, so it’s imperative that you regularly identify what’s working, what’s not, why, and how to keep improving. To identify the Right Employee Training Metrics for Your Training Program, here are a few important pointers: ✅ Consult with key stakeholders – before development, on the metrics they care about. Make sure to use your L&D expertise to inform your collaboration. ✅Avoid using L&D jargon when collaborating with stakeholders – Modify your language to suit the audience. ✅Determine the value of measuring the effectiveness of a training program. It takes effort to evaluate training effectiveness, and those that support key strategic outcomes should be the focus of your training metrics. ✅Avoid highlighting low-level metrics, such as enrollment and completion rates. 9 Examples of Commonly Used Training Metrics and L&D Metrics 📌 Completion Rates: The percentage of employees who successfully complete the training program. 📌Knowledge Retention: Measured through pre- and post-training assessments to evaluate how much information participants have retained. 📌Skill Improvement: Assessed through practical tests or simulations to determine how effectively the training has improved specific skills. 📌Behavioral Changes: Observing changes in employee behavior in the workplace that can be attributed to the training. 📌Employee Engagement: Employee feedback and surveys post-training to assess their engagement and satisfaction with the training. 📌Return on Investment (ROI): Calculating the financial return on investment from the training, considering costs vs. benefits. 📌Application of Skills: Evaluating how effectively employees are applying new skills or knowledge in their day-to-day work. 📌Training Cost per Employee: Calculating the total cost of training per participant. 📌Employee Turnover Rates: Assessing whether the training has an impact on employee retention and turnover rates. Let's discuss in comments which training metrics are you using and your experience of using it. #MeetaMeraki #Trainingeffectiveness

As a young VC, I find myself diving into numerous books, each promising to offer a fresh perspective or insight. Yet, the challenge lies in truly absorbing and retaining the valuable lessons they contain. This changed when I discovered Shane Parrish’s Blank Sheet Method.....a straightforward, yet powerful approach that transformed my learning process. 🔹 Step 1: Set the Stage - Before starting any book, grab a blank sheet of paper. - On this sheet, outline what you already know about the topic. 🔹Step 2: Track Your Progress - At the end of each reading session, spend a few minutes updating your mind map using a different color to highlight new insights. 🔹 Step 3: Review and Reinforce - Before picking up the book again, go through your mind map to refresh your memory. - This review process helps solidify your grasp on what you’ve read and primes your brain to link upcoming ideas with what you already know. 🔹 Step 4: Build a Knowledge Vault - Keep these annotated sheets organized in a binder for easy access. - Regularly review them to reinforce your learning and connect concepts across various books and subjects. Why This Method Works Wonders: - Strengthens memory by recalling and building upon what you know. - Identifies missing pieces and clears up misconceptions. - Helps in connecting themes across disciplines - Stimulates unique thinking and insights - Periodic review solidifies information With each book, I find that my understanding grows not just in depth but in scope, creating a network of knowledge that extends far beyond a single subject. Have you tried using this or any other method for better retention? I’d love to hear what’s worked for you! #ReadingWisdom #LearningMethods #VentureLife #KnowledgeRetention