Assessing the Reliability of Generative AI

Generative AI is transforming industries, but as adoption grows, so does the need for trust and reliability. Evaluation frameworks ensure that generative AI models perform as intended—not just in controlled environments, but in the real world. Key Insights from GCP Blog : Scalable Evaluation - new batch evaluation API allows you to assess large datasets efficiently, making it easier to validate model performance at scale. Customizable Autoraters - Benchmark automated raters against human judgments to build confidence in your evaluation process and highlight areas for improvement. Agentic Workflow Assessment - For AI agents, evaluate not just the final output, but also the reasoning process, tool usage, and decision trajectory. Continuous Monitoring - Implement ongoing evaluation to detect performance drift and ensure models remain reliable as data and user needs evolve. - Key Security Considerations: - Data Privacy: Ensure models do not leak sensitive information and comply with data protection regulations - Bias and Fairness: Regularly test for unintended bias and implement mitigation strategies[3]. - Access Controls:Restrict model access and implement audit trails to track usage and changes. - Adversarial Testing:Simulate attacks to identify vulnerabilities and strengthen model robustness **My Perspective: ** I see robust evaluation and security as the twin pillars of trustworthy AI. #Agent Evaluation is Evolving : Modern AI agent evaluation goes beyond simple output checks. It now includes programmatic assertions, embedding-based similarity scoring, and grading the reasoning path—ensuring agents not only answer correctly but also think logically and adapt to edge cases. Automated evaluation frameworks, augmented by human-in-the-loop reviewers, bring both scale and nuance to the process. - Security is a Lifecycle Concern: Leading frameworks like OWASP Top 10 for LLMs, Google’s Secure AI Framework (SAIF), and NIST’s AI Risk Management Framework emphasize security by design—from initial development through deployment and ongoing monitoring. Customizing AI architecture, hardening models against adversarial attacks, and prioritizing input sanitization are now standard best practices. - Continuous Improvement: The best teams integrate evaluation and security into every stage of the AI lifecycle, using continuous monitoring, anomaly detection, and regular threat modeling to stay ahead of risks and maintain high performance. - Benchmarking and Transparency: Standardized benchmarks and clear evaluation criteria not only drive innovation but also foster transparency and reproducibility—key factors for building trust with users and stakeholders. Check GCP blog post here: [How to Evaluate Your Gen AI at Every Stage](https://lnkd.in/gDkfzBs8) How are you ensuring your AI solutions are both reliable and secure?

Generative AI systems are increasingly evaluated for their social impact, but there's no standardized approach yet. This paper from June 2023 presents a framework for evaluating the social impact of generative AI systems, catering to researchers and developers, third-party auditors and red-teamers, and policymakers. Social impact is defined by the authors "as the effect of a system on people and communities along any timeline with a focus on marginalization, and active, harm that can be evaluated." The framework defines 7 categories of social impact: - bias, stereotypes, and representational harms; - cultural values and sensitive content; - disparate performance; - privacy and data protection; - financial costs; - environmental costs; - data and content moderation labor costs. E.g., the paper explains that safeguarding personal information and privacy relies on proper handling of training data, methods, and security measures. The paper stresses that there is great potential for more comprehensive privacy evaluations of GenAI systems: - Addressing the issue of memorization of training examples. - Ensure that only lawfully obtained data is shared with third parties. - Prioritize individual consent and choices. GenAI systems are harder to evaluate without clear documentation, systems for obtaining consent (e.g., opt-out mechanisms), and appropriate technical and process controls. Rules for leveraging end-user data for training purposes are often unclear, and the immense size of training datasets makes scrutiny increasingly difficult. Therefore, privacy risk assessments should go beyond proxies, focusing on memorization, data sharing, and security controls, and require extensive audits of processes and governance. 5 overarching categories for evaluation in society are suggested: - trustworthiness and autonomy; - inequality, marginalization, and violence; - concentration of authority; - labor and creativity; - ecosystem and environment. Each category includes subcategories and recommendations for mitigating harm. E.g., the category of trustworthiness and autonomy includes "Personal Privacy and Sense of Self". The authors emphasize that the impacts and harms from the violation of privacy are difficult to enumerate and evaluate. Mitigation first should determine who is responsible for an individual’s privacy, but requires both individual and collective action. The paper points out that technical methods to preserve privacy in a GenAI system, as seen in privacy-preserving approaches to language modeling, cannot guarantee full protection. Improving common practices and better global regulation for collecting training data can help. By Irene Solaiman, Zeerak Talat, William Agnew, Lama Ahmad, Dylan Baker, Su Lin Blodgett, Hal Daumé III, Jesse Dodge, Ellie Evans, Sara Hooker, Yacine Jernite, Alexandra Sasha Luccioni, Alberto Lusoli, Margaret Mitchell, Jessica Newman, Marie-Therese Png, Andrew Strait, Apostol Vassilev

Search is (currently) the surface on which AI will affect human decision making at the greatest scale. But, we know AI Search hallucinates, as Google’s AI Search has advised users to “eat rocks," glue cheese to pizza, and that “doctors recommend smoking 2-3 cigarettes per day during pregnancy.” So, we ran 12,000 search queries across 7 countries, generating 80,000 real-time GenAI and traditional search results, to understand current global exposure to GenAI search. We then used a preregistered, randomized experiment on a large study sample to understand when humans trust AI Search. The results were surprising and a bit unnerving... 🚩 First, our study shows that GenAI search results are globally pervasive but vary greatly by topic. Over half of all Health (51%) and General Knowledge (56%) queries returned AI results while only 5% of Shopping and 1% of Covid queries returned AI results. The pervasiveness of AI in search results suggests we should be concerned with the conditions under which humans trust AI search. 🤔 🚩 Second, the format of the query predicts whether you get AI or traditional search results with questions returning GenAI answers 49% of the time, statements 16% of the time and navigational searches returning GenAI only 4% of the time. 🚩 Third, in the RCT, while participants trust GenAI search less than traditional search on average, reference links and citations significantly increase trust in GenAI, even when those links and citations are incorrect or hallucinated. In other words, the veneer of rigor in AI design creates trust even when references and links are not rigorous. 🤯 🚩 Uncertainty highlighting, which reveals GenAI’s confidence in its own conclusions, makes us less willing to trust and share generative information whether that confidence is high or low. 🚩 Positive social feedback increases trust in GenAI while negative feedback reduces trust. These results imply that GenAI interface designs can increase trust in inaccurate and hallucinated information and reduce trust when GenAI’s certainty is made explicit. 🚩 Trust in GenAI varies by topic and with users’ education, industry employment and GenAI experience, revealing which sub-populations are most vulnerable to GenAI misrepresentations. 🚩 Trust then predicts behavior as those who trust GenAI more click more and spend less time evaluating GenAI search results. These findings suggest directions for GenAI design to address the AI "trust gap.” The paper, coauthored with Haiwen Li, is linked in the first comment. We thank the MIT Initiative on the Digital Economy for support and are grateful to SerpApi for assistance with query scaling. As always, thoughts and comments highly encouraged! Wondering especially what Erik Brynjolfsson Edward McFowland III Iavor Bojinov John Horton Karim Lakhani Azeem Azhar Sendhil Mullainathan Nicole Immorlica Alessandro Acquisti Ethan Mollick Katy Milkman and others think!

I've been digging into the latest NIST guidance on generative AI risks—and what I’m finding is both urgent and under-discussed. Most organizations are moving fast with AI adoption, but few are stopping to assess what’s actually at stake. Here’s what NIST is warning about: 🔷 Confabulation: AI systems can generate confident but false information. This isn’t just a glitch—it’s a fundamental design risk that can mislead users in critical settings like healthcare, finance, and law. 🔷 Privacy exposure: Models trained on vast datasets can leak or infer sensitive data—even data they weren’t explicitly given. 🔷 Bias at scale: GAI can replicate and amplify harmful societal biases, affecting everything from hiring systems to public-facing applications. 🔷 Offensive cyber capabilities: These tools can be manipulated to assist with attacks—lowering the barrier for threat actors. 🔷 Disinformation and deepfakes: GAI is making it easier than ever to create and spread misinformation at scale, eroding public trust and information integrity. The big takeaway? These risks aren't theoretical. They're already showing up in real-world use cases. With NIST now laying out a detailed framework for managing generative AI risks, the message is clear: Start researching. Start aligning. Start leading. The people and organizations that understand this guidance early will become the voices of authority in this space. #GenerativeAI #Cybersecurity #AICompliance

Presentations of the FDA Digital Health Advisory Committee Meeting on Generative AI-Enabled Devices: Evaluating and Addressing Risks in Generative AI for Healthcare Regulatory Science Challenges of Generative AI Victor Garcia and aldo badano, Director, FDA, discussed the regulatory science challenges posed by generative AI-enabled devices. He highlights their commitment to innovation and development of open-source regulatory science tools. Generative AI’s ability to create novel outputs introduces unique risks, such as hallucinations, adaptive system oversight, and data diversity issues. He presented a use case of a generative AI-enabled radiology device, demonstrating challenges in benchmarking, expert evaluation, and model-based evaluation. He proposed strategies for evaluation, including using external datasets, expert oversight, and model-driven tests. He concluded by emphasizing the need for robust premarket and post-market evaluation frameworks to address the dynamic nature of generative AI models. Computational Pathology and Generative AI Faisal Mahmood, Associate Professor, Harvard University, presented his lab's work in computational pathology and its integration with generative AI. He detailed how large gigapixel pathology images are analyzed for early diagnosis, prognosis, and biomarker discovery. He introduced PathChat, a multimodal large language model trained on pathology data, which can generate diagnostic reports and adapt to resource-limited settings. He stressed the importance of bias mitigation and equity in deploying AI systems globally. Generative AI’s Role in Medical Imaging Parminder Bhatia, Chief AI Officer, GE Healthcare, provided insights into how generative AI and foundation models are revolutionizing medical imaging. He explained the unique characteristics of foundation models, such as their ability to handle multimodal data and perform diverse tasks with minimal additional training. To mitigate risks like hallucinations and output inconsistency, he recommended strategies such as ontology-based reasoning, visual grounding systems, and temperature control mechanisms. He emphasized the importance of preconfigured change control plan (PCCP) to safely manage updates and scalability of generative AI models. Evaluating Generative AI in Clinical Settings Pranav Rajpurkar, Assistant Professor, Harvard University, discussed methodologies for evaluating generative AI models in clinical applications. He emphasized the need for robust metrics to assess the safety and effectiveness of AI-generated outputs. He showcased MedVersa, a multimodal AI system capable of processing diverse medical images and generating comprehensive reports. He demonstrated its superior performance compared to specialized models and emphasized the value of human-centered evaluations, such as expert reviews and real-world usability studies. Video Link: https://lnkd.in/eH--UzNH #GenAI #Regulation #FDA