Flurry Analytics

🎥 Link to full mini-documentary in the first comment 👇👇 We sat down with Vikram Sreekanti, Co-founder & CEO of RunLLM, UC Berkeley College of Engineering Ph.D. graduate, and former researcher at the Berkeley RISE Lab, where he worked on cloud infrastructure, serverless systems, and data platforms with Professors Joe Hellerstein and Joseph Gonzalez — before co-founding RunLLM with them and Chenggang Wu. Vikram's take? AI by itself isn’t a silver bullet. You can’t just throw a model at a problem and expect results. The real challenge is building a thoughtful application of that technology, and that’s actually the hardest part. Hear from Vikram and other experts on what it takes to assess and adopt AI in a way that actually works for companies. #AI #LLM #EnterpriseAI #TrustworthyAI #RunLLM #BerkeleyRISE #SupportEngineering #MachineLearning #AIProductDesign #VikramSreekanti

I've spent about a decade deeply understanding Apple's approach to product design, usability and all the things since spending time at Flurry Analytics, which saw app usage **every day** on 2 billion smartphones, most of them iPhones. I had unusual access, and it was pretty cool if I'm being honest. Anyway, their approach to layering in AI is very "Apple" which means they have to consider myriad use cases on a massive existing multi-product surface area and they will not rush into it. Remember, Apple is a hardware company first which means they "measure twice and cut once" (maybe even three times). In complex production with massive supply chains, you're working multiple years out, so little problems become existential problems. For people who do "run-and-gun" fast cycle iterative software work, it's very, very different. So it doesn't surprise me that they're first looking for the intersections where AI can add outsized, obvious value and layering those experiences, first, in a slow and measured way. At the same time, a hardware-first company, they are not going to have the leading AI lab building the most cutting edge models. In the early days of AI, they will need that trust, however, as users are experiencing what LLMs can do for the first time through Apple products. And that experience needs to deliver.

Evaluating LLMs: A Conversation with Joseph Gonzalez Our CEO and cofounder, Lukas Biewald, recently sat down with Joseph Gonzalez, EECS Professor at UC Berkeley and Co-Founder at RunLLM, to discuss the research he and his team have done on evaluating LLMs. Here are some of the highlights from this conversation: 🔹 Vibes-Based Model Evaluation Joseph introduced the concept of "vibes," which evaluates not just accuracy but also the style of a model’s response—whether it’s friendly, concise, or narrative-driven. This approach is transforming how LLMs are refined for human interaction. 👉 “Correctness is only part of the story—how a model communicates is just as critical. Llama is funnier and friendlier; OpenAI tends to be more formal and tends towards longer responses.” – Joseph Gonzalez 🔹 Chatbot Arena: A Global Benchmark for LLMs Chatbot Arena (lmarena.ai) lets users compare LLMs side-by-side, creating a community-driven leaderboard for open-source and commercial models. Using the Bradley-Terry approach to analyze pairwise comparisons, this initiative segments performance by tasks like creative writing, coding, or instruction following, helping developers optimize workflows for their specific application. 👉 “We want to democratize LLM evaluation—helping developers and the community improve models collaboratively.” – Joseph Gonzalez 🔹 Collaborative AI Evaluation and Development Joseph shared insights on how LLM evaluation is evolving to incorporate human feedback and community input, offering a deeper understanding of model strengths and weaknesses. This participatory approach ensures that LLMs meet user needs across diverse use cases and applications. 👉 “Human preference is about much more than accuracy—it’s about trust, interaction, and experience.” – Joseph Gonzalez 🎥 Check out the full episode to explore Joseph’s insights on advancing LLM evaluation, fostering community collaboration, and refining AI-human interactions. https://lnkd.in/exD3xSui