Download to read offline
Uploaded byEdge AI and Vision Alliance
“Unlocking Visual Intelligence: Advanced Prompt Engineering for Vision-language Models,” a Presentation from LinkedIn Learning
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2025/09/unlocking-visual-intelligence-advanced-prompt-engineering-for-vision-language-models-a-presentation-from-linkedin-learning/ Alina Li Zhang, Senior Data Scientist and Tech Writer at LinkedIn Learning, presents the “Unlocking Visual Intelligence: Advanced Prompt Engineering for Vision-language Models” tutorial at the May 2025 Embedded Vision Summit. Imagine a world where AI systems automatically detect thefts in grocery stores, ensure construction site safety and identify patient falls in hospitals. This is no longer science fiction, as companies today are building powerful applications that integrate visual content with textual data to understand context and act intelligently. In this talk, Zhang delves into vision-language models (VLMs), the core technology behind these intelligent applications, and introduce the Pentagram framework, a structured approach to prompt engineering that significantly improves VLM accuracy and effectiveness. Zhang shows, step-by-step, how to use this prompt engineering process to create an application that uses a VLM to detect suspicious behaviors such as item concealment in grocery stores. She also explores the broader applications of these techniques in a variety of real-world scenarios. You’ll discover the possibilities of vision-language models and learn how to unlock their full potential.
“Unlocking Visual Intelligence: Advanced Prompt Engineering for Vision-language Models,” a Presentation from LinkedIn Learning
- 1. Unlocking Visual Intelligence: AdvancedPrompt Engineering for Vision-Language Models Alina Li Zhang Senior Data Scientist LinkedIn Learning
- 2. © 2025 LinkedInLearning Advanced Prompt Engineering for Vision-Language Models What will happen in the next second? © 2025 Linkedin Learning 2
- 3. © 2025 LinkedInLearning Beyond Text: The Era of AI with Eyes GPT-4o: “The cat is likely to knock over the glass of water. Cats are known for their playful and mischievous behavior, and this one appears to be testing the glass with its paw.” 3
- 4. © 2025 LinkedInLearning What are Vision-Language Models (VLMs)? “Sight turned into insight; Seeing became understanding; Understanding led to action. ” – Fei Fei Li 4
- 5. © 2025 LinkedInLearning Talk to AI: What is Prompt Engineering? 5 Prompt engineering is the art and science of designing effective prompts to guide AI models toward generating accurate, relevant, and desired outputs. Why prompt engineering matters: ● Improving output quality and relevance ● Enhancing control and predictability ● Enabling flexibility and reusability
- 6. © 2025 LinkedInLearning The Pentagram Framework for Prompt Engineering 6
- 7. © 2025 LinkedInLearning Prompt Engineering: Persona 7
- 8. © 2025 LinkedInLearning Prompt Engineering: Context 8
- 9. © 2025 LinkedInLearning Prompt Engineering: Task 9
- 10. © 2025 LinkedInLearning Prompt Engineering: Output 10
- 11. © 2025 LinkedInLearning Prompt Engineering: Constraint 11
- 12. © 2025 LinkedInLearning The Pentagram Framework for Shop Safe AI 12 How to instruct AI models to detect suspicious behaviors in a grocery store?
- 13. © 2025 LinkedInLearning Shop Safe AI: Persona 13 Persona You are an AI security analyst specializing in monitoring grocery store surveillance footage. Your expertise lies in detecting suspicious behaviors such as theft, item concealment, and price tag tampering.
- 14. © 2025 LinkedInLearning Shop Safe AI: Context 14 Context You operate within a real-time video surveillance system for retail environments. Users will provide one or more extracted frames from security camera footage for analysis. The store layouts typically include aisles, checkout zones, and product shelves.
- 15. © 2025 LinkedInLearning Shop Safe AI: Task 15 Task Analyze the provided frame(s) to identify any suspicious or unlawful behavior. For each detected incident, determine the subject involved, describe their actions, assess the risk level, and recommend appropriate responses.
- 16. © 2025 LinkedInLearning Shop Safe AI: Output 16 Output Present your findings in a structured, dashboard-style alert format using clear and concise language. Incorporate relevant emojis to enhance readability.
- 17. © 2025 LinkedInLearning Shop Safe AI: Output 17 For example: Theft Alert – Self-Checkout Zone ● Subject: Male, dark hoodie, jeans ● Item: Yellow packaged goods ● Behavior: Running past self-checkout area with item, no scan recorded ● Confidence: 98.9% ● Risk Level: HIGH ● Action: Initiate immediate floor alert. Block exit routes if safe. Notify security or law enforcement Few-shot learning
- 18. © 2025 LinkedInLearning Shop Safe AI: Constraint 18 Constraint ● Rely only on visible evidence and avoid assumptions. ● Keep reports clear and concise. ● Follow privacy and security policies. ● Focus on actionable insights.
- 19. © 2025 LinkedInLearning Testing: Concealment Alert 19
- 20. © 2025 LinkedInLearning Testing: Price Tag Tampering 20
- 21. © 2025 LinkedInLearning AI Eyes on the World: More Applications 21
- 22. © 2025 LinkedInLearning Conclusions 22 “My sword I leave to him who can wield it.” – Charlie Munger ● What are VLMs? ● What is the Pentagram Framework for prompt engineering? ● Practical example: How to craft the Pentagram prompt for a Shop Safe AI system?
- 23. © 2025 LinkedInLearning What’s Next? 23