DEV Community

Cover image for Docker Model Runner Cheatsheet 2025
Ajeet Singh Raina
Ajeet Singh Raina

Posted on

Docker Model Runner Cheatsheet 2025

📑 Table of Contents

  1. What is Docker Model Runner?
  2. 🚀 Quick Setup Guide
  3. 📋 Essential Commands
  4. 🔗 API Integration
  5. 🐳 Docker Compose Integration

What is Docker Model Runner?

Docker Model Runner is a new feature integrated into Docker Desktop that enables developers to run AI models locally with zero setup complexity. Built into Docker Desktop 4.40+, it brings LLM (Large Language Model) inference directly into your GenAI development workflow.

Key Benefits

  • ✅ No extra infrastructure - Runs natively on your machine
  • ✅ OpenAI-compatible API - Drop-in replacement for OpenAI calls
  • ✅ GPU acceleration - Optimized for Apple Silicon and NVIDIA GPUs
  • ✅ OCI artifacts - Models distributed as OCI artifacts
  • ✅ Host-based execution - Maximum performance, no VM overhead

🚀 Quick Setup Guide

Prerequisites

  • Docker Desktop 4.40+ (4.41+ for Windows GPU support)
  • macOS: Apple Silicon (M1/M2/M3) for optimal performance
  • Windows: NVIDIA GPU (for GPU acceleration)
  • Linux: Docker Engine with Model Runner

Enable Docker Model Runner

Docker Desktop (GUI)

  1. Open Docker Desktop Settings
  2. Navigate to Features in development → Beta
  3. Enable "Docker Model Runner"
  4. Apply & Restart

Docker Desktop (CLI) 

# Enable Model Runner docker desktop enable model-runner # Enable with TCP support (for host access) docker desktop enable model-runner --tcp 12434 # Check status docker desktop status
Enter fullscreen mode Exit fullscreen mode

Docker Engine (Linux) 

sudo apt-get update sudo apt-get install docker-model-plugin
Enter fullscreen mode Exit fullscreen mode

📋 Essential Commands

Model Management

Pull Models 

# Pull latest version docker model pull ai/smollm2
Enter fullscreen mode Exit fullscreen mode

List Models 

# List all local models docker model ls
Enter fullscreen mode Exit fullscreen mode

Remove Models 

# Remove specific model docker model rm ai/smollm2
Enter fullscreen mode Exit fullscreen mode

Running Models

Interactive Mode 

# Quick inference docker model run ai/smollm2 "Explain Docker in one sentence"
Enter fullscreen mode Exit fullscreen mode

Model Information 

# Inspect model details docker model inspect ai/smollm2
Enter fullscreen mode Exit fullscreen mode

🔗 API Integration

OpenAI-Compatible Endpoints

From Containers 

# Base URL for container access http://model-runner.docker.internal/engines/llama.cpp/v1/
Enter fullscreen mode Exit fullscreen mode

From Host (with TCP enabled) 

# Base URL for host access http://localhost:12434/engines/llama.cpp/v1/
Enter fullscreen mode Exit fullscreen mode

Chat Completions API

cURL Example 

curl http://localhost:12434/engines/llama.cpp/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "ai/smollm2", "messages": [ { "role": "system", "content": "You are a helpful coding assistant." }, { "role": "user", "content": "Write a Docker Compose file for a web app" } ], "temperature": 0.7, "max_tokens": 500 }'
Enter fullscreen mode Exit fullscreen mode

Python Example 

import openai # Configure client for local Model Runner client = openai.OpenAI( base_url="http://model-runner.docker.internal/engines/llama.cpp/v1", api_key="not-needed" # Local inference doesn't need API key ) # Chat completion response = client.chat.completions.create( model="ai/smollm2", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain containerization benefits"} ], temperature=0.7, max_tokens=200 ) print(response.choices[0].message.content)
Enter fullscreen mode Exit fullscreen mode

Node.js Example 

import OpenAI from 'openai'; const openai = new OpenAI({ baseURL: 'http://model-runner.docker.internal/engines/llama.cpp/v1', apiKey: 'not-needed' }); async function chatWithModel() { const completion = await openai.chat.completions.create({ model: 'ai/smollm2', messages: [ { role: 'system', content: 'You are a DevOps expert.' }, { role: 'user', content: 'Best practices for Docker in production?' } ], temperature: 0.8, max_tokens: 300 }); console.log(completion.choices[0].message.content); }
Enter fullscreen mode Exit fullscreen mode

🐳 Docker Compose Integration 

services: chat: image: my-chat-app depends_on: - ai_runner ai_runner: provider: type: model options: model: ai/smollm2
Enter fullscreen mode Exit fullscreen mode

🐳 Docker Model Management Endpoints 

POST /models/create GET /models GET /models/{namespace}/{name} DELETE /models/{namespace}/{name}
Enter fullscreen mode Exit fullscreen mode

OpenAI Endpoints: 

GET /engines/llama.cpp/v1/models GET /engines/llama.cpp/v1/models/{namespace}/{name} POST /engines/llama.cpp/v1/chat/completions POST /engines/llama.cpp/v1/completions POST /engines/llama.cpp/v1/embeddings
Enter fullscreen mode Exit fullscreen mode

Top comments (0)