Run Qwen2.5 on Raspberry Pi 5 using Ollama

Qwen is a series of LLMs (large language models) developed by the Qwen team of Alibaba Group. At present, the SLM has been upgraded to the Qwen2.5 version. Qwen2.5 has multiple capabilities such as natural language understanding, text synthesis, visual understanding, audio understanding, human use, role-playing, and interaction as an agent. Qwen2.5, an advanced small language model, has gained significant attention for its efficiency and accuracy. With a model size of 398MB, qwen2.5-0.5b-int4 AI model is designed to be compact and fast, making it an ideal choice for Single Board Computers (SBC) with limited resources, such as on a Raspberry Pi. This article will explore how to install and run Qwen2.5 on a Raspberry Pi 5 development board using the Ollama runtime framework.

Raspberry Pi has become a popular choice for various applications, from educational projects to home media centers. The Raspberry Pi 5, the latest model in the series, offers improved performance and more resources, making it capable of running more demanding applications, including SLMs (small language models) like Qwen2.5.

Ollama, an open-source runtime framework, is designed to facilitate the deployment of LLMs on edge devices. It is lightweight, efficient, and supports a variety of models and architectures, making it an excellent choice for running qwen2.5 on a Raspberry Pi.
 

Installation

To run qwen2.5 on a Raspberry Pi 5 using Ollama, follow these steps:

1.Install Ollama: Begin by installing the Ollama runtime framework on your Raspberry Pi. You can do this by running the following command in the terminal:

2. Run qwen2.5-0.5b: Once Ollama and the qwen2.5 model are installed, you can run the model by executing the following command:
In verbose mode, the program will provide more output information to help developers or users understand detailed information during program operation.

Performance

One of the key considerations when running an NLP model like qwen2.5 on a Raspberry Pi is its performance. The qwen2.5 model, with its compact size and efficient design, is well-suited for the Raspberry Pi’s limited resources. However, it is essential to evaluate the model’s performance in terms of token speed to ensure that it meets the requirements of your specific application.
On a Raspberry Pi 5, the qwen2.5 model is expected to achieve a token speed of approximately 20 tokens per second, which is sufficient for many real-time applications, such as chatbots or text summarization.

token speed of qwen2.5-0.5b running on Raspberry Pi 5: 19.41 tokens/s

Conclusion

Running qwen2.5 on a Raspberry Pi 5 using the Ollama runtime framework is a straightforward process that can enable a wide range of SLMs on a compact and affordable device. With its small size, efficient design, and open-source license, qwen2.5 is an excellent choice for developers and enthusiasts looking to implement SLM on edge devices.

Learn More About SLMs on Raspberry Pi 5:

Run Small Language Models (Mathstral, phi 3, Llama 3.1, Gemma2 2b, Qwen) on Raspberry Pi 5

Run phi3.5 on Raspberry Pi 5 using Ollama

Run Gemma2 on Raspberry Pi 5 using Ollama