Posted on Sep 3

Maia - Multi-AI Agent Test Framework

Hey Dev community!
I want to share with you my recent open-source project which I am working on - Test Framework for testing Multi-AI Agent systems.

Website: Maia

Framework is written in Python and uses standard pytest approach.

The main features are:

Multi-Agent Simulation - Simulate conversations and interactions between multiple AI agents
Extensible Provider Model - Easily integrate with various AI model providers (e.g., LiteLLM, LangChain, CrewAI)
Built-in Assertions - A suite of assertions to verify agent behavior, including content analysis and participation checks
Dashboard for visualization - NextJS application to show test results for checking and debugging purpose.

You can use the framework for testing such scenarios like:

asking various models for the same thing and check the results
broadcasting a prompt and wait for the completion without user intervention (using not only CrewAI but also other providers!)
simulate tool calling, so checking if your AI Agent uses your tool in a proper way
much much more

As an example, please see how easy is to write a test:

class TestConversationSessions(MaiaTest): def setup_agents(self): self.create_agent( name="Alice", provider=GenericLiteLLMProvider(config={ "model": "ollama/mistral", "api_base": "http://localhost:11434" }), system_message="You are a weather assistant. Only describe the weather.", ) self.create_agent( name="Bob", provider=GenericLiteLLMProvider(config={ "model": "ollama/mistral", "api_base": "http://localhost:11434" }), system_message="You are an assistant who only suggests clothing.", ) @pytest.mark.asyncio async def test_agent_to_agent_conversation(self): session = self.create_session(["Alice", "Bob"]) # Alice initiates conversation with Bob  await session.agent_says("Alice", "Bob", "Given the weather: rainy and 20 degrees Celsius, what clothes should I wear?") response = await session.agent_responds("Bob") assert_agent_participated(session, "Bob") # Bob responds back to Alice  await session.agent_says("Bob", "Alice", f"Based on my info: {response.content}") response = await session.agent_responds("Alice") assert_agent_participated(session, "Alice")