Test Execution Problems in MCP Codebase Insight

Issue Description

The test suite in MCP Codebase Insight is experiencing several critical issues that prevent proper test execution and coverage reporting.

Current Problems

1. Environment Setup Issues

   • Virtual environment activation problems
   • Environment variables not properly loaded from .env file
   • PYTHONPATH configuration issues affecting module imports

2. Test Execution Failures

   • Module import errors for mcp_codebase_insight package
   • Event loop conflicts in async tests
   • Test isolation problems between components
   • Coverage reporting not working as expected

3. Integration Test Challenges

   • Conflicts between different test categories (component vs integration)
   • Async test execution causing race conditions
   • SSE (Server-Sent Events) tests failing intermittently

4. Configuration and Infrastructure

   • Qdrant collection naming conflicts in parallel test runs
   • Docker container access issues for certain tests
   • Git repository detection problems in CI/CD environment

Expected Behavior

1. All test categories should execute successfully:

   • Component tests
   • Integration tests
   • Configuration tests
   • API endpoint tests
   • SSE endpoint tests

2. Test isolation should work properly:

   • Each test module should run in isolation when needed
   • No cross-contamination between test environments
   • Clean test cache between runs

3. Coverage reporting should:

   • Generate accurate reports
   • Include all relevant code paths
   • Produce both console and HTML output

Current Workarounds

1. Using --fully-isolated flag to run tests in separate processes
2. Manual sourcing of .env file before test execution
3. Sequential test execution with --sequential flag to avoid event loop issues
4. Custom component test runner for better isolation

Technical Details

Test Runner Configuration

# Key environment variables needed MCP_TEST_MODE=1 MCP_HOST=localhost MCP_PORT=8000 QDRANT_URL=http://localhost:6333 MCP_COLLECTION_NAME=test_collection_{unique_id}

Test Categories

1. Component Tests (tests/components/)

   • Require strict asyncio mode
   • Need isolated event loops
   • Database dependencies

2. Integration Tests (tests/integration/)

   • API endpoint tests
   • SSE functionality
   • External service interactions

3. Configuration Tests (tests/config/)

   • Environment validation
   • Configuration loading
   • Default settings

Steps to Reproduce

1. Clone the repository
2. Set up virtual environment and install dependencies
3. Copy .env.example to .env and configure
4. Run test suite: python run_tests.py --all

Error Messages

1. Module Import Errors:

ModuleNotFoundError: No module named 'mcp_codebase_insight' 

2. Event Loop Errors:

RuntimeError: Event loop is closed 

3. Coverage Reporting Issues:

Coverage.py warning: No data was collected 

Proposed Solutions

1. Environment Management

   • Implement robust virtual environment handling
   • Automate .env file loading
   • Standardize PYTHONPATH configuration

2. Test Isolation

   • Enhance the component test runner
   • Implement better cleanup between tests
   • Add transaction rollback for database tests

3. Async Testing

   • Improve event loop management
   • Add better error handling for async operations
   • Implement proper test timeouts

4. Infrastructure

   • Add Docker compose for test dependencies
   • Implement better Qdrant collection management
   • Add CI/CD specific configurations

Additional Context

• Project uses pytest for testing
• Coverage reporting through pytest-cov
• Async operations with asyncio
• Docker containers for isolation
• Qdrant for vector storage
• SSE for real-time updates

Related Files

• run_tests.py
• component_test_runner.py
• pytest.ini
• .env.example
• tests/conftest.py

Labels

• bug
• testing
• high-priority
• needs-triage
• infrastructure