A distributed system for processing and analyzing large log files in real-time, built with Go and Next.js.

This project consists of multiple microservices working together to handle log file processing, analysis, and visualization:

• Backend Service: REST API and WebSocket server for file management and real-time updates
• Frontend Service: Next.js web application for file upload and result visualization
• Log Generator Service: Utility service for generating test log files
• Log Processor Service: Service for processing and analyzing log files
• Redis: For real-time communication via Pub/Sub and caching
• PostgreSQL: For storing processing results and metadata

• Docker and Docker Compose
• Go 1.21 or later
• Node.js 18 or later
• pnpm (for frontend development)
• Redis Pub/Sub

. ├── backend-service/ # Go backend API service ├── frontend-service/ # Next.js frontend application ├── log-generator-service/ # Log file generator utility ├── log-processor-service/ # Log processing service ├── uploads/ # Shared volume for log files └── docker-compose.yml # Docker Compose configuration 

The backend service provides:

• REST API endpoints for file management
• WebSocket server for real-time updates
• Authentication using JWT Token via Supabase
• File upload handling
• Integration with Redis and PostgreSQL
• Communication between services via Redis Pub/Sub
• CORS configuration for frontend communication

The frontend service features:

• Modern UI built with Next.js
• Real-time updates via WebSocket
• File upload interface
• Results visualization
• Supabase Authentication:
  • Email/Password authentication
  • GitHub OAuth
  • Protected routes and API endpoints
  • User session management

A utility service that generates test log files with:

• Random log levels (DEBUG, INFO, WARN, ERROR, FATAL)
• Timestamped entries
• Optional JSON payloads
• Configurable file sizes

Processes log files and provides:

• Log analysis
• Pattern detection
• Real-time processing
• Result storage in PostgreSQL

1. Clone the repository:

git clone https://github.com/ijasmoopan/log-file-processor.git cd intucloud-task

2. Set up environment variables:

cp .env.example .env cp backend-service/.env.local.example backend-service/.env.local cp frontend-service/.env.local.example frontend-service/.env.local cp log-processor-service/.env.local.example log-processor-service/.env.local

3. Start the services using Docker Compose:

docker-compose up -d

4. Access the application:

• Frontend: http://localhost:3000
• Backend API: http://localhost:8080
• PostgreSQL: localhost:5432
• Redis: localhost:6379

1. Backend Service:

cd backend-service go run main.go

2. Frontend Service:

cd frontend-service pnpm install pnpm dev

3. Log Processor Service:

cd log-processor-service go run cmd/processor/main.go

4. Log Generator Service:

cd log-generator-service go run main.go

Key environment variables needed:

• DB_HOST: PostgreSQL host
• DB_PORT: PostgreSQL port
• DB_USER: Database user
• DB_PASSWORD: Database password
• DB_NAME: Database name
• REDIS_HOST: Redis host
• REDIS_PORT: Redis port
• SUPABASE_URL: Supabase URL
• SUPABASE_KEY: Supabase API key

• NEXT_PUBLIC_BACKEND_URL: Backend service URL
• NEXT_PUBLIC_SUPABASE_URL: Supabase URL
• NEXT_PUBLIC_SUPABASE_ANON_KEY: Supabase anonymous key

• REDIS_HOST: Redis host
• REDIS_PORT: Redis port

• POST /api/v1/upload: Upload log files
• GET /api/v1/files: List uploaded files
• POST /api/v1/process: Process uploaded files
• GET /api/v1/ws: WebSocket endpoint for real-time updates
• GET /api/v1/results: Get processing results
• GET /api/v1/results/:id: Get result by ID
• GET /api/v1/results/filename/:filename: Get result by filename

The project includes Docker configurations for all services:

• Each service has its own Dockerfile
• docker-compose.yml orchestrates all services
• Shared volumes for file storage and database persistence
• Network isolation using Docker networks

1. Fork the repository
2. Create your feature branch
3. Commit your changes
4. Push to the branch
5. Create a new Pull Request

This project is licensed under the MIT License - see the LICENSE file for details.