DEV Community

Mr Vi
Mr Vi

Posted on

The Python Developer's Guide to Background Process Management

#python #programming #tutorial #beginners

As a Python developer, you've probably faced this scenario: you've built an amazing bot, API, or data processing script, but keeping it running 24/7 is a nightmare. SSH sessions disconnect, scripts crash, servers reboot, and suddenly your carefully crafted automation is offline.

In this comprehensive guide, we'll explore different approaches to running Python applications in the background, from simple development solutions to production-ready process management.

The Problem with "Just Running Python"

When you develop locally, you run scripts like this:

python my_awesome_bot.py
Enter fullscreen mode Exit fullscreen mode

But in production, this approach fails because:

  • SSH disconnections kill your process
  • Script crashes require manual intervention
  • Server reboots lose all running processes
  • Managing multiple bots becomes chaotic
  • No visibility into what's actually running

Let's fix this with proper process management.

Method 1: Screen - The Development Solution

Screen is perfect for development and testing. It creates persistent terminal sessions that survive SSH disconnections.

Getting Started with Screen 

# Install screen sudo apt install screen # Ubuntu/Debian sudo yum install screen # CentOS/RHEL # Start a new session screen # Activate your virtual environment source ./venv/bin/activate # Run your script python my_bot.py # Detach (Ctrl+A, then D) # Your script keeps running!
Enter fullscreen mode Exit fullscreen mode

Managing Screen Sessions 

# List all sessions screen -ls # Reattach to a session screen -r session_name # Kill a session (from within screen) # Ctrl+A, then K
Enter fullscreen mode Exit fullscreen mode

When to use Screen:

  • ✅ Development and testing
  • ✅ Quick one-off scripts
  • ✅ When you need terminal access

Limitations:

  • ❌ No automatic restart on crashes
  • ❌ Manual process management
  • ❌ Sessions can be lost on server reboot
  • ❌ No built-in monitoring

Method 2: Systemd - The System Service Approach

Systemd is Linux's service manager, perfect for production environments where you need guaranteed startup and restart capabilities.

Creating a Systemd Service

Create a service file:

sudo nano /etc/systemd/system/mybot.service
Enter fullscreen mode Exit fullscreen mode

Add this configuration:

[Unit] Description=My Python Bot Service After=network.target [Service] Type=idle Restart=always RestartSec=3 User=myuser WorkingDirectory=/home/myuser/bot ExecStart=/home/myuser/bot/venv/bin/python /home/myuser/bot/my_bot.py Environment=PATH=/home/myuser/bot/venv/bin [Install] WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Managing the Service 

# Reload systemd configuration sudo systemctl daemon-reload # Enable auto-start on boot sudo systemctl enable mybot.service # Start the service sudo systemctl start mybot.service # Check status sudo systemctl status mybot.service # View logs sudo journalctl -u mybot.service -f # Stop the service sudo systemctl stop mybot.service
Enter fullscreen mode Exit fullscreen mode

When to use Systemd:

  • ✅ Production servers
  • ✅ Critical services that must survive reboots
  • ✅ System-level integration
  • ✅ When you need fine-grained control

Limitations:

  • ❌ Requires root access
  • ❌ Complex configuration
  • ❌ System-level management
  • ❌ No built-in process monitoring

Method 3: Modern Process Managers - The Python Way

For Python developers, there's a better way: Pyker - a process manager built specifically for Python applications.

Why Pyker?

Pyker combines the simplicity of screen with the power of systemd, but designed specifically for Python workflows:

  • Python-native: Built for Python developers, by Python developers
  • Virtual environment support: Native venv/conda integration
  • Zero configuration: Works out of the box
  • Beautiful monitoring: Real-time process tables
  • User-space installation: No root access required

Installation 

# One-line installation curl -fsSL https://raw.githubusercontent.com/mrvi0/pyker/main/install.sh | bash # Or with Python installer curl -fsSL https://raw.githubusercontent.com/mrvi0/pyker/main/install.py | python3 # Manual installation git clone https://github.com/mrvi0/pyker.git cd pyker python3 install.py
Enter fullscreen mode Exit fullscreen mode

Basic Usage 

# Start a simple script pyker start mybot my_bot.py # Start with virtual environment pyker start webapp app.py --venv ./venv # List all processes pyker list # View logs in real-time pyker logs mybot -f # Get detailed process information pyker info mybot # Restart a process pyker restart mybot # Stop a process pyker stop mybot
Enter fullscreen mode Exit fullscreen mode

Advanced Features

Auto-restart on failure:

pyker start critical-service app.py --auto-restart
Enter fullscreen mode Exit fullscreen mode

Virtual environment support:

# Works with any Python environment pyker start ml-worker train.py --venv /path/to/conda/envs/pytorch pyker start webapp app.py --venv ./venv pyker start data-processor process.py --venv /home/user/projects/venv
Enter fullscreen mode Exit fullscreen mode

Process monitoring:

pyker list
Enter fullscreen mode Exit fullscreen mode

Shows a beautiful, adaptive table with:

  • Process status (✓ running, ✗ stopped, ⚠ error)
  • Real-time CPU and memory usage
  • Start/stop times
  • Virtual environment information
  • Script paths

Real-time logs:

pyker logs mybot -f # Follow logs in real-time pyker logs mybot -n 100 # Show last 100 lines
Enter fullscreen mode Exit fullscreen mode

Detailed process information:

pyker info mybot
Enter fullscreen mode Exit fullscreen mode

Shows comprehensive information including:

  • Virtual environment path and Python executable
  • Resource usage (CPU, memory)
  • Runtime information
  • Log file location
  • Auto-restart status

Tab completion:

pyker <TAB> # Shows available commands pyker stop <TAB> # Completes with running process names pyker start app script.py --<TAB> # Shows --venv, --auto-restart
Enter fullscreen mode Exit fullscreen mode

Real-World Examples

Discord Bot Management 

# Traditional approach with screen screen source ./venv/bin/activate python discord_bot.py # Ctrl+A, D # With Pyker pyker start discord-bot discord_bot.py --venv ./venv --auto-restart pyker logs discord-bot -f
Enter fullscreen mode Exit fullscreen mode

Web API Deployment 

# With systemd (complex setup) sudo nano /etc/systemd/system/api.service # ... complex configuration ... sudo systemctl enable api.service # With Pyker (simple) pyker start api app.py --venv ./venv --auto-restart
Enter fullscreen mode Exit fullscreen mode

Data Processing Pipeline 

# Multiple related processes pyker start scraper scraper.py --venv ./data-env pyker start processor process_data.py --venv ./data-env pyker start uploader upload_to_s3.py --venv ./data-env # Monitor all processes pyker list # Check specific process pyker info scraper
Enter fullscreen mode Exit fullscreen mode

Machine Learning Workflows 

# Training with proper environment pyker start training train_model.py --venv /opt/conda/envs/pytorch --auto-restart # Inference service pyker start inference inference_server.py --venv ./ml-env # Data preprocessing pyker start preprocess preprocess_data.py --venv ./data-env
Enter fullscreen mode Exit fullscreen mode

Comparison: Choosing the Right Method

Method Complexity Auto-restart Virtual Env Monitoring Best For
Screen Low ❌ Manual Basic Development, testing
Systemd High ✅ Manual System logs Production servers
Pyker Low ✅ Native Built-in Python applications

Best Practices for Production

1. Always Use Virtual Environments 

# Create and activate environment python -m venv venv source venv/bin/activate pip install -r requirements.txt # Run with Pyker pyker start mybot my_bot.py --venv ./venv
Enter fullscreen mode Exit fullscreen mode

2. Enable Auto-restart for Critical Services 

pyker start critical-service app.py --auto-restart
Enter fullscreen mode Exit fullscreen mode

3. Implement Proper Logging 

import logging import sys # Configure logging logging.basicConfig( level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', handlers=[ logging.StreamHandler(sys.stdout), logging.FileHandler('app.log') ] ) logger = logging.getLogger(__name__) def main(): logger.info("Application started") try: # Your application logic  pass except Exception as e: logger.error(f"Application error: {e}") raise if __name__ == "__main__": main()
Enter fullscreen mode Exit fullscreen mode

4. Handle Graceful Shutdowns 

import signal import sys import logging logger = logging.getLogger(__name__) def signal_handler(sig, frame): logger.info("Received shutdown signal, cleaning up...") # Cleanup code here  sys.exit(0) # Register signal handlers signal.signal(signal.SIGINT, signal_handler) signal.signal(signal.SIGTERM, signal_handler)
Enter fullscreen mode Exit fullscreen mode

5. Monitor Your Processes 

# Regular status checks pyker list # Monitor logs pyker logs mybot -f # Check system resources htop
Enter fullscreen mode Exit fullscreen mode

Troubleshooting Common Issues

Process Won't Start 

# Test the script manually first python my_bot.py # Check Pyker logs pyker logs mybot # Verify process information pyker info mybot
Enter fullscreen mode Exit fullscreen mode

Virtual Environment Issues 

# Verify venv path exists ls -la ./venv/bin/python # Test with absolute path pyker start mybot my_bot.py --venv /full/path/to/venv # Check Python executable ./venv/bin/python --version
Enter fullscreen mode Exit fullscreen mode

High Resource Usage 

# Monitor resource usage pyker list # Shows CPU and memory columns # Check system resources htop top # Kill problematic processes pyker stop process_name
Enter fullscreen mode Exit fullscreen mode

Log Management 

# View log files directly ls -la ~/.pyker/logs/ # Clean old logs rm ~/.pyker/logs/*.log.* # Configure log rotation nano ~/.pyker/config.json
Enter fullscreen mode Exit fullscreen mode

Security Considerations

User Permissions 

# Run as non-root user pyker start mybot my_bot.py --venv ./venv # Check process ownership ps aux | grep mybot
Enter fullscreen mode Exit fullscreen mode

Environment Variables 

# Set environment variables export API_KEY="your-secret-key" pyker start mybot my_bot.py --venv ./venv
Enter fullscreen mode Exit fullscreen mode

Network Security 

# Bind to localhost only pyker start api app.py --venv ./venv # Configure your app to bind to 127.0.0.1:8000
Enter fullscreen mode Exit fullscreen mode

Conclusion

Running Python applications in production doesn't have to be complicated. The key is choosing the right tool for your needs:

  • Development/Testing: Use Screen for quick testing
  • System Services: Use Systemd for critical system services
  • Python Applications: Use Pyker for modern Python process management

Pyker offers the perfect balance of simplicity and power for Python developers. It's designed specifically for Python workflows, supports virtual environments natively, and provides beautiful monitoring without the complexity of systemd.

The best part? You can start simple and scale up. Begin with Screen for development, then move to Pyker for production, and only use Systemd when you need deep system integration.

Your Python applications deserve better than manual restarts and SSH sessions. Choose the right tool, and let your code run reliably 24/7.

What's your experience with running Python applications in production? Have you tried any of these approaches? Share your thoughts in the comments below!

python #devops #processmanagement #automation #production #linux #systemd #backgroundprocesses

Top comments (0)