DEV Community

Akhilesh Mishra
Akhilesh Mishra

Posted on • Originally published at livingdevops.com

How to Auto-Generate requirements.txt for Python Projects: Complete Guide to pipreqs and pigar

#programming #python

Stop manually hunting for Python dependencies - automate requirements.txt generation with these powerful tools

Python dependency management tools

As a DevOps engineer, I frequently collaborate with multiple teams, diving into various projects and repositories. One common challenge I face is encountering Python code without a requirements.txt file.

Every time I run such code, I need to figure out and manually install the necessary dependencies. I found it a waste of time and effort. I am sure you guys also feel the same way.

So I dug around, did some Google searches, experimented with a few tools, and found many good ones. My favorites are pipreqs and pigar.

In this comprehensive guide, I'll show you how to use these Python dependency management tools to generate requirements.txt files automatically for any Python project.

Table of Contents

Why Not Use pip freeze?

Before diving into the solutions, let's address why pip freeze isn't always the best choice for generating requirements.txt:

❌ pip freeze problems:

  • Only saves packages installed with pip install in your current environment
  • Includes ALL packages in the environment, even those not used in your project (if you don't use virtualenv)
  • Sometimes you need to create requirements.txt for a new project without installing modules first
  • Can include system-wide packages that aren't relevant to your project

✅ What we need instead:

  • Tools that analyze your actual Python code
  • Generate requirements based on import statements
  • Create clean, project-specific dependency lists

Method 1: Using pipreqs - The Import-Based Analyzer

What is pipreqs?

pipreqs is a Python tool that analyzes your Python files and generates requirements.txt based on the actual imports in your code. It's smart, fast, and doesn't require installing packages beforehand.

Setting Up Your Environment

I recommend working with isolated Python virtual environments for better dependency management:

# Create a project directory mkdir python-requirements-demo cd python-requirements-demo # Create and activate virtual environment python3 -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate
Enter fullscreen mode Exit fullscreen mode

Installing pipreqs 

pip install pipreqs
Enter fullscreen mode Exit fullscreen mode

Practical Example: Word Cloud Generator

Let's create a real Python project to demonstrate pipreqs. Create a directory and file:

mkdir project cd project
Enter fullscreen mode Exit fullscreen mode

Create word_cloud_generator.py with the following code:

import matplotlib.pyplot as plt from wordcloud import WordCloud from collections import Counter import requests from PIL import Image import numpy as np import argparse from io import BytesIO def get_mask_image(url): """Download and process mask image from URL""" response = requests.get(url) img = Image.open(BytesIO(response.content)) return np.array(img) def create_word_cloud(text, mask_url=None): """Generate word cloud from text with optional mask""" if mask_url: mask = get_mask_image(mask_url) else: mask = None wordcloud = WordCloud( width=800, height=400, background_color='white', mask=mask, contour_width=1, contour_color='black' ).generate(text) plt.figure(figsize=(10, 5)) plt.imshow(wordcloud, interpolation='bilinear') plt.axis('off') plt.show() def parse_arguments(): """Parse command line arguments""" parser = argparse.ArgumentParser(description="Generate a word cloud from input text") parser.add_argument('--text', '-t', required=True, help="Input text for word cloud") parser.add_argument('--mask_url', '-m', help="URL of mask image for word cloud shape") return parser.parse_args() if __name__ == "__main__": args = parse_arguments() create_word_cloud(args.text, args.mask_url)
Enter fullscreen mode Exit fullscreen mode

Generating requirements.txt with pipreqs

Now comes the magic! Instead of manually figuring out dependencies, run:

# Generate requirements.txt for the project directory pipreqs project
Enter fullscreen mode Exit fullscreen mode

Output in requirements.txt:

matplotlib==3.9.1 numpy==2.0.0 Pillow==10.4.0 Requests==2.32.3 wordcloud==1.9.3
Enter fullscreen mode Exit fullscreen mode

Installing and Testing Dependencies 

# Install all dependencies at once pip install -r project/requirements.txt # Test the script python project/word_cloud_generator.py --text "Python dependency management made easy" # With mask image python project/word_cloud_generator.py --text "pipreqs is awesome" --mask_url "https://example.com/mask.jpg"
Enter fullscreen mode Exit fullscreen mode

Advanced pipreqs Options 

# Generate requirements for specific directory pipreqs /path/to/your/project # Specify output file name pipreqs . --savepath requirements-dev.txt # Force overwrite existing requirements.txt pipreqs . --force # Use local database for package names pipreqs . --use-local # Debug mode for troubleshooting pipreqs . --debug
Enter fullscreen mode Exit fullscreen mode

Method 2: Using pigar - The Advanced Dependency Scanner

What is pigar?

pigar is another excellent tool for generating Python requirements. It offers more advanced features and better handling of complex projects.

Installing pigar 

# Using pip pip install pigar # From source (latest features) pip install git+https://github.com/damnever/pigar.git@main --upgrade # Using conda conda install -c conda-forge pigar
Enter fullscreen mode Exit fullscreen mode

Real-World Example: Flask Application

Let's test pigar with a Flask application:

# Clone a sample Flask project git clone https://github.com/akhileshmishrabiz/Devops-zero-to-hero cd Devops-zero-to-hero/project5 # Remove existing requirements.txt for demo rm requirements.txt
Enter fullscreen mode Exit fullscreen mode

Generating requirements with pigar 

# Generate requirements.txt for current directory pigar generate # Check the generated file cat requirements.txt
Enter fullscreen mode Exit fullscreen mode

Output:

# Automatically generated by https://github.com/damnever/pigar. Flask==3.0.3 Flask-SQLAlchemy==3.1.1
Enter fullscreen mode Exit fullscreen mode

Advanced pigar Usage 

# Generate requirements for specific directory with custom filename pigar gen -f custom-requirements.txt /path/to/project # Example with Python Lambda project pigar gen -f python-lambda-requirements.txt python-for-devops/python-lambda-runtime/
Enter fullscreen mode Exit fullscreen mode

Output for Lambda project:

# Automatically generated by https://github.com/damnever/pigar. boto3==1.34.142 colorlog==6.8.2 packaging==24.1
Enter fullscreen mode Exit fullscreen mode

pigar Advanced Features 

# Check for outdated packages pigar check # Search for packages pigar search numpy # Show package information pigar show requests
Enter fullscreen mode Exit fullscreen mode

Comparison: pipreqs vs pigar

Feature pipreqs pigar
Speed ⚡ Very Fast 🔄 Moderate
Accuracy ✅ High ✅ Very High
Local imports ❌ Limited ✅ Excellent
Complex projects ✅ Good ✅ Excellent
Additional features ❌ Basic ✅ Package search, outdated checks
File size 📦 Lightweight 📦 Slightly larger

Best Practices for Python Dependency Management

1. Use Virtual Environments 

# Always create isolated environments python -m venv project-env source project-env/bin/activate
Enter fullscreen mode Exit fullscreen mode

2. Pin Exact Versions 

# Good - specific versions requests==2.32.3 numpy==2.0.0 # Avoid - unpinned versions requests numpy
Enter fullscreen mode Exit fullscreen mode

3. Separate Development Dependencies 

# Create different requirement files pigar gen -f requirements.txt . # Production pigar gen -f requirements-dev.txt . # Development
Enter fullscreen mode Exit fullscreen mode

4. Regular Dependency Audits 

# Check for security vulnerabilities pip-audit -r requirements.txt # Update outdated packages pip list --outdated
Enter fullscreen mode Exit fullscreen mode

Troubleshooting Common Issues

pipreqs Not Finding All Imports 

# Use debug mode pipreqs . --debug # Force scan all files pipreqs . --scan-notebooks
Enter fullscreen mode Exit fullscreen mode

pigar Missing Local Modules 

# Include local packages pigar gen --with-referenced-comments
Enter fullscreen mode Exit fullscreen mode

Version Conflicts 

# Generate without version pins pipreqs . --no-version # Then manually specify versions
Enter fullscreen mode Exit fullscreen mode

Automation with CI/CD

GitHub Actions Example 

name: Check Requirements on: [push, pull_request] jobs: check-deps: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Generate requirements run: | pip install pipreqs pipreqs . --force git diff --exit-code requirements.txt
Enter fullscreen mode Exit fullscreen mode

Conclusion

Both pipreqs and pigar are excellent tools for automating Python dependency management:

  • Choose pipreqs for: Simple projects, fast generation, lightweight solution
  • Choose pigar for: Complex projects, advanced features, comprehensive analysis

Stop wasting time manually managing Python dependencies. These tools will save you hours and reduce deployment errors significantly.

Next Steps

  1. Try both tools on your existing Python projects
  2. Integrate dependency generation into your development workflow
  3. Set up automated checks in your CI/CD pipeline
  4. Share this guide with your team to standardize dependency management

Related Python Articles

Tags

#python #devops #dependencies #automation #pip #requirements #development #productivity #tools #tutorial

Found this helpful? Follow for more Python and DevOps tutorials!

Connect with the author:

Top comments (0)