A Python-based web scraper that collects GitHub developer information, their followers, and repository details using Selenium and stores the data in a MySQL database.

• Scrapes trending developers across multiple programming languages
• Collects follower information (up to 1000 per developer)
• Gathers repository details including name, URL, description, language, stars, and forks
• Supports authentication via cookies or username/password
• Stores data in a MySQL database with automatic schema creation
• Includes error handling and logging
• Follows clean architecture principles

github-toolkit/ ├── config/ │ └── settings.py # Configuration and environment variables ├── core/ │ ├── entities.py # Domain entities │ └── exceptions.py # Custom exceptions ├── infrastructure/ │ ├── database/ # Database-related code │ │ ├── connection.py │ │ └── models.py │ └── auth/ # Authentication service │ └── auth_service.py ├── services/ │ └── scraping/ # Scraping services │ ├── github_developer_scraper.py │ └── github_repo_scraper.py ├── utils/ │ └── helpers.py # Utility functions ├── controllers/ │ └── github_scraper_controller.py # Main controller ├── main.py # Entry point └── README.md 

• Python 3.8+
• MySQL database
• Chrome browser
• Chrome WebDriver

1. Clone the repository:

git clone https://github.com/yourusername/github-scraper.git cd github-scraper

2. Create a virtual environment and activate it:

python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate

3. Install dependencies:

pip install -r requirements.txt

4. Create a .env file in the root directory with the following variables:

GITHUB_USERNAME=your_username GITHUB_PASSWORD=your_password DB_USERNAME=your_db_username DB_PASSWORD=your_db_password DB_HOST=your_db_host DB_NAME=your_db_name 

5. Create a config directory:

mkdir config

Create a requirements.txt file with:

selenium sqlalchemy python-dotenv 

Run the scraper:

python main.py

The scraper will:

1. Authenticate with GitHub
2. Scrape trending developers for specified languages
3. Collect their followers (up to 1000 per developer)
4. Scrape their repositories
5. Store all data in the MySQL database

• Modify config/settings.py to change:
  • LANGUAGES: List of programming languages to scrape
  • USE_COOKIE: Toggle between cookie-based and credential-based authentication
• Adjust sleep times in services if needed for rate limiting

• id (PK)
• username (unique)
• profile_url
• created_at
• updated_at
• published_at

• id (PK)
• username
• repo_name
• repo_intro
• repo_url (unique)
• repo_lang
• repo_stars
• repo_forks
• created_at
• updated_at
• published_at

• Custom exceptions for authentication, scraping, and database operations
• Logging configured at INFO level
• Graceful shutdown of browser instance

1. Fork the repository.
2. Create a feature branch (git checkout -b feature/your-feature).
3. Commit changes (git commit -m "Add your feature").
4. Push to the branch (git push origin feature/your-feature).
5. Open a pull request.

This project is licensed under the MIT License - see the LICENSE file for details (create one if needed).

• Built with Selenium, SQLAlchemy, and Python.
• Inspired by the need to automate GitHub data collection.