DEV Community

Chris Moore
Chris Moore Subscriber

Posted on

Building GitNarrative: How I Parse Git History with Python to Extract Development Patterns

#git #ai #webdev #python

When I started building GitNarrative, I thought the hardest part would be the AI integration. Turns out, the real challenge was analyzing git repositories in a way that actually captures meaningful development patterns.

Here's how I built the git analysis engine that powers GitNarrative's story generation.

The Challenge: Making Sense of Messy Git History

Every git repository tells a story, but extracting that story programmatically is complex. Consider these real commit messages from a typical project:

"fix bug" "refactor" "update dependencies" "THIS FINALLY WORKS" "revert last commit" "actually fix the bug this time"
Enter fullscreen mode Exit fullscreen mode

The challenge is identifying patterns that reveal the actual development journey - the struggles, breakthroughs, and decision points that make compelling narratives.

Library Choice: pygit2 vs GitPython

I evaluated both major Python git libraries:

GitPython: More Pythonic, easier to use

import git repo = git.Repo('/path/to/repo') commits = list(repo.iter_commits())
Enter fullscreen mode Exit fullscreen mode

pygit2: Lower-level, better performance, more control

import pygit2 repo = pygit2.Repository('/path/to/repo') walker = repo.walk(repo.head.target)
Enter fullscreen mode Exit fullscreen mode

I chose pygit2 because GitNarrative needs to process repositories with thousands of commits efficiently. The performance difference is significant for large repositories.

Core Analysis Architecture

Here's the foundation of my git analysis engine:

from dataclasses import dataclass from datetime import datetime from typing import List, Dict, Set import pygit2 @dataclass class CommitAnalysis: sha: str message: str timestamp: datetime files_changed: List[str] additions: int deletions: int author: str is_merge: bool complexity_score: float commit_type: str # 'feature', 'bugfix', 'refactor', 'docs', etc.  class GitAnalyzer: def __init__(self, repo_path: str): self.repo = pygit2.Repository(repo_path) def analyze_repository(self) -> Dict: commits = self._extract_commits() patterns = self._identify_patterns(commits) timeline = self._build_timeline(commits) milestones = self._detect_milestones(commits) return { "commits": commits, "patterns": patterns, "timeline": timeline, "milestones": milestones, "summary": self._generate_summary(commits, patterns) }
Enter fullscreen mode Exit fullscreen mode

Pattern Recognition: The Heart of Story Extraction

The key insight is that commit patterns reveal development phases. Here's how I identify them:

1. Commit Type Classification 

def _classify_commit(self, commit_message: str, files_changed: List[str]) -> str: message_lower = commit_message.lower() # Bug fix patterns  if any(keyword in message_lower for keyword in ['fix', 'bug', 'issue', 'error']): return 'bugfix' # Feature patterns  if any(keyword in message_lower for keyword in ['add', 'implement', 'create', 'feature']): return 'feature' # Refactor patterns  if any(keyword in message_lower for keyword in ['refactor', 'restructure', 'reorganize']): return 'refactor' # Documentation  if any(keyword in message_lower for keyword in ['doc', 'readme', 'comment']): return 'docs' # Dependency/config changes  if any(file.endswith(('.json', '.yml', '.yaml', '.toml')) for file in files_changed): return 'config' return 'other'
Enter fullscreen mode Exit fullscreen mode

2. Development Phase Detection 

def _identify_development_phases(self, commits: List[CommitAnalysis]) -> List[Dict]: phases = [] current_phase = None for i, commit in enumerate(commits): # Look for phase transition indicators  if self._is_architecture_change(commit): if current_phase: phases.append(current_phase) current_phase = { 'type': 'architecture_change', 'start_commit': commit.sha, 'description': 'Major architectural refactoring', 'commits': [commit] } elif self._is_feature_burst(commits[max(0, i-5):i+1]): # Multiple feature commits in short timeframe  if not current_phase or current_phase['type'] != 'feature_development': if current_phase: phases.append(current_phase) current_phase = { 'type': 'feature_development', 'start_commit': commit.sha, 'description': 'Rapid feature development phase', 'commits': [commit] } if current_phase: current_phase['commits'].append(commit) return phases def _is_architecture_change(self, commit: CommitAnalysis) -> bool: # High file change count + specific patterns  return (len(commit.files_changed) > 10 and commit.complexity_score > 0.8 and any(keyword in commit.message.lower() for keyword in ['refactor', 'restructure', 'migrate']))
Enter fullscreen mode Exit fullscreen mode

3. Struggle and Breakthrough Detection

This is where the storytelling magic happens:

def _detect_struggle_patterns(self, commits: List[CommitAnalysis]) -> List[Dict]: struggles = [] for i in range(len(commits) - 3): window = commits[i:i+4] # Look for multiple attempts at same issue  if self._is_struggle_sequence(window): struggles.append({ 'type': 'debugging_struggle', 'commits': window, 'description': self._describe_struggle(window), 'resolution_commit': self._find_resolution(commits[i+4:i+10]) }) return struggles def _is_struggle_sequence(self, commits: List[CommitAnalysis]) -> bool: # Multiple bug fix attempts in short timeframe  bugfix_count = sum(1 for c in commits if c.commit_type == 'bugfix') # Time clustering (all within days of each other)  time_span = (commits[-1].timestamp - commits[0].timestamp).days return bugfix_count >= 2 and time_span <= 3 def _find_resolution(self, following_commits: List[CommitAnalysis]) -> CommitAnalysis: # Look for commit that likely resolved the issue  for commit in following_commits: if ('work' in commit.message.lower() or 'fix' in commit.message.lower() or commit.complexity_score > 0.6): return commit return None
Enter fullscreen mode Exit fullscreen mode

Timeline Correlation: When Things Happened

Understanding timing is crucial for narrative flow:

def _build_timeline(self, commits: List[CommitAnalysis]) -> Dict: # Group commits by time periods  monthly_activity = defaultdict(list) for commit in commits: month_key = commit.timestamp.strftime('%Y-%m') monthly_activity[month_key].append(commit) timeline = {} for month, month_commits in monthly_activity.items(): timeline[month] = { 'total_commits': len(month_commits), 'commit_types': self._analyze_commit_distribution(month_commits), 'major_changes': self._identify_major_changes(month_commits), 'development_velocity': self._calculate_velocity(month_commits) } return timeline def _calculate_velocity(self, commits: List[CommitAnalysis]) -> float: if not commits: return 0.0 # Factor in commit frequency, complexity, and file changes  total_complexity = sum(c.complexity_score for c in commits) total_files = sum(len(c.files_changed) for c in commits) return (total_complexity * total_files) / len(commits)
Enter fullscreen mode Exit fullscreen mode

Performance Optimizations

Processing large repositories efficiently required several optimizations:

1. Lazy Loading 

def _extract_commits(self, max_commits: int = 1000) -> List[CommitAnalysis]: # Process commits in batches to avoid memory issues  walker = self.repo.walk(self.repo.head.target) commits = [] for i, commit in enumerate(walker): if i >= max_commits: break commits.append(self._analyze_single_commit(commit)) return commits
Enter fullscreen mode Exit fullscreen mode

2. Caching Results 

from functools import lru_cache @lru_cache(maxsize=128) def _calculate_complexity_score(self, sha: str) -> float: # Expensive calculation cached per commit  commit = self.repo[sha] # ... complexity calculation  return score
Enter fullscreen mode Exit fullscreen mode

3. Parallel Processing for Multiple Repositories 

import asyncio from concurrent.futures import ProcessPoolExecutor async def analyze_multiple_repos(repo_paths: List[str]) -> List[Dict]: with ProcessPoolExecutor() as executor: loop = asyncio.get_event_loop() tasks = [ loop.run_in_executor(executor, analyze_single_repo, path) for path in repo_paths ] return await asyncio.gather(*tasks)
Enter fullscreen mode Exit fullscreen mode

Integration with AI Story Generation

The analysis output feeds directly into AI prompts:

def format_for_ai_prompt(self, analysis: Dict) -> str: prompt_data = { 'repository_summary': analysis['summary'], 'development_phases': analysis['patterns']['phases'], 'key_struggles': analysis['patterns']['struggles'], 'breakthrough_moments': analysis['milestones'], 'timeline': analysis['timeline'] } return self._build_narrative_prompt(prompt_data)
Enter fullscreen mode Exit fullscreen mode

Challenges and Solutions

Challenge 1: Repositories with inconsistent commit message styles
Solution: Pattern matching with multiple fallback strategies and file-based analysis

Challenge 2: Merge commits creating noise in analysis
Solution: Filtering strategy that focuses on meaningful commits while preserving merge context

Challenge 3: Very large repositories (10k+ commits)
Solution: Sampling strategy that captures representative commits from different time periods

Results and Validation

The analysis engine successfully processes repositories ranging from small personal projects to large open source codebases. When tested on React's repository, it correctly identified:

  • The initial experimental phase (2013)
  • Major architecture rewrites (Fiber, Hooks)
  • Performance optimization periods
  • API stabilization phases

What's Next

Current improvements in development:

  • Better natural language processing of commit messages
  • Machine learning models for commit classification
  • Integration with issue tracker data for richer context
  • Support for monorepo analysis

The git analysis engine is the foundation that makes GitNarrative's storytelling possible. By extracting meaningful patterns from commit history, we can transform boring git logs into compelling narratives about software development.

GitNarrative is available at https://gitnarrative.io - try it with your own repositories to see these patterns in action.

What patterns have you noticed in your own git history? I'd love to hear about interesting commit patterns you've discovered in your projects.

Top comments (0)