Concurrency patterns in Go; worker pools and fan-out/fan-in

Go is known for its exceptional concurrency model, but many developers focus only on goroutines and channels. However, concurrency patterns like worker pools and fan-out/fan-in provide real efficiency.

This article will get into these advanced concepts, helping you maximize throughput in your Go applications.

Why Concurrency Matters

Concurrency allows programs to perform tasks efficiently, especially when dealing with tasks like I/O operations, web requests, or background processing. In Go, goroutines provide a lightweight way to manage thousands of concurrent tasks, but without structure, you can run into bottlenecks. That’s where worker pools and fan-out/fan-in patterns come in.

Worker Pools

Worker pools allow you to limit the number of goroutines by assigning tasks to fixed "workers." This prevents oversubscription, reduces resource consumption, and makes task execution manageable.

package main import ( "fmt" "sync" "time" ) func worker(id int, jobs <-chan int, results chan<- int, wg *sync.WaitGroup) { defer wg.Done() for j := range jobs { fmt.Printf("Worker %d started job %d\n", id, j) time.Sleep(time.Second) // Simulate work fmt.Printf("Worker %d finished job %d\n", id, j) results <- j * 2 } } func main() { jobs := make(chan int, 100) results := make(chan int, 100) var wg sync.WaitGroup // Start 3 workers for w := 1; w <= 3; w++ { wg.Add(1) go worker(w, jobs, results, &wg) } // Send jobs for j := 1; j <= 5; j++ { jobs <- j } close(jobs) // Wait for workers to finish wg.Wait() close(results) for result := range results { fmt.Println("Result:", result) } } 

In this example:

• We have three workers that process jobs concurrently.
• Each job is passed to the workers via channels, and results are gathered for processing.

Fan-Out/Fan-In Pattern

The fan-out/fan-in pattern allows multiple goroutines to process the same task, while fan-in gathers the results back into a single output. This is useful for dividing tasks and then aggregating results.

The fan-in mechanism here will collect results after each task is processed by the workers.

package main import ( "fmt" "sync" "time" ) // Task struct represents the data being processed type Task struct { id int } // processTask simulates work by returning a processed result func processTask(task Task) string { time.Sleep(time.Second) // Simulate work return fmt.Sprintf("Processed task %d", task.id) } // worker processes tasks and sends results to the results channel func worker(tasks <-chan Task, results chan<- string, wg *sync.WaitGroup) { defer wg.Done() for task := range tasks { results <- processTask(task) } } func main() { tasks := make(chan Task, 5) results := make(chan string, 5) var wg sync.WaitGroup // Fan-out: Start 3 workers for i := 0; i < 3; i++ { wg.Add(1) go worker(tasks, results, &wg) } // Send tasks to the tasks channel go func() { for i := 1; i <= 10; i++ { tasks <- Task{id: i} } close(tasks) }() // Fan-in: Close results channel once all workers complete go func() { wg.Wait() close(results) }() // Collect results from the results channel for result := range results { fmt.Println(result) } } 

In this code:

• Fan-Out: Multiple goroutines (workers) handle tasks concurrently, each calling processTask.
• Fan-In: The results from each worker are collected back through the results channel and processed.

Concurrency patterns like worker pools and fan-out/fan-in are excellent for optimizing web servers, batch processing, and other I/O-bound applications, making sure resources are efficiently managed.

Next Steps to increase your knowledge:

• Experiment with applying these patterns to more complex concurrency challenges.
• Build out a web service using a worker pool to manage incoming requests.

The key to success in Go’s concurrency is structure. Mastering these concurrency patterns will level up your Go skills and help you write highly performant applications.

Stay tuned for more insights into Go in the next post!

You can support me by buying me a book :)