Concurrency Patterns in Go: Fan-Out, Fan-In Explained

Concurrency is one of Go’s standout features, enabling developers to write high-performing applications with ease. By using goroutines and channels, Go provides the tools to build efficient, concurrent systems. Among the concurrency design patterns commonly used in Go are Fan-Out and Fan-In, which facilitate parallelism and data aggregation respectively.

In this blog, we’ll explore these patterns in depth, understand their use cases, and see how to implement them in Go.

---

What is Fan-Out?

Fan-Out refers to the process of distributing work across multiple goroutines. This pattern is used when a single task can be divided into smaller, independent tasks that can execute concurrently. By “fanning out” work to multiple workers, the system can perform tasks more quickly by utilizing all available CPU cores.

Use Case for Fan-Out

Imagine a scenario where you need to fetch data from multiple APIs or process a large batch of data concurrently. Instead of processing them sequentially, you can spawn multiple workers to perform these tasks in parallel.

Implementing Fan-Out in Go

Here’s an example of how to implement the Fan-Out pattern in Go:

go
package main

import (
	"fmt"
	"math/rand"
	"sync"
	"time"
)

func worker(id int, jobs <-chan int, results chan<- int) {
	for job := range jobs {
		fmt.Printf("Worker %d processing job %d\n", id, job)
		// Simulate work
		time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
		results <- job * 2 // Example processing: double the number
	}
}

func main() {
	const numJobs = 10
	const numWorkers = 3

	jobs := make(chan int, numJobs)
	results := make(chan int, numJobs)

	var wg sync.WaitGroup

	// Start worker goroutines
	for w := 1; w <= numWorkers; w++ {
		wg.Add(1)
		go func(id int) {
			defer wg.Done()
			worker(id, jobs, results)
		}(w)
	}

	// Send jobs
	for j := 1; j <= numJobs; j++ {
		jobs <- j
	}
	close(jobs)

	// Wait for all workers to finish
	go func() {
		wg.Wait()
		close(results)
	}()

	// Collect results
	for result := range results {
		fmt.Printf("Result: %d\n", result)
	}
}

Output Example

bash
Worker 1 processing job 1
Worker 2 processing job 2
Worker 3 processing job 3
Result: 2
Result: 4
Result: 6
...

---

What is Fan-In?

Fan-In is the process of aggregating results from multiple sources (or goroutines) into a single channel. This pattern simplifies the collection of outputs, especially when you have multiple producers.

Use Case for Fan-In

Consider a scenario where multiple APIs are queried in parallel, and the results need to be combined and processed sequentially.

Implementing Fan-In in Go

Here’s how you can implement the Fan-In pattern in Go:

go
package main

import (
	"fmt"
	"time"
)

func producer(id int, out chan<- string) {
	for i := 1; i <= 5; i++ {
		out <- fmt.Sprintf("Producer %d: item %d", id, i)
		time.Sleep(time.Millisecond * 100)
	}
}

func fanIn(inputs ...<-chan string) <-chan string {
	out := make(chan string)
	go func() {
		for _, input := range inputs {
			go func(ch <-chan string) {
				for v := range ch {
					out <- v
				}
			}(input)
		}
	}()
	return out
}

func main() {
	// Create multiple input channels
	producer1 := make(chan string)
	producer2 := make(chan string)

	go producer(1, producer1)
	go producer(2, producer2)

	// Aggregate results
	result := fanIn(producer1, producer2)

	// Read and print aggregated results
	for i := 0; i < 10; i++ {
		fmt.Println(<-result)
	}

	close(producer1)
	close(producer2)
}

Output Example

bash
Producer 1: item 1
Producer 2: item 1
Producer 1: item 2
Producer 2: item 2
...

---

Combining Fan-Out and Fan-In

Often, real-world scenarios involve a combination of these two patterns. For example, a task can be divided into subtasks (Fan-Out), and their results need to be aggregated back (Fan-In).

Here’s a combined example:

go
package main

import (
	"fmt"
	"sync"
)

func worker(id int, jobs <-chan int, results chan<- int) {
	for job := range jobs {
		results <- job * job // Example: square the number
	}
}

func main() {
	jobs := make(chan int, 10)
	results := make(chan int, 10)

	// Start workers
	var wg sync.WaitGroup
	for i := 1; i <= 3; i++ {
		wg.Add(1)
		go func(id int) {
			defer wg.Done()
			worker(id, jobs, results)
		}(i)
	}

	// Send jobs
	go func() {
		for i := 1; i <= 10; i++ {
			jobs <- i
		}
		close(jobs)
	}()

	// Wait for workers to finish
	go func() {
		wg.Wait()
		close(results)
	}()

	// Collect results
	for result := range results {
		fmt.Println("Result:", result)
	}
}

---

Conclusion

Concurrency patterns like Fan-Out and Fan-In are powerful tools for building efficient and parallel systems in Go. The Fan-Out pattern distributes workload to maximize CPU utilization, while the Fan-In pattern aggregates results to simplify data collection. Together, they form the foundation of many real-world Go applications.

Understanding and applying these patterns effectively can significantly enhance the performance and scalability of your programs.