Worker pools in Go

Go: Worker Pools

What is Worker Pools in Go?

In Go, a worker pool is a design pattern that involves a group of workers that perform a fixed set of tasks. A worker pool is useful when you have a large number of tasks that need to be completed, and you want to distribute these tasks among a group of workers to increase efficiency.

How to create a worker pool in Go?

To create a worker pool in Go, you typically use a combination of goroutines, channels, and sync.WaitGroup.

Here's an example of how to create a worker pool with a fixed number of workers:

    package main

    import (
        "fmt"
        "sync"
    )

    func worker(id int, jobs <-chan int, results chan<- int) {
        for j := range jobs {
            fmt.Printf("Worker %d started job %d\n", id, j)
            results <- j * 2
            fmt.Printf("Worker %d finished job %d\n", id, j)
        }
    }

    func main() {
        const numJobs = 5
        jobs := make(chan int, numJobs)
        results := make(chan int, numJobs)

        // Create a group to wait for all the workers to finish.
        var wg sync.WaitGroup

        // Start the workers.
        const numWorkers = 3
        for i := 1; i <= numWorkers; i++ {
            wg.Add(1)
            go func(id int) {
                defer wg.Done()
                worker(id, jobs, results)
            }(i)
        }

        // Send the jobs to the workers.
        for j := 1; j <= numJobs; j++ {
            jobs <- j
        }
        close(jobs)

        // Wait for all the workers to finish.
        wg.Wait()

        // Collect the results.
        for a := 1; a <= numJobs; a++ {
            fmt.Println(<-results)
        }
    }

Explainations

• We create a job queue and a results channel with a buffer size of 5. We also create 3 worker goroutines, which will be waiting for jobs to be added to the job queue.

• We add 5 jobs to the job queue, and then close it to signal that there are no more jobs. The worker goroutines will keep processing jobs until the channel is closed.

• The sync.WaitGroup is used to wait for all the worker goroutines to finish. Each worker goroutine is created with a defer wg.Done() statement, which will signal that the goroutine has finished processing a job. Finally, we print the results of the processing.