Handling Parallel Image Generation in Go

@ Go

Introduction

While learning Go, I had been looking for an opportunity to try a parallel scenario, and one of the most fundamental Worker Pool (Thread Pool) patterns is a great entry point: “keep a fixed number of workers running in the background, waiting for work to be assigned to them.”

As it happens, I recently built a small tool called goog🔗, with practical examples available in goog-demo🔗.

This scenario is well suited for practicing parallel programming because each image is independent of the others. You do not need to wait for the previous image to finish before generating the next one, but you also cannot run everything at the same time without limits, because the implementation starts headless Chrome tabs to take screenshots behind the scenes. Running everything at once is not practical.

So the approach I ultimately adopted was: “process tasks in parallel with goroutines, and use the properties of buffered channels to achieve the effect of a semaphore, limiting the number of parallel operations.”

Real-World Case Study

For example, my blog has Open Graph preview images to generate for thousands of articles. Usually, generating each image with satori🔗 (a JavaScript-based image generation library) takes around 800ms, and generating all images takes tens of minutes overall. If I could take advantage of Go’s strengths—a compiled language, native concurrency support, and a lightweight runtime—I was curious how much more hardware potential could be squeezed out. So I tried building a CLI tool to generate images:

  1. Read an HTML template
  2. Use Go template🔗 to fill in variables such as the article title, description, and category
  3. Open headless Chrome through chromedp🔗
  4. Inject the HTML into the page
  5. Capture a 1200x630 image
  6. Write the image file

A single image can be generated like this:

Terminal window
goog \
  --var "tag=Tutorial" \
  --var "title=How to Generate OG Images in Go" \
  --var "description=Learn how to automate social card generation with chromedp" \
  --var "site=example.com" \
  --out tutorial-og.png

The template is just a regular HTML file, except that Go Template variables can be injected into it:

<body>
    <div class="card">
        <div class="tag">{{.tag}}</div>
        <div class="title">{{.title}}</div>
        <div class="description">{{.description}}</div>
        <div class="footer">
            <div class="dot"></div>
            <span>{{.site}}</span>
        </div>
    </div>
</body>

But what is truly useful is batch mode, for example reading multiple tasks at once from images.json:

[
  {
    "vars": {
      "tag": "Tutorial",
      "title": "Getting Started with Go",
      "description": "Learn the basics of Go programming language.",
      "site": "example.com"
    },
    "out": "out/getting-started.png"
  },
  {
    "vars": {
      "tag": "Deep Dive",
      "title": "Concurrency in Go",
      "description": "Goroutines, channels, and patterns for concurrent programming.",
      "site": "example.com"
    },
    "out": "out/concurrency.png"
  }
]

Then specify the number of workers when running it:

Terminal window
goog --config images.json --workers 4

--workers 4 does not mean that only four images in total will be processed. It means that at most four image generation tasks are allowed to run at the same time.

Core Implementation in goog

Process tasks in parallel with goroutines, and use the properties of buffered channels to achieve the effect of a semaphore, limiting the number of parallel operations
generator.go
func (g *Generator) Generate(ctx context.Context, jobs []ImageJob) error {
  if len(jobs) == 0 {
    return fmt.Errorf("no image jobs to process")
  }


  sem := make(chan struct{}, g.workers)
  var wg sync.WaitGroup
  var mu sync.Mutex
  var errs []error
  start := time.Now()
  for i, job := range jobs {


    wg.Add(1)


    sem <- struct{}{} // acquire slot
    go func(idx int, j ImageJob) {


      defer wg.Done()


      defer func() { <-sem }() // release slot
      if err := g.processJob(ctx, j); err != nil {


        mu.Lock()
        errs = append(errs, fmt.Errorf("job %d (%s): %w", idx, j.Out, err))
        mu.Unlock()
        log.Printf("Job %d failed (%s): %v", idx, j.Out, err)
      } else {
        log.Printf("[%d/%d] saved %s", idx+1, len(jobs), j.Out)
      }
    }(i, job)
  }


  wg.Wait()
  elapsed := time.Since(start)
  fmt.Printf("\nGenerated %d/%d images in %s\n", len(jobs)-len(errs), len(jobs), elapsed.Round(time.Millisecond))
  if len(errs) > 0 {
    return fmt.Errorf("%d job(s) failed", len(errs))
  }
  return nil
}

What Problem Does a Worker Pool Solve?

Parallel execution in Go can easily be achieved by starting a goroutine:

for _, job := range jobs {
  go processJob(job)
}

If you throw thousands of image generation jobs into goroutines all at once, it will be a disaster. The operation mixes file I/O, browser resources, memory, and coordination with external programs, causing resources to be exhausted. The idea behind a worker pool is therefore to “put an upper bound on parallel execution.”

What Is a Semaphore?

A semaphore is one way to implement a worker pool. You can think of it as a counter with a fixed number of passes. If workers = 4, that means there are only four passes available at the same time. Each task must take one before it starts and return it after it finishes. When all four passes have been taken, the fifth task waits in place until someone completes and releases a pass. In Go, this can be achieved naturally with a buffered channel:

sem := make(chan struct{}, workers)


sem <- struct{}{} // acquire: take a pass
<-sem             // release: return the pass

Here we use struct{}{} because we do not care about the values inside the channel; we only care how many elements have already been placed in the buffer. An empty struct carries no additional data meaning, making it well suited to represent “one slot.” When the channel buffer is full, sem <- struct{}{} blocks until another goroutine reads a value from sem.

WaitGroup: Waiting for All Work to Complete

sync.WaitGroup is responsible for letting the main flow know that all goroutines have completed.

wg.Add(1)


go func() {
  defer wg.Done()
  // do work
}()


wg.Wait()

Without wg.Wait(), the main program might continue executing before the goroutines have finished, or even exit directly. I put wg.Done() inside defer to ensure that regardless of whether a task succeeds, fails, or returns midway, the WaitGroup is always notified: this piece of work has ended.

Buffered Channels: Limiting the Number of Simultaneous Executions

The line that actually controls concurrency is this:

sem := make(chan struct{}, g.workers)

Together with the acquire operation before each goroutine starts:

sem <- struct{}{}

And the release operation when the goroutine finishes:

defer func() { <-sem }()

There is one detail in this approach: sem <- struct{}{} is placed before go func(...), so the main goroutine attempts to acquire a slot before creating a new goroutine.

In other words, if workers = 4, the first four jobs will start successfully. When it reaches the fifth job, the main goroutine will block at sem <- struct{}{} until one of the previous jobs finishes and releases a slot.

This way does not require actually pre-creating four long-lived worker goroutines. Instead, it uses a semaphore so that at most the specified number of “short-lived goroutines” exist at the same time.

Mutex: Protecting the Error List

Multiple goroutines may fail at the same time and append errors to errs simultaneously:

errs = append(errs, err)

append modifies the slice header and the underlying array, and is not a thread-safe operation. Therefore, sync.Mutex is needed here:

mu.Lock()
errs = append(errs, fmt.Errorf("job %d (%s): %w", idx, j.Out, err))
mu.Unlock()

The purpose of this code is not to slow down image generation, but to protect shared data. Image generation itself still runs concurrently; only writing to the error list briefly queues up.

Sharing a Headless Browser

The easiest version to write at first is to start a new Chrome instance for every image. This provides the best isolation, but the cost is very high. In the end, goog uses a shared browser context:

allocCtx, allocCancel := chromedp.NewExecAllocator(
  context.Background(),
  append(chromedp.DefaultExecAllocatorOptions[:],
    chromedp.Flag("disable-gpu", true),
    chromedp.Flag("no-sandbox", true),
  )...,
)


browserCtx, browserCancel := chromedp.NewContext(allocCtx)


if err := chromedp.Run(browserCtx); err != nil {
  allocCancel()
  return nil, fmt.Errorf("failed to start browser: %w", err)
}

Then each job opens its own tab context:

tabCtx, tabCancel := chromedp.NewContext(g.browserCtx)
defer tabCancel()


tabCtx, timeoutCancel := context.WithTimeout(tabCtx, 30*time.Second)
defer timeoutCancel()

The benefit is that Chrome only needs to be started once, while each screenshot task still has its own page context. Combined with a worker pool, this avoids opening too many tabs at the same time.

Conclusion

The biggest takeaway from writing goog was this: concurrency is not about throwing everything out at the same time, but about knowing where things should run in parallel and where they should be rate-limited. A semaphore-based worker pool is not necessarily the answer to every worker pool problem, but it is a good fit for this kind of CLI batch-processing scenario where “the task list is known, each task is independent, external resources are expensive, and the number of simultaneous executions must be limited.”

Headless Browser Screenshots Are Actually Slower for Rendering a Single Image

goog takes around 1 second to generate a single image, which is actually slower than the roughly 800ms mentioned earlier for satori. The reason is that compared with an approach that renders SVG with JSX, this one drives an entire browser behind the scenes to render and capture a screenshot, but it provides much greater rendering flexibility.

Although the two approaches differ in technology choice and direction, I still made a rough replacement for my blog from Satori generation🔗 to goog generation🔗, and saw at least a 3x speed difference in overall image rendering (18 minutes > 6 minutes). So is goog better than satori? Not necessarily, but it does have some interesting characteristics:

  1. It can render with any web page template instead of rendering SVG through JSX
  2. It supports parallel execution
  3. It provides ready-made GitHub Action integration and Markdown Frontmatter parsing

Further Reading