Designing and Running a Stress Test for Container Build Services at Scale

Designing and Running a Stress Test for Container Build Services at Scale

In modern cloud-native architectures, dynamically building and caching container images is a core part of operating scalable compute jobs. When hundreds or thousands of background jobs trigger simultaneously, the underlying container build service is hit with massive request spikes. This surge can expose architectural bottlenecks—from image layer caching inefficiencies to API latency, and aggressive rate limiting thresholds.

To ensure stability, running structured stress tests against your build engine is critical. In this post, we’ll explore how to design and run a highly concurrent stress test for a container build service using Go.

The Challenge of High Concurrency

When a large batch of cloud computing jobs starts, each job often requests a container image to execute its workload. If the images are slightly modified or need to be dynamically built, the container build engine receives a flood of requests.

The primary challenges under this high-load scenario are:

1. API Latency: The control plane parsing requests and scheduling builds might slow down.
2. Caching Overhead: Checking whether a container layer already exists in the cache across multiple concurrent requests introduces heavy I/O operations and database lookups.
3. Rate Limits: Upstream container registries (like Docker Hub, AWS ECR, or Google Artifact Registry) may aggressively rate-limit pulling base images.
4. Connection Timeouts: HTTP connection pools can become exhausted, resulting in dropped connections and failed builds.

To prepare for these challenges, we need a load generation tool capable of simulating thousands of simultaneous image build requests, tracking success rates, and measuring latency percentiles.

Why Go for Stress Testing?

Go (Golang) is uniquely positioned for building load testing tools. Its concurrency model, built on goroutines and channels, allows developers to spawn thousands of concurrent tasks with minimal memory overhead compared to traditional threading models. Additionally, its robust standard library (net/http) makes handling network connections and HTTP requests extremely efficient.

Designing the Load Tester

Our load testing tool needs a few core components:

1. Worker Pool: A set of goroutines executing requests concurrently.
2. Task Queue: A channel feeding build configurations (e.g., Dockerfiles, context URLs) to the workers.
3. Metrics Aggregator: A thread-safe way to collect success counts, error rates, and response times.

Defining the Data Structures

First, we define the structures to represent our tasks and results:

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"net/http"
	"sync"
	"time"
)

// BuildRequest represents the payload sent to the container build service.
type BuildRequest struct {
	ImageName  string `json:"image_name"`
	Dockerfile string `json:"dockerfile"`
	Context    string `json:"context_url,omitempty"`
}

// Result tracks the outcome of a single build request.
type Result struct {
	Duration   time.Duration
	StatusCode int
	Error      error
}

Creating the Worker Pool

We use a standard worker pool pattern to control the exact amount of concurrency. A sync.WaitGroup ensures the main thread waits until all requests are completed.

func worker(id int, jobs <-chan BuildRequest, results chan<- Result, wg *sync.WaitGroup, endpoint string) {
	defer wg.Done()
	
	client := &http.Client{
		Timeout: 60 * time.Second, // Allow adequate time for dynamic builds
	}

	for job := range jobs {
		start := time.Now()
		
		payload, err := json.Marshal(job)
		if err != nil {
			results <- Result{Error: err}
			continue
		}

		req, err := http.NewRequest("POST", endpoint, bytes.NewBuffer(payload))
		if err != nil {
			results <- Result{Error: err}
			continue
		}
		req.Header.Set("Content-Type", "application/json")

		resp, err := client.Do(req)
		duration := time.Since(start)

		if err != nil {
			results <- Result{Duration: duration, Error: err}
			continue
		}
		
		// Read/close body to reuse connection
		resp.Body.Close()
		
		results <- Result{
			Duration:   duration,
			StatusCode: resp.StatusCode,
		}
	}
}

Orchestrating the Test

Next, we orchestrate the queue, workers, and metrics collection.

func main() {
	endpoint := "https://api.build-service.internal/v1/build"
	totalRequests := 5000
	concurrencyLimit := 200

	jobs := make(chan BuildRequest, totalRequests)
	results := make(chan Result, totalRequests)
	var wg sync.WaitGroup

	// Start workers
	for i := 1; i <= concurrencyLimit; i++ {
		wg.Add(1)
		go worker(i, jobs, results, &wg, endpoint)
	}

	// Queue up jobs
	go func() {
		for i := 0; i < totalRequests; i++ {
			jobs <- BuildRequest{
				ImageName:  fmt.Sprintf("load-test-image-%d", i),
				Dockerfile: "FROM alpine:latest\nRUN echo 'Hello World'",
			}
		}
		close(jobs)
	}()

	// Wait in a separate goroutine to close results channel
	go func() {
		wg.Wait()
		close(results)
	}()

	// Aggregate metrics
	var successCount, errorCount int
	var totalDuration time.Duration

	for res := range results {
		if res.Error != nil || res.StatusCode >= 400 {
			errorCount++
		} else {
			successCount++
		}
		totalDuration += res.Duration
	}

	fmt.Printf("Test Completed.\n")
	fmt.Printf("Total Requests: %d\n", totalRequests)
	fmt.Printf("Successes: %d\n", successCount)
	fmt.Printf("Errors: %d\n", errorCount)
	fmt.Printf("Average Latency: %v\n", totalDuration/time.Duration(totalRequests))
}

Tracking Metrics and Handling Failures

While average latency is useful, it obscures outliers. In a real-world scenario, you should track percentiles (p50, p95, p99). You can store durations in a slice, sort them, and extract the required percentiles to see how the long-tail requests behave under pressure.

Rate Limits and Backoff

If your error rate is high due to 429 Too Many Requests, your build service might be hitting upstream container registry limits. When testing, you might want to implement an exponential backoff in your HTTP client, or conversely, verify that your build service implements caching and retries correctly internally so that your client doesn’t have to.

Conclusion

Stress testing a container build engine exposes bottlenecks in caching mechanisms, database I/O, and network connection pools. By leveraging Go’s lightweight concurrency primitives, you can easily generate thousands of simultaneous requests and carefully measure how your cloud-native infrastructure holds up under real-world pressure.

In future iterations, you can expand this load tester to randomly vary Dockerfiles, attach large build contexts, and simulate real API authentication headers to perfectly replicate production traffic.