The Fourth Phase of Testing: Why Chaos Engineering Belongs in Your CI Pipeline
Engineering category.testing category.chaos-engineering Featured

The Fourth Phase of Testing: Why Chaos Engineering Belongs in Your CI Pipeline

Unit tests, integration tests, container tests โ€” and now chaos tests. We built a three-layer chaos engineering stack in C99, Java, and JUnit 5 that turns production incidents into reproducible CI gates.

E
Engineering Team
Senior Solutions Architects
June 26, 2026
12 min read

The Test That Passed. The Production Incident That Didn't.

It's 3 AM. Your service is throwing 503s. The circuit breaker you're confident works correctly โ€” because it has tests โ€” is not opening. Your retry logic, also tested, is amplifying the problem by 9x. Your DNS resolver is churning EAI_AGAIN against a CoreDNS cluster that's overloaded, and every retry is making it worse. You roll back. The incident closes. You write the post-mortem. Six weeks later, you deploy again. The same failure mode, slightly different shape. The tests still pass. We've seen this pattern dozens of times. The tests aren't wrong. The tested behavior is real. But there's a category of failure that no amount of unit tests, integration tests, or container tests will ever catch: what happens when the infrastructure beneath your application starts behaving in ways that are technically valid but operationally hostile. What happens when DNS starts flapping? When network writes return short? When your connection pool drains to zero, not because of a bug, but because of coordinated load? That's the gap. And it's why we built a chaos testing stack.

The Four Phases of Modern Testing

Most engineering teams think about testing in three phases. We argue there are four, and that skipping the fourth is what makes night-time incidents possible.

                      Phase 1: Unit Tests
  โ””โ”€โ”€ Tests a single unit of logic in isolation
  โ””โ”€โ”€ Fast, precise, tells you the code is correct
  โ””โ”€โ”€ Cannot tell you the system handles failure

Phase 2: Integration Tests
  โ””โ”€โ”€ Tests interactions between components
  โ””โ”€โ”€ Catches interface mismatches and contract violations
  โ””โ”€โ”€ Still uses predictable, cooperative dependencies

Phase 3: Container Tests (Testcontainers, Docker Compose)
  โ””โ”€โ”€ Runs real databases, brokers, caches
  โ””โ”€โ”€ Validates real-system behavior with real dependencies
  โ””โ”€โ”€ Dependencies are healthy, well-behaved, and available

Phase 4: Chaos Tests   โ† the missing phase
  โ””โ”€โ”€ Injects controlled failures into real system components
  โ””โ”€โ”€ Validates recovery logic, circuit breakers, retry strategies
  โ””โ”€โ”€ Proves your system handles the unexpected, not just the expected
                    

What the First Three Phases Cannot Test

Let's be specific about the gap. Here's what Phases 1โ€“3 cannot validate: **Your circuit breaker actually opens.** You can unit test the Resilience4j configuration, but the circuit breaker won't open unless the downstream call actually fails at the right rate, at the right timing, with the right exception type. Mocks make the failure too clean. **Your retry logic doesn't amplify failure.** A 5x retry across three replicas against a failing backend isn't 5x amplification โ€” it's 15x. This is invisible in unit tests. It's invisible in integration tests unless you carefully engineer partial failures. **Your DNS client handles transient resolution failures.** getaddrinfo() returning EAI_AGAIN is a valid, expected POSIX response. Most services never test this path. The result: a CoreDNS hiccup cascades into a 30-second brownout. **Your connection pool recovers under pressure.** Pool exhaustion followed by partial recovery is a nuanced state machine. The recovery logic is usually written once, tested never, and discovered in production. **Your timeout stack composes correctly.** A 30s read timeout sitting behind a 10s HTTP client timeout behind a 5s load balancer timeout โ€” the composed behavior is not what any individual test exercises.

Introducing the Chaos Testing Stack

We built three open-source libraries that together cover the entire surface area of Phase 4 chaos testing. **Library 1 โ€” C99 LD_PRELOAD Chaos (`chaos-testing-libraries`):** A set of shared objects that intercept libc calls at the dynamic linker level. Language-agnostic โ€” works on any ELF binary: Java, Go, Python, Node.js, Rust with CGO. Six fault domains: file I/O, networking, DNS, clock, memory, and process lifecycle. Zero application code changes required. The entire interface is a plain-text config file. **Library 2 โ€” JVM Chaos Agent (`chaos-testing-java-agent`):** A Java agent that instruments 62 JDK call sites using ByteBuddy. Injects failures, delays, and resource pressure at the JVM level โ€” inside HikariCP, inside Netty, inside the JDK's own SSL and DNS stacks. Session-isolated per test, so multiple chaos tests can run concurrently in the same JVM without interfering with each other. **Library 3 โ€” Java Testing Framework (`chaos-testing`):** A JUnit 5 extension system with 448 L1 annotations (raw syscall primitives), 92 L2 composites (documented failure patterns), and 64 L3 scenarios (real production incidents reproduced as single annotations). Integrates with Spring Boot 3/4, Quarkus, and Micronaut. Handles LD_PRELOAD injection into Docker containers automatically via the Docker API. All three share the same selector ร— effect ร— probability DSL grammar. You learn the mental model once. You apply it at every layer.

Layer 1: Kernel-Real Faults for Any Process

The C99 library set operates at the lowest level available to user-space code: the libc boundary. When your Java process calls read() on a socket, it goes through glibc. When your Python script calls socket.connect(), it goes through glibc. That's where we intercept โ€” before the kernel, after your code. Six independent shared objects, each owning a disjoint set of symbols so they can be composed without conflict: - `libchaos-io.so` โ€” file read/write, fsync, torn writes (short returns), single-bit read corruption - `libchaos-net.so` โ€” connect, send, recv, accept, with ERRNO injection, latency, and payload corruption - `libchaos-dns.so` โ€” getaddrinfo/getnameinfo, with name rewrites, EAI_AGAIN, EAI_FAIL, answer shuffling and limiting - `libchaos-time.so` โ€” clock_gettime, nanosleep with signed time skew and latency injection - `libchaos-memory.so` โ€” mmap/munmap/mprotect with ENOMEM injection at configurable probability - `libchaos-process.so` โ€” fork, pthread_create, posix_spawn, exec โ€” fail-after-N semantics and latency The interface is a plain-text config file at a well-known path. No daemons, no APIs, no agents to run. Write a config, preload the library, run your process.

                      # /tmp/.chaos-net.conf โ€” 10% connection failures to your downstream
tcp4://payments-service:8080:connect:ECONNREFUSED:0.10

# /tmp/.chaos-dns.conf โ€” transient DNS failures for internal services
dns://*.internal:EAI_AGAIN:0.15

# Inject faults into any process โ€” no code changes
LD_PRELOAD=libchaos-net.so:libchaos-dns.so ./your-service
                    

Layer 2: 62 JDK Interception Points

The C99 layer is powerful but coarse: it operates on OS-level calls. For JVM workloads, you need finer precision. The difference between HikariCP timing out a connection borrow versus a TCP connection timing out at the kernel is exactly the distinction that matters for circuit breaker tuning. The JVM agent instruments 62 JDK call sites using ByteBuddy with @Advice inlining. After JIT warm-up, the chaos dispatch compiles down to a null check and an untaken branch โ€” near-zero overhead in the common case. Critically: each test gets a ChaosSession backed by ThreadLocal<UUID>. Scenarios register as session-scoped, firing only on threads that carry the test's session ID. Multiple tests run concurrently in the same JVM without interference โ€” the chaos from test A is invisible to test B. Spring Boot 3 and 4 starters make the wiring transparent โ€” one test dependency, one annotation, and the agent self-attaches:

                      @SpringBootTest
@ChaosTest
class PaymentServiceChaosTest {

    @Test
    void circuitBreaker_opens_on_jdbcPoolExhaustion(ChaosSession session) {
        try (var h = session.activate(
                ChaosScenario.jdbc()
                    .reject("chaos: pool exhausted")
                    .probability(1.0)
                    .maxApplications(3)
                    .build())) {

            try (var scope = session.bind()) {
                // First 3 JDBC borrows fail โ†’ circuit must open
                assertThat(service.charge(order))
                    .isEqualTo(PaymentResult.REJECTED_CIRCUIT_OPEN);
            }
        }
    }
}
                    

Layer 3: 604 Annotations, 64 Production Incidents

The Java framework is what makes chaos testing accessible at the team level. Instead of learning fault DSLs and tuning probabilities, a developer annotates a test with the name of a real incident pattern and the framework handles the composition. **448 L1 annotations** give surgical control at the syscall level: `@ChaosRecvEconnreset(probability = 0.05)`, `@ChaosDnsGetaddrinfoEaiAgain(hostPattern = "*.internal", probability = 0.20)`, `@ChaosMmapEnomem(probability = 0.001)` **92 L2 composites** group related faults into documented patterns: `@CompositeChaosTransientDnsFailure`, `@CompositeChaosConnectionRefused`, `@CompositeChaosLowMemoryPressure` **64 L3 scenarios** encode real production incidents as single annotations โ€” each with a severity rating, source reference, and a multi-domain Composer class that knows how to decompose the incident into the right mix of syscall faults: `@IncidentChaosK8sRollingUpdateRst` โ€” ECONNRESET on 30% of RECV calls, reproducing the iptables lag window during Kubernetes rolling updates `@IncidentChaosFeignRetryAmplification` โ€” ECONNREFUSED on 50% of connect() plus IOException injection at the Feign level, reproducing the retry storm pattern that creates 9x load amplification `@IncidentChaosSpringTransactionalPoolDeadlock` โ€” the @Transactional(REQUIRES_NEW) deadlock pattern that drains the connection pool from the inside The framework automatically injects the right LD_PRELOAD libraries into your Docker container before the test starts โ€” via the Docker API tar stream, with no shell exec, no sidecar, no image modification required.

                      @Testcontainers
@ExtendWith(ChaosTestingExtension.class)
@SyscallLevelChaos(LibchaosLib.NET)
class OrderServiceChaosTest {

    @Container @AppContainer
    static GenericContainer<?> app =
        new GenericContainer<>("order-service:latest")
            .withExposedPorts(8080);

    @Test
    @IncidentChaosK8sRollingUpdateRst(toxicity = 0.3)
    void service_handles_rst_during_rolling_deploy() {
        // 30% of RECV calls return ECONNRESET โ€” exactly what happens
        // during a Kubernetes rolling update when iptables lags behind
        assertThat(callService("/api/order/42")).isEqualTo(200);
    }
}
                    

One DSL, Three Layers, No Gaps

The key design decision is that all three layers share the same conceptual grammar: selector ร— effect ร— policy. A **selector** targets what to fault: a file path prefix, a TCP endpoint, a JDK operation type, a DNS name pattern. An **effect** defines how to fault it: ERRNO injection, latency, data corruption, rejection, value boundary corruption. A **policy** controls when to fault: probability, rate limit, warm-up count, time window, max applications. Whether you're writing a C99 config file, building a ChaosScenario for the JVM agent, or picking a JUnit 5 annotation, you're working with the same mental model. The developer who understands `@IncidentChaosFeignRetryAmplification` can drill into the underlying L1 annotations and read exactly which syscalls are being faulted, at what rates, and why. And all three layers can be active simultaneously. A single test run can have: - The container running with `libchaos-net.so` injected (30% ECONNRESET) - The JVM agent producing code cache pressure (background stressor) - A ChaosSession with JDBC rejection active (first 3 pool borrows fail) All three independently controlled, all three scoped correctly, none interfering with each other.

Chaos as a First-Class CI Citizen

The shift we're advocating is not about Game Days or chaos experiments in production. It's simpler and more valuable: **every PR that touches error handling, retry logic, circuit breakers, or connection management should have a chaos test that gates the build**. This reframes chaos engineering from an ops discipline into a development discipline. The question is no longer "can our platform survive a network partition?" โ€” answered in production, too late. The question becomes "does this PR's circuit breaker open correctly when the JDBC pool exhausts, before this code ships?" The tooling exists. The annotations are one import away. The Docker injection is automatic. The overhead of adding `@IncidentChaosK8sRollingUpdateRst` to an integration test is the same as adding any other JUnit annotation. The overhead of not having it is what you pay at 3 AM.

Where to Start

Each library has its own deep-dive post, but if you want chaos tests running today, start with the Java framework โ€” it has the most complete getting-started story and the most immediate impact. **Step 1:** Add the dependency for your framework (Spring Boot 3/4, Quarkus, or Micronaut). **Step 2:** Annotate your existing container tests with @ExtendWith(ChaosTestingExtension.class) and @SyscallLevelChaos. **Step 3:** Pick one L3 annotation that matches a failure mode you've seen in production or worry about. **Step 4:** Run the test. Watch it fail if your resilience code isn't there yet. Fix it. Ship it with confidence. The chaos layer doesn't replace your other test phases. It completes them.
  • chaos-testing-libraries โ€” C99 LD_PRELOAD faults for any ELF process, any language
  • chaos-testing-java-agent โ€” JVM bytecode agent, 62 JDK call sites, per-test session isolation
  • chaos-testing โ€” JUnit 5 framework, Spring Boot / Quarkus / Micronaut, 64 incident annotations

tags

#chaos-engineering #testing #ci-cd #java #c99 #ldpreload #jvm #junit5 #spring-boot #resilience

conclusion

The testing pyramid was never wrong โ€” it was incomplete. Unit tests verify correctness. Integration tests verify contracts. Container tests verify behavior against real dependencies. Chaos tests verify survival against a hostile environment. We built three libraries that cover this last mile: a C99 LD_PRELOAD toolkit for kernel-real libc interception, a JVM bytecode agent with 62 interception points and per-test session isolation, and a JUnit 5 annotation framework with 64 real production incidents encoded as reusable test scenarios. Phase 4 is ready. Your CI pipeline is waiting.
E

Engineering Team

Senior Solutions Architects

We've been building distributed systems since before 'microservices' was a thing. Our scars tell stories.