The Test That Passed. The Production Incident That Didn't.
The Four Phases of Modern Testing
Phase 1: Unit Tests
โโโ Tests a single unit of logic in isolation
โโโ Fast, precise, tells you the code is correct
โโโ Cannot tell you the system handles failure
Phase 2: Integration Tests
โโโ Tests interactions between components
โโโ Catches interface mismatches and contract violations
โโโ Still uses predictable, cooperative dependencies
Phase 3: Container Tests (Testcontainers, Docker Compose)
โโโ Runs real databases, brokers, caches
โโโ Validates real-system behavior with real dependencies
โโโ Dependencies are healthy, well-behaved, and available
Phase 4: Chaos Tests โ the missing phase
โโโ Injects controlled failures into real system components
โโโ Validates recovery logic, circuit breakers, retry strategies
โโโ Proves your system handles the unexpected, not just the expected
What the First Three Phases Cannot Test
Introducing the Chaos Testing Stack
Layer 1: Kernel-Real Faults for Any Process
# /tmp/.chaos-net.conf โ 10% connection failures to your downstream
tcp4://payments-service:8080:connect:ECONNREFUSED:0.10
# /tmp/.chaos-dns.conf โ transient DNS failures for internal services
dns://*.internal:EAI_AGAIN:0.15
# Inject faults into any process โ no code changes
LD_PRELOAD=libchaos-net.so:libchaos-dns.so ./your-service
Layer 2: 62 JDK Interception Points
@SpringBootTest
@ChaosTest
class PaymentServiceChaosTest {
@Test
void circuitBreaker_opens_on_jdbcPoolExhaustion(ChaosSession session) {
try (var h = session.activate(
ChaosScenario.jdbc()
.reject("chaos: pool exhausted")
.probability(1.0)
.maxApplications(3)
.build())) {
try (var scope = session.bind()) {
// First 3 JDBC borrows fail โ circuit must open
assertThat(service.charge(order))
.isEqualTo(PaymentResult.REJECTED_CIRCUIT_OPEN);
}
}
}
}
Layer 3: 604 Annotations, 64 Production Incidents
@Testcontainers
@ExtendWith(ChaosTestingExtension.class)
@SyscallLevelChaos(LibchaosLib.NET)
class OrderServiceChaosTest {
@Container @AppContainer
static GenericContainer<?> app =
new GenericContainer<>("order-service:latest")
.withExposedPorts(8080);
@Test
@IncidentChaosK8sRollingUpdateRst(toxicity = 0.3)
void service_handles_rst_during_rolling_deploy() {
// 30% of RECV calls return ECONNRESET โ exactly what happens
// during a Kubernetes rolling update when iptables lags behind
assertThat(callService("/api/order/42")).isEqualTo(200);
}
}
One DSL, Three Layers, No Gaps
Chaos as a First-Class CI Citizen
Where to Start
- chaos-testing-libraries โ C99 LD_PRELOAD faults for any ELF process, any language
- chaos-testing-java-agent โ JVM bytecode agent, 62 JDK call sites, per-test session isolation
- chaos-testing โ JUnit 5 framework, Spring Boot / Quarkus / Micronaut, 64 incident annotations
tags
conclusion
Engineering Team
Senior Solutions Architects
We've been building distributed systems since before 'microservices' was a thing. Our scars tell stories.