The Layer Toxiproxy Can't Reach
62 JDK Call Sites: The Full Surface Area
// 62 interception targets across the JDK surface
Sealed interface ChaosSelector permits
ThreadSelector, // Thread.start, pool operations
ExecutorSelector, // ThreadPoolExecutor, ForkJoinPool
QueueSelector, // BlockingQueue operations
FutureSelector, // CompletableFuture
SocketSelector, // Socket, ServerSocket
NioSelector, // SocketChannel, Selector
JdbcSelector, // DataSource.getConnection, Statement
HttpSelector, // HttpURLConnection, HttpClient
DnsSelector, // InetAddress resolution
SslSelector, // SSLEngine, SSLSocket
ClockSelector, // System.currentTimeMillis, nanoTime
GcSelector, // System.gc
ClassloadingSelector, // ClassLoader.loadClass
SerializationSelector,// ObjectInputStream, ObjectOutputStream
ThreadLocalSelector, // ThreadLocal.get, set
VirtualThreadSelector;// VirtualThread lifecycle
ByteBuddy + Bootstrap Bridge: How It Works
// Hot path after JIT compilation โ effectively just:
public static void connect(Socket socket, SocketAddress endpoint, int timeout) {
Object delegate = BootstrapDispatcher.delegate; // volatile read
if (delegate != null) { // null check โ fast path
BootstrapDispatcher.handles[SOCKET_CONNECT] // MethodHandle
.invoke(delegate, socket, endpoint); // โ ChaosRuntime
}
originalConnect(socket, endpoint, timeout); // real JDK call
}
// Zero-scenario overhead: one volatile read + one null check + one untaken branch
// Measured: ~60 ns per call in the uncontested case after warm-up
Session Isolation: Each Test Gets Its Own Chaos
@ChaosTest // @SpringBootTest composed annotation
class OrderServiceChaosTest {
@Test
void retryLogic_handles_transient_jdbc_failures(ChaosSession session) {
var scenario = ChaosScenario.builder("transient-db")
.selector(ChaosSelector.jdbc(OperationType.JDBC_CONNECTION_ACQUIRE))
.effect(ChaosEffect.reject("chaos: simulated pool timeout"))
.activationPolicy(ActivationPolicy.builder()
.probability(1.0)
.maxApplications(2) // First 2 borrows fail, 3rd succeeds
.build())
.build();
try (var handle = session.activate(scenario)) {
try (var scope = session.bind()) {
// This thread and any executor tasks it submits
// will see the JDBC failures โ nothing else will
var result = orderService.placeOrder(testOrder);
assertThat(result.status()).isEqualTo(OrderStatus.CONFIRMED);
}
}
// Chaos cleaned up โ next test is unaffected
}
}
Spring Boot 3 and 4: Zero-Config Wiring
<!-- Spring Boot 3 โ add to pom.xml testImplementation -->
<dependency>
<groupId>com.macstab.chaos</groupId>
<artifactId>chaos-agent-spring-boot3-test-starter</artifactId>
<version>${chaos-agent.version}</version>
<scope>test</scope>
</dependency>
<!-- That's it. No -javaagent flag. No JVM args. No beans to define. -->
<!-- Spring Boot 4 -->
<dependency>
<groupId>com.macstab.chaos</groupId>
<artifactId>chaos-agent-spring-boot4-test-starter</artifactId>
<version>${chaos-agent.version}</version>
<scope>test</scope>
</dependency>
The Eight-Gate Activation Pipeline
Background Stressors: Beyond Request-Path Faults
- Heap pressure: retain configurable MB/s in byte[] allocations to force GC pressure
- Metaspace pressure: define synthetic classes into an isolated ClassLoader to eat permgen/metaspace
- Code cache pressure: generate ByteBuddy classes to saturate the JIT compiler
- Safepoint storm: force periodic retransformation to trigger stop-the-world pauses
- Real deadlock: two threads acquire monitors in opposite order โ detected by ThreadMXBean
- Thread leak: park threads permanently to exhaust -Xss stack memory over time
Example: Validating SLA Behavior Under JDBC Exhaustion
@SpringBootTest
@ChaosTest
class PaymentSLAChaosTest {
@Autowired PaymentService paymentService;
@Test
void payment_meets_sla_under_intermittent_jdbc_pressure(ChaosSession session) {
// Activate 20% JDBC connection failure โ simulates pool under load
var jdbcChaos = ChaosScenario.builder("jdbc-pressure")
.selector(ChaosSelector.jdbc(OperationType.JDBC_CONNECTION_ACQUIRE))
.effect(ChaosEffect.delay(Duration.ofMillis(150))) // Not rejection โ delay
.activationPolicy(ActivationPolicy.probability(0.20))
.build();
// Also activate a background heap stressor
var heapStressor = ChaosScenario.builder("heap-pressure")
.effect(ChaosEffect.stressor(StressorType.HEAP_PRESSURE)
.retainMbPerSecond(50)
.build())
.build();
try (var h1 = session.activate(jdbcChaos);
var h2 = session.activate(heapStressor)) {
try (var scope = session.bind()) {
long start = System.nanoTime();
var result = paymentService.process(testPayment);
long elapsedMs = (System.nanoTime() - start) / 1_000_000;
assertThat(result.status()).isEqualTo(PaymentStatus.COMPLETED);
assertThat(elapsedMs).isLessThan(500); // SLA: p99 < 500ms
}
}
}
}
JMX and Observability
- JMX MBean: active scenario list, match counts, applied counts, current state per scenario
- In-process snapshot API: queryable from test code for assertions on chaos behavior
- Debug dump: text representation of all scenarios and state โ useful in test failure output
- Observability bus: pluggable event publisher for metrics/tracing integration
tags
conclusion
Engineering Team
Senior Solutions Architects
We've been building distributed systems since before 'microservices' was a thing. Our scars tell stories.