62 JDK Interception Points: In-Process Chaos Engineering with a Java Agent
Engineering category.testing category.chaos-engineering

62 JDK Interception Points: In-Process Chaos Engineering with a Java Agent

A Java agent that instruments 62 JDK call sites using ByteBuddy. Per-test session isolation, Spring Boot 3/4 auto-wiring, and near-zero JIT overhead. Chaos engineering that runs inline in your JUnit test suite and gates the build.

E
Engineering Team
Senior Solutions Architects
June 26, 2026
18 min read

The Layer Toxiproxy Can't Reach

Toxiproxy is excellent. tc netem is excellent. The LD_PRELOAD chaos libraries are excellent. They all operate at the network or OS boundary โ€” intercepting TCP connections, packet delivery, and system calls. But here's what none of them can see: what happens inside your HikariPool when it tries to borrow a connection and hits a timeout. What happens inside your ScheduledExecutorService when it's been saturated. What happens inside the JDK's SSL implementation when it encounters a retransmit. What happens inside your thread pool when tasks start piling up faster than workers can drain them. These are JVM-internal failure modes. They don't manifest as network failures at the TCP layer. They manifest as exceptions thrown from JDK classes โ€” `SQLException`, `RejectedExecutionException`, `SSLHandshakeException`, `TimeoutException` โ€” and as resource exhaustion patterns that only become visible when you instrument the right call sites inside the JVM itself. This is the gap the JVM chaos agent fills. It doesn't replace the LD_PRELOAD layer โ€” it composes with it. Together, they give you fault injection at both the OS boundary and the JVM-internal boundary: the complete surface area of a Java application's failure modes.

62 JDK Call Sites: The Full Surface Area

The agent instruments 62 specific JDK call sites โ€” not random ones, but the ones that matter for enterprise backend workloads. The selection was driven by incident analysis across distributed systems failures: **Threading & Executors** Thread.start(), ThreadPoolExecutor.execute(), ScheduledExecutorService.schedule(), CompletableFuture.runAsync/supplyAsync, BlockingQueue operations (put, offer, poll, take), ForkJoinPool submissions, virtual thread lifecycle **Network & I/O** Socket.connect/read/write, SocketChannel.connect/read/write, Selector.select/selectNow, DatagramChannel operations, ServerSocket.accept **DNS & Name Resolution** InetAddress.getByName/getAllByName โ€” the JDK's internal resolution path before the OS resolver is called **JDBC & Data Access** DataSource.getConnection (connection pool borrow), Statement.execute/executeQuery, PreparedStatement operations โ€” at the level where Hikari, DBCP, and c3p0 all pass through **HTTP Clients** HttpURLConnection, HttpClient (JDK 11+) โ€” send, sendAsync **SSL/TLS** SSLEngine.wrap/unwrap, SSLSocket.startHandshake **Time** System.currentTimeMillis(), System.nanoTime() โ€” with FIXED offset, DRIFT, and FREEZE effects **JVM Internals** System.gc(), class loading, reflection, object serialization/deserialization, ThreadLocal.get/set, JNDI lookups, JMX operations, native library loading, ZIP inflation

                      // 62 interception targets across the JDK surface
Sealed interface ChaosSelector permits
    ThreadSelector,       // Thread.start, pool operations
    ExecutorSelector,     // ThreadPoolExecutor, ForkJoinPool
    QueueSelector,        // BlockingQueue operations
    FutureSelector,       // CompletableFuture
    SocketSelector,       // Socket, ServerSocket
    NioSelector,          // SocketChannel, Selector
    JdbcSelector,         // DataSource.getConnection, Statement
    HttpSelector,         // HttpURLConnection, HttpClient
    DnsSelector,          // InetAddress resolution
    SslSelector,          // SSLEngine, SSLSocket
    ClockSelector,        // System.currentTimeMillis, nanoTime
    GcSelector,           // System.gc
    ClassloadingSelector, // ClassLoader.loadClass
    SerializationSelector,// ObjectInputStream, ObjectOutputStream
    ThreadLocalSelector,  // ThreadLocal.get, set
    VirtualThreadSelector;// VirtualThread lifecycle
                    

ByteBuddy + Bootstrap Bridge: How It Works

Instrumenting JDK classes is non-trivial. JDK classes are loaded by the bootstrap classloader โ€” the root classloader that has no parent. Agent code lives in the agent classloader, which the bootstrap classloader cannot see by name. Bridging this gap correctly requires careful use of JVMTI and the Java Memory Model. **Step 1 โ€” Premain or agentmain.** The agent attaches either at startup (via `-javaagent:`) or dynamically at test time (via the JDK Attach API). When attaching dynamically, the Spring Boot test starter calls `VirtualMachine.attach(processId)` and injects the agent jar. **Step 2 โ€” Bootstrap injection.** A `BootstrapDispatcher` class is extracted from the agent jar and written to a temporary JAR. This JAR is appended to the bootstrap classpath via `Instrumentation.appendToBootstrapClassLoaderSearch()`. The bootstrap classloader can now see `BootstrapDispatcher`. **Step 3 โ€” MethodHandle wiring.** A 62-slot `MethodHandle[]` array is built, one handle per interception target, pointing into the agent classloader's implementation. The array and a `delegate` object are published into `BootstrapDispatcher` as `volatile` fields. The JMM guarantees that any thread observing `delegate != null` also sees the fully initialized `handles` array. **Step 4 โ€” Bytecode advice inlining.** ByteBuddy's `@Advice` mechanism copies the advice bytecode directly into the instrumented JDK method body. There's no virtual dispatch, no interface call โ€” the woven bytecode is inline. After JIT compilation at warm-up (~10,000 invocations), the entire dispatch chain compiles to native code.

                      // Hot path after JIT compilation โ€” effectively just:
public static void connect(Socket socket, SocketAddress endpoint, int timeout) {
    Object delegate = BootstrapDispatcher.delegate;  // volatile read
    if (delegate != null) {                           // null check โ€” fast path
        BootstrapDispatcher.handles[SOCKET_CONNECT]   // MethodHandle
            .invoke(delegate, socket, endpoint);       // โ†’ ChaosRuntime
    }
    originalConnect(socket, endpoint, timeout);       // real JDK call
}
// Zero-scenario overhead: one volatile read + one null check + one untaken branch
// Measured: ~60 ns per call in the uncontested case after warm-up
                    

Session Isolation: Each Test Gets Its Own Chaos

The hardest engineering problem in a JVM-level chaos agent is not the bytecode instrumentation โ€” it's isolation. Multiple JUnit tests run concurrently in the same JVM. If test A injects JDBC connection failures, test B cannot be allowed to see those failures. The solution is `ChaosSession`, backed by `ThreadLocal<UUID>`. Each test method (or test class, depending on lifecycle) gets a unique session ID. Chaos scenarios can be registered as either JVM-scoped (affects all threads) or session-scoped (affects only threads that carry this test's session ID). The `ThreadLocal` provides the binding. For executor-submitted tasks, the session ID must propagate from the submitting thread into the worker thread. The agent instruments `ThreadPoolExecutor.execute()` to wrap the submitted `Runnable` with a decorator that captures the current session ID and restores it on the worker thread before the task runs. This propagation happens transparently โ€” the application code doesn't know it's happening.

                      @ChaosTest  // @SpringBootTest composed annotation
class OrderServiceChaosTest {

    @Test
    void retryLogic_handles_transient_jdbc_failures(ChaosSession session) {
        var scenario = ChaosScenario.builder("transient-db")
            .selector(ChaosSelector.jdbc(OperationType.JDBC_CONNECTION_ACQUIRE))
            .effect(ChaosEffect.reject("chaos: simulated pool timeout"))
            .activationPolicy(ActivationPolicy.builder()
                .probability(1.0)
                .maxApplications(2)   // First 2 borrows fail, 3rd succeeds
                .build())
            .build();

        try (var handle = session.activate(scenario)) {
            try (var scope = session.bind()) {
                // This thread and any executor tasks it submits
                // will see the JDBC failures โ€” nothing else will
                var result = orderService.placeOrder(testOrder);
                assertThat(result.status()).isEqualTo(OrderStatus.CONFIRMED);
            }
        }
        // Chaos cleaned up โ€” next test is unaffected
    }
}
                    

Spring Boot 3 and 4: Zero-Config Wiring

The Spring Boot test starters exist because the wiring should be invisible. You should not need to understand ByteBuddy, the bootstrap classloader, or JVMTI to run your first chaos test. The starter handles: - Programmatically setting `jdk.attach.allowAttachSelf=true` before the JVM processes this flag (using a static initializer in the starter's auto-configuration class) - Detecting whether the agent is already attached (idempotent โ€” safe in parallel test class execution) - Attaching the agent via `VirtualMachine.attach("0")` (self-attach) if not already present - Registering `ChaosControlPlane` and `ChaosSession` as Spring beans in the test ApplicationContext - Providing JUnit 5 parameter resolution for `ChaosSession` and `ChaosControlPlane` constructor arguments The runtime starter (for non-test deployment) additionally exposes `/actuator/chaos` as a protected Actuator endpoint for live scenario management.

                      <!-- Spring Boot 3 โ€” add to pom.xml testImplementation -->
<dependency>
    <groupId>com.macstab.chaos</groupId>
    <artifactId>chaos-agent-spring-boot3-test-starter</artifactId>
    <version>${chaos-agent.version}</version>
    <scope>test</scope>
</dependency>

<!-- That's it. No -javaagent flag. No JVM args. No beans to define. -->

<!-- Spring Boot 4 -->
<dependency>
    <groupId>com.macstab.chaos</groupId>
    <artifactId>chaos-agent-spring-boot4-test-starter</artifactId>
    <version>${chaos-agent.version}</version>
    <scope>test</scope>
</dependency>
                    

The Eight-Gate Activation Pipeline

Every chaos scenario evaluation goes through eight gates in sequence. All gates must pass for the effect to fire. The gates are fast โ€” evaluated on the calling thread with no I/O and minimal locking. **Gate 1 โ€” Started check.** Has the scenario been activated? (Cheap flag read.) **Gate 2 โ€” Session ID match.** If session-scoped, does the current thread's `ThreadLocal` carry the right UUID? (Thread-local read, no lock.) **Gate 3 โ€” Selector match.** Does this call site match the scenario's selector? (Enum comparison, O(1).) **Gate 4 โ€” Activation window.** Is the current time within the scenario's configured start/end window? (Two long comparisons.) **Gate 5 โ€” Warm-up count.** Has this scenario matched at least N times before starting to apply? (AtomicLong read.) **Gate 6 โ€” Rate limit.** Does the sliding-window token bucket have capacity? (Synchronized block; uncontested path ~5 ns.) **Gate 7 โ€” Probability.** Random draw against configured probability. (SplittableRandom seeded with scenario ID + match count โ€” deterministic for reproducing failures.) **Gate 8 โ€” Max applications.** CAS loop on AtomicLong to enforce a hard cap. (compareAndSet loop; correct under contention โ€” unlike a simple incrementAndGet which can overshoot.) All eight gates pass โ†’ apply the effect. Any gate fails โ†’ call through to the real JDK method unchanged.

Background Stressors: Beyond Request-Path Faults

In addition to inline request-path effects (delay, rejection, exception injection), the agent supports background stressors โ€” lifecycle-bound threads that continuously apply resource pressure independent of traffic. Stressors simulate the slow-burn failure modes that aren't triggered by specific operations but accumulate over time: **Memory stressors:** Heap pressure (retain byte[] chunks), GC pressure (allocation churn), Metaspace pressure (synthetic class definitions), direct buffer pressure (off-heap ByteBuffer), string intern flooding **JVM stressors:** Code cache pressure (ByteBuddy-generated class flood for JIT thrashing), finalizer backlog (phantom-reference queue flood), reference queue flooding, safepoint storms (periodic retransformation to trigger stop-the-world pauses) **Thread stressors:** Thread leak (permanently parked threads consuming thread stack memory), ThreadLocal leak (entries on pooled threads that accumulate), deadlock injection (real JVM monitor deadlock between N threads โ€” verified with ThreadMXBean), monitor contention (background threads competing for a shared lock), keep-alive threads (prevent JVM shutdown) Stressors are started when a scenario is activated and stopped when it's closed. They compose with inline effects: you can simultaneously have request-path JDBC rejections and a code-cache stressor active, testing whether your application recovers from connection failures while already under JIT compilation pressure.
  • Heap pressure: retain configurable MB/s in byte[] allocations to force GC pressure
  • Metaspace pressure: define synthetic classes into an isolated ClassLoader to eat permgen/metaspace
  • Code cache pressure: generate ByteBuddy classes to saturate the JIT compiler
  • Safepoint storm: force periodic retransformation to trigger stop-the-world pauses
  • Real deadlock: two threads acquire monitors in opposite order โ€” detected by ThreadMXBean
  • Thread leak: park threads permanently to exhaust -Xss stack memory over time

Example: Validating SLA Behavior Under JDBC Exhaustion

A concrete test scenario that exercises the agent's core value: proving that your service responds correctly to a payment with an SLA target when the database connection pool is under pressure.

                      @SpringBootTest
@ChaosTest
class PaymentSLAChaosTest {

    @Autowired PaymentService paymentService;

    @Test
    void payment_meets_sla_under_intermittent_jdbc_pressure(ChaosSession session) {
        // Activate 20% JDBC connection failure โ€” simulates pool under load
        var jdbcChaos = ChaosScenario.builder("jdbc-pressure")
            .selector(ChaosSelector.jdbc(OperationType.JDBC_CONNECTION_ACQUIRE))
            .effect(ChaosEffect.delay(Duration.ofMillis(150)))  // Not rejection โ€” delay
            .activationPolicy(ActivationPolicy.probability(0.20))
            .build();

        // Also activate a background heap stressor
        var heapStressor = ChaosScenario.builder("heap-pressure")
            .effect(ChaosEffect.stressor(StressorType.HEAP_PRESSURE)
                .retainMbPerSecond(50)
                .build())
            .build();

        try (var h1 = session.activate(jdbcChaos);
             var h2 = session.activate(heapStressor)) {

            try (var scope = session.bind()) {
                long start = System.nanoTime();
                var result = paymentService.process(testPayment);
                long elapsedMs = (System.nanoTime() - start) / 1_000_000;

                assertThat(result.status()).isEqualTo(PaymentStatus.COMPLETED);
                assertThat(elapsedMs).isLessThan(500);  // SLA: p99 < 500ms
            }
        }
    }
}
                    

JMX and Observability

The agent exposes a JMX MBean at `com.macstab.chaos.jvm:type=ChaosDiagnostics` that provides a live snapshot of all active scenarios โ€” their current state, match counts, and applied counts โ€” without requiring any code changes or additional configuration. This is useful during test debugging: if a chaos test is failing unexpectedly, you can connect JConsole or VisualVM to the test JVM and inspect exactly which scenarios are active and how many times they've fired. The observability bus allows publishing chaos events to external metrics systems. If you're running chaos tests in a shared integration environment and want to correlate chaos events with Grafana dashboards or distributed traces, the bus provides the integration point.
  • JMX MBean: active scenario list, match counts, applied counts, current state per scenario
  • In-process snapshot API: queryable from test code for assertions on chaos behavior
  • Debug dump: text representation of all scenarios and state โ€” useful in test failure output
  • Observability bus: pluggable event publisher for metrics/tracing integration

tags

#chaos-engineering #java #jvm #java-agent #bytebuddy #junit5 #spring-boot #bytecode #resilience #ci-cd

conclusion

The JVM chaos agent fills the gap between infrastructure-level fault injection and real JVM-internal failure modes. 62 JDK call sites, per-test session isolation, Spring Boot 3/4 auto-wiring, background stressors for slow-burn failures, and a JIT-optimized dispatch path that costs ~60 ns per call in the zero-scenario case. The agent composes with the C99 LD_PRELOAD library for full-stack fault coverage: syscall-level failures below the JVM, bytecode-level failures inside it. Both are orchestrated by the Java testing framework's annotation system โ€” one test, both layers active, zero configuration overlap. Chaos engineering as a CI gate is ready. The only question is which failure mode your circuit breaker is not handling yet.
E

Engineering Team

Senior Solutions Architects

We've been building distributed systems since before 'microservices' was a thing. Our scars tell stories.