The Problem with Mocking System Calls
Fault injection libraries fall into two broad camps. The first camp mocks at the application layer: mock a repository, mock an HTTP client, return an exception. Clean. Fast. And utterly disconnected from what the real operating environment does when it degrades.
The second camp reaches for infrastructure tools: Toxiproxy for TCP proxy faults, tc netem for kernel-level packet shaping, Pumba or Chaos Monkey for container killing. Powerful, but require external setup, fight with test isolation, and often run in a separate process that the application cannot be aware of.
Both camps miss something: the libc boundary. The layer where your application, regardless of language or runtime, hands off to the operating system. Where `read()` actually happens. Where `getaddrinfo()` resolves names. Where `mmap()` allocates memory. Where `clock_gettime()` returns time.
If you intercept here, you intercept everything. And if the faults you inject at this layer are indistinguishable from the faults the kernel produces — because they use the same errno codes, the same return value conventions, the same timing characteristics — then what you've built isn't a mock. It's a simulation. A kernel-real simulation.
Why the libc Boundary Is the Right Layer
LD_PRELOAD is a Linux dynamic linker feature that inserts shared objects into a process's link-map before any other library — including libc. When an application calls `connect()`, the dynamic linker's symbol resolution walks the link-map in order. If a preloaded library exports a symbol named `connect`, that symbol wins.
This gives us interposition: the ability to wrap any libc function without modifying the calling code or recompiling anything. And it's been available since before most current production systems were designed.
The key properties that make this the right layer:
**Language-agnostic universality.** Python's `socket.connect()` goes through glibc. Node.js's `net.connect()` goes through glibc. Java's `Socket.connect()` — via JNI — goes through glibc. A single preloaded library intercepts all of them.
**Kernel-real faults.** We return real errno values. The calling code cannot distinguish a real ECONNREFUSED from our injected ECONNREFUSED. The circuit breaker sees a real failure, not a test-aware simulated one.
**Zero application changes.** No agents. No API imports. No test framework integration required at the application level. The application runs normally; the faults appear beneath it.
**Process scope.** Faults are scoped to a single process. Run your test process with one set of faults; run your dependency in a separate container without any faults. No cross-contamination.
Six Fault Domains
The library set covers six independent fault domains, each as a separate shared object. They can be loaded individually or composed — the composition is safe because each library owns a disjoint set of symbols (the Symbol Ownership Theorem, documented in detail in the architecture docs).
libchaos-io.so — File I/O
Hooks: open/openat, read/readv, write/writev, pread/preadv, pwrite/pwritev,
close, fsync/fdatasync, ftruncate, fallocate, unlinkat, renameat,
sendfile, copy_file_range
Effects: ERRNO (EIO, ENOSPC, EDQUOT, ...), LATENCY,
TORN writes (short return < requested), CORRUPT reads (bit flip)
libchaos-net.so — Network Sockets
Hooks: socket/socketpair, bind, listen, connect, accept,
send/sendto/sendmsg/sendmmsg, recv/recvfrom/recvmsg/recvmmsg,
poll, select, epoll_wait/epoll_pwait
Effects: ERRNO (ECONNREFUSED, ECONNRESET, ETIMEDOUT, ...), LATENCY, CORRUPT
libchaos-dns.so — DNS Resolution
Hooks: getaddrinfo, getnameinfo
Effects: Name rewrites, address synthesis, EAI_AGAIN, EAI_FAIL, EAI_NONAME,
EAI_MEMORY, answer shuffling, family filtering, result limiting
libchaos-time.so — Clock & Timing
Hooks: clock_gettime (all POSIX clocks), nanosleep, usleep
Effects: ERRNO, LATENCY, OFFSET (signed millisecond time skew)
libchaos-memory.so — Memory Allocation
Hooks: mmap, munmap, mprotect, madvise
Effects: ERRNO (ENOMEM, ENFILE), LATENCY
libchaos-process.so — Process Lifecycle
Hooks: pthread_create, fork, posix_spawn/posix_spawnp, execve/execveat,
waitpid
Effects: ERRNO, LATENCY, FAIL_AFTER (allow N calls then fail permanently)
Symbol Interposition: How It Actually Works
The mechanism is simple in principle, careful in execution.
**Step 1 — Library init.** When the preloaded `.so` is loaded, its constructor runs. It calls `dlsym(RTLD_NEXT, "connect")` to find the real libc `connect`. RTLD_NEXT skips the calling DSO in the link-map and returns the next matching symbol — the real libc implementation. This pointer is cached in a static variable. No per-call dlsym in the hot path.
**Step 2 — Call interception.** When the application calls `connect()`, the dynamic linker finds our symbol first (link-map order). Our wrapper runs: it reads the config file (or a cached snapshot), evaluates whether a fault should fire based on the endpoint selector and probability, and either injects the fault or calls through to the real libc function via the cached pointer.
**Step 3 — Symbol ownership invariant.** If two preloaded libraries both try to wrap `read()`, one of them will get RTLD_NEXT pointing to the other, causing double injection. We prevent this with strict symbol assignment: each libc symbol is owned by exactly one library in the set. `libchaos-net.so` does not own `read()` — I/O symbols belong to `libchaos-io.so`. The libraries compose safely because they're designed to.
# Compose safely — net + dns + io loaded simultaneously, no double-injection
LD_PRELOAD=libchaos-net.so:libchaos-dns.so:libchaos-io.so ./your-service
# Each library reads its own config file independently
/tmp/.chaos-net.conf
/tmp/.chaos-dns.conf
/tmp/.chaos-io.conf
The Config File Interface
There is no API. There is no public header. There is no library to link against. The entire interface is a plain-text config file.
This was a deliberate choice. A config file can be written by a shell script, a CI job, a test harness in any language, or a human in a terminal. It requires no build-system integration. It can be hot-swapped while the process is running. It can be version-controlled alongside test data. And it creates zero coupling between the chaos library and the application under test.
Config syntax is `selector:operation:effect:value:probability`. A few representative examples:
# libchaos-net.conf — network fault examples
#
# ECONNREFUSED on 10% of connects to the payments service
tcp4://payments.internal:8080:connect:ECONNREFUSED:0.10
#
# ECONNRESET on 5% of recv() on any TCP socket
tcp4://*:*:recv:ECONNRESET:0.05
#
# 200ms latency on 100% of connects to the analytics backend
tcp4://analytics.internal:9000:connect:LATENCY:200
#
# Corrupt 1% of sent packets to any endpoint (single bit flip)
tcp4://*:*:send:CORRUPT:0.01
# libchaos-dns.conf — DNS fault examples
#
# Transient failure for all *.internal lookups
dns://*.internal:EAI_AGAIN:0.20
#
# Permanent failure for a specific service
dns://legacy-payments.svc.cluster.local:EAI_FAIL:1.0
# libchaos-io.conf — filesystem fault examples
#
# Torn writes (short return) on WAL file
/data/wal:write:TORN:0.05
#
# EIO on 1% of reads from the config directory
/etc/app:read:EIO:0.01
Lock-Free Config Reload at Runtime
One of the less obvious engineering challenges is hot config reload in a multi-threaded process. The naive approach — read the file on every call — is too expensive. The naive lock-based approach — hold a mutex while updating the config snapshot — blocks every intercepted syscall during reload.
We use a lock-free approach based on atomic compare-and-swap:
**Two snapshot buffers.** One is active; the other is available for writing. A 64-bit atomic slot holds the mtime of the currently active config.
**Reload winner detection.** Each interception checks the config file's mtime against the cached value. On a mismatch, one thread wins the CAS race to begin reloading. Losers see the RELOADING state and spin briefly.
**Atomic flip.** The winner writes the new config to the inactive buffer, then atomically swaps the active index. From this point, all threads see the new config. The old buffer is available for the next reload cycle.
The reload window is typically under 15 microseconds. Readers never block on writers in the steady state. This makes it practical to change fault rates between test methods without restarting the process.
CI Integration: Three Lines of YAML
Integrating into a CI pipeline is intentionally minimal. Pre-compiled binaries for four platform combinations (glibc/musl × amd64/arm64) are available. No build step needed — download the right binary for your CI runner OS and architecture, write a config file, set LD_PRELOAD.
# GitHub Actions example: inject network chaos for integration tests
- name: Download libchaos
run: |
curl -fsSL https://github.com/macstab/chaos-testing-libraries/releases/latest/\
download/libchaos-net-glibc-amd64.so -o /tmp/libchaos-net.so
- name: Write chaos config
run: |
cat > /tmp/.chaos-net.conf << 'EOF'
tcp4://db.internal:5432:connect:ECONNREFUSED:0.10
tcp4://cache.internal:6379:connect:ETIMEDOUT:0.05
dns://*.internal:EAI_AGAIN:0.15
EOF
- name: Run integration tests with chaos
env:
LD_PRELOAD: /tmp/libchaos-net.so
run: ./gradlew integrationTest
Thread-Local Internals: No Global State
Multi-threaded correctness is enforced through thread-local storage rather than locks wherever possible.
**PRNG per thread.** Each thread has its own random number generator state. Fault sequences are independent per thread — no global randomness bottleneck, no contention on shared state. This also means fault sequences are deterministic per thread given the same seed, which matters for reproducing test failures.
**Recursion guard.** When the chaos code calls libc internals (stat to check mtime, read to read the config file), those calls would themselves be intercepted, causing infinite recursion. A `__thread int guard` variable prevents this: if the recursion depth is nonzero, we call through to the real function immediately.
**FD-to-path cache.** The file I/O interceptor needs to match a file descriptor to a path (for path-prefix rule matching). Reading `/proc/self/fd/<fd>` on every I/O call is too expensive. We maintain a 32-slot direct-mapped thread-local cache (fd % 32) seeded on `open()` and invalidated on `close()`. Cache miss rate for real workloads is under 3%.
What Can and Cannot Be Intercepted
We document the limits explicitly rather than letting users discover them.
**Cannot intercept:**
- **Direct syscalls.** The Go runtime issues syscalls directly via the `syscall` package, bypassing libc entirely. Go programs compiled without CGO are immune to LD_PRELOAD chaos on their network I/O. (Java via JNI is fine; JNI goes through glibc.)
- **Static binaries.** LD_PRELOAD is a dynamic linker mechanism. Static binaries have no dynamic linker; preloading has no effect.
- **Setuid/setgid binaries.** The kernel strips LD_PRELOAD for setuid binaries (AT_SECURE semantics). This is a security feature that we correctly rely on.
- **vDSO fast-path clocks.** Some clock_gettime calls go through the vDSO, not glibc. Our time library intercepts at the glibc wrapper level, which sits downstream of the vDSO. In practice, this means the fault fires on the glibc call before the vDSO lookup — the net effect is the same, but the kernel's internal clock is not affected.
**Can intercept:**
- Any dynamically-linked process on Linux (glibc or musl)
- Java via JNI — including all socket, file, DNS, and time calls in the JDK
- Python, Node.js, Ruby, PHP — all use glibc for system I/O
- Rust binaries that use std (which uses glibc) or CGO-bridged C libraries
- Java (JNI path): full coverage — network, DNS, file, clock, memory
- Python / Node.js / Ruby: full coverage via glibc
- Go (CGO=1 or C libraries): partial coverage — CGO calls intercepted, pure Go syscalls bypass
- Go (pure, CGO=0): no coverage — direct syscall ABI, LD_PRELOAD has no effect
- Static binaries: no coverage by definition
Performance: The Overhead Numbers
We measure passthrough overhead with rdtsc-serialized microbenchmarks to get worst-case numbers. The hot path for a passing call (no fault configured):
- **Config cache hit (no reload needed):** ~20–40 ns overhead beyond the actual syscall cost. This is a mtime stat check (cached in an atomic slot), a rule scan (linear over N rules, typically N < 16), and a call through the cached function pointer.
- **Cache miss (config file reloaded):** One stat() syscall plus a read() of the config file — typically 150–300 ns total, and only on the first call after a config change.
- **Fault injected:** The fault overhead itself (sleeping for LATENCY, setting errno for ERRNO) dominates; the wrapper cost is negligible.
For a well-configured test suite with a small number of rules, the passthrough overhead is below the noise floor of a network call. You're paying tens of nanoseconds on top of operations that cost microseconds to milliseconds.
tags
#chaos-engineering
#c99
#ldpreload
#elf
#libc
#fault-injection
#ci-cd
#linux
#resilience
conclusion
The C99 LD_PRELOAD chaos library is what kernel-real fault injection looks like when it's designed from first principles rather than bolted together from existing tools.
Six independent fault domains. Disjoint symbol ownership so they compose safely. A plain-text config interface that works from any language or toolchain. Lock-free hot reload so fault rates can change between test methods. Thread-local PRNG so fault sequences are independent and deterministic. Pre-compiled binaries for glibc and musl on amd64 and arm64.
And 100% line coverage enforced in CI, because we've shipped enough software on cartridges without patch buttons to know that every uncovered line is a production incident waiting to happen.
The source is available at chaos-testing-libraries. The Java testing framework that injects these libraries into your Docker containers automatically is at chaos-testing.
E
Engineering Team
Senior Solutions Architects
We've been building distributed systems since before 'microservices' was a thing. Our scars tell stories.