The First Automated Redis Sentinel Failover Test — And It's One Annotation

Your Redis is up. It answers PING. Its replicas are in sync, every dashboard is green — and it has refused every single write for the last forty minutes. One background snapshot failed, and that was enough. stop-writes-on-bgsave-error ships set to yes, so from that moment Redis turns away anything that modifies the data set. Reads still look perfect. The master answers PING, the only thing Sentinel's health check consults, so nothing escalates to ODOWN and no failover is ever called. And the code you wrote for the other failure, the one where a node genuinely dies, has never run at all. Rehearsing it honestly means one master, two replicas, three Sentinels, a correct quorum, converged replication and a kill at exactly the right instant. So the cache gets mocked, and the first time your client library observes a +switch-master is at 03:14 in production. macstab-chaos-redis, the third layer of the Macstab Chaos Stack, takes that excuse away. Below it sit six C99 LD_PRELOAD libraries and a JVM bytecode agent. This layer supplies what neither can: a production-shaped topology worth injecting faults into.

After the dependency, one class-level annotation is the whole setup. @RedisStandalone gives you a real Redis 7.4 on a random port. @RedisSentinel gives you one master, two replicas and three Sentinels with quorum 2, on their own private Docker network. No @Testcontainers, no static container field, no @DynamicPropertySource, no @AfterAll you meant to write and never did. Handles such as SentinelCluster arrive as test-method parameters. The hand-rolled Sentinel equivalent is about ninety lines, load-bearing in ways the code never states: one specific log line is the only proof a replica has finished syncing, Sentinel must be handed an IP rather than an alias, and 2 000 has to replace the 30 000 default or no assertion can ever finish. Delete any of them and the tests still compile. They just start lying. With the annotation, that knowledge lives in the framework, written once and covered by tests. Teardown runs through JUnit's CloseableResource store, with Testcontainers' Ryuk reaper as an independent second path.

                      // Gradle
testImplementation 'com.macstab.chaos:macstab-chaos-redis:1.0.0'

@RedisSentinel                       // 1 master + 2 replicas + 3 sentinels, quorum 2
class FailoverTest { }
                    

Sentinel tracks a master by address, so the startup order is not negotiable. Four phases run in sequence, each gated on a semantic signal instead of a sleep: the master on Ready to accept connections, the replicas on MASTER <-> REPLICA sync: Finished with success, the Sentinels on a successful SENTINEL master mymaster, then a fixed two-second stabilisation window. The replica signal matters most: an open port or a PING succeeds while a replica is still loading the RDB, and your first real command gets -LOADING. Each Sentinel is told only the master's internal Docker IP and discovers peers and replicas over __sentinel__:hello. Redis documents full discovery at within 10 seconds, so the two-second wait is a pragmatic floor, not proof the gossip has settled: assert that a failover completes, not on num-other-sentinels at t=0. down-after-milliseconds drops from the Redis default of 30 000 to 2 000, and failover-timeout from 180 000 to 5 000. The default is right in production, where a GC pause should not trigger a failover, but it makes the assertion unwritable within a CI budget. The price is a tight window: on a loaded runner, a test asserting that no failover happened can flake, so widen it for that class.

Quorum is the number of Sentinels that must independently reach SDOWN before the verdict becomes ODOWN, and ODOWN only authorises a failover attempt. Performing it requires a leader elected by a majority of all Sentinels, versioned by a unique configuration epoch. So quorum = 1 across five Sentinels produces fast ODOWN and nothing else if only two of them can see each other. The majority rule is also what stops a stranded minority from promoting a second master. The module derives quorum = (sentinels / 2) + 1, lining ODOWN up with the election majority. One sharp edge, stated before you hit it: today quorum(), version() and masterName() on @RedisSentinel do not reach the container configuration. Defaults behave correctly, but the config always monitors mymaster and the nodes run redis:7-alpine. Until that is wired through, pass mymaster to your client.

Sentinel is a discovery protocol, not a proxy. A Sentinel-aware client asks for the master, receives the internal address, say 172.18.0.2:6379, and connects to it directly. On a Linux host, or inside a container sharing the daemon's network, that routes. On macOS or Windows, Docker Desktop keeps the bridge inside a VM, and every Sentinel-aware client dies on that second hop, right after the handshake succeeded. The module has builders for the announce directives that would fix this, but the current extension path does not invoke them. So @RedisSentinel carries @DisabledOnNonLinuxHost, which checks for /.dockerenv before os.name, so a dev container on a Mac still runs. On a macOS laptop the Sentinel suite finishes in two seconds with four SKIPPED tests and the reason printed. A red build nobody can fix locally teaches a team to ignore red builds; a skip with a reason keeps the suite trustworthy.

triggerFailover() finds whichever container reports role:master right now, by asking Redis over INFO replication rather than Docker, and kills it with SIGKILL. SIGTERM would let Redis close its sockets cleanly and send the Sentinels down a different, usually faster detection path than a real node death. It then polls every 500 ms, clearing the role cache on each pass, and returns the Duration from the kill to the first sighting of a new master, or throws a FailoverException with the timeout, elapsed time and retry count. The clock is only half the story. Redis replicates asynchronously, so a promotion can discard the tail of the write stream. Write a monotonic sequence, trigger the failover, read it back from the new master and assert the loss you can tolerate. If the answer is none, WAIT, min-replicas-to-write and min-replicas-max-lag are testable the same way.

                      @RedisSentinel(replicas = 2, sentinels = 3)
class FailoverTest {

  @Test
  void promotesWithinSlo(SentinelCluster cluster) {
    Duration elapsed = cluster.getControl().triggerFailover(Duration.ofSeconds(30));

    assertThat(elapsed).isLessThan(Duration.ofSeconds(15));
    assertThat(cluster.getControl().getMaster().isRunning()).isTrue();
  }
}
                    

With a real topology underneath, the lower layers become instruments. enableNetworkChaos = true grants NET_ADMIN, never --privileged, for tc/netem latency, jitter, packet loss and iptables partitions. enableConnectionChaos = true loads libchaos-net before the container starts, for errno injection on connect, send, recv and other socket calls. They compose: a retry loop can shrug off 5 % packet loss and still fall over on one ECONNRESET from send(). Clock skew of two seconds or more pushes a Sentinel into TILT mode, where it keeps monitoring but stops acting, visible as sentinel_tilt in INFO. And the opening failure becomes reproducible: one libchaos-io rule returning EIO on write fails the snapshot, and the assertion is one line — your write path must raise a specific, actionable error instead of dropping writes. On top, macstab-chaos-testpacks-l3-redis bundles compound faults into named, method-level incidents: