CascadeShield — measuring what a circuit breaker actually prevents

Resilience4j lets you configure a circuit breaker with either a count-based or a time-based sliding window. Practitioners pick one mostly by habit. As far as we could find, nobody had studied what that choice does to failure propagation at the application layer — the closest work fixes the window type and varies everything else.

So that became the capstone: a microservice harness with injected faults, a parameter sweep across window type, window size, failure threshold and wait duration, and a measurement of how far a single downstream fault spreads.

This is joint work with Jay Joshi, who owns the services, the fault-injection harness, the sweep runner, the observability stack and the dashboard. I own the Resilience4j integration, the blast-radius measurement, the dataset and its schema, and the ML pipeline.

The result we did not get to keep

The first 486 runs said something clean: time-based windows produced 0.0% blast radius across every fault type, while count-based averaged around 19–20%. A usable headline.

A result that clean is a reason to be suspicious, not pleased. Two things were wrong with it, and neither one crashed anything.

The window never filled. Window size was swept over 5, 10 and 20 — which count-based reads as calls and time-based reads as seconds. The load generator issued around fifty requests over roughly two and a half seconds. A five-second time window cannot close inside a two-and-a-half-second test. The time-based breakers weren’t outperforming anything; they were never given a chance to trip. The 0.0% wasn’t a finding, it was the shape of the test envelope.

The metric measured the wrong event. The data dictionary defined blast radius as the share of services that breached their error-rate SLO. The implementation polled /actuator/health and counted services with an OPEN breaker. But an open breaker is the circuit breaker working — it is protection, not damage. Under the metric we said we were using, the headline finding could plausibly invert.

Both problems were invisible from the outside. Every run completed, every CSV row was well-formed, and the summary statistics looked like science.

Fixing the instrument before trusting the numbers

The load plan is now derived from the configuration under test rather than fixed, so a time-based window is guaranteed enough wall-clock time to fill, trip and recover. minimumNumberOfCalls was also exposed across all five services — it had been sitting at the Resilience4j default of 100, which no run ever reached.

Alongside the old metric, a second one now reads Resilience4j’s own not.permitted.calls and failed-outcome counters, excludes business 4xx responses, and persists the raw per-leg failure rates rather than just a threshold verdict. Keeping the raw rates means the threshold can be re-swept from the CSV without re-running anything, which turned a blocking decision into a non-blocking one.

The old metric was kept, not deleted. Being able to show where two definitions disagree is more useful than quietly replacing one with the other — and they do disagree. On one run both reported 0.2, which looked like agreement until I checked which node each was counting. One was counting a downstream service; the other was counting the gateway. The numbers matched by coincidence.

The confound underneath

Tracing why blast radius kept pinning to exactly one node turned up the real problem. The gateway’s own circuit breaker was configured from the same swept parameters as everything else, and it observes the summed latency of the whole chain against a two-second slow-call threshold. Under any downstream latency fault, the gateway crosses that threshold first, opens, and fast-fails locally — starving the entire chain of the traffic the interior breakers need in order to engage.

Those zeros are not missing data. The legs are present, so they were exercised — every call through them succeeded, because a slow-but-completed call counts as a success. The experiment had been measuring the edge breaker’s behaviour and calling it the system’s.

The fix is to take the gateway out of the sweep and give it a fixed permissive configuration, so it stops being an uncontrolled variable. What’s interesting is that the confound is arguably a better paper than the one we set out to write: a uniformly configured edge breaker trips first and suppresses downstream breaker engagement, collapsing observed blast radius to a single node. Same phenomenon, demonstrated deliberately instead of stumbled into.

The ML layer, and why it isn’t trained yet

The goal is a recommender: given a target detection latency, suggest a breaker configuration. Two things had to be sorted out first.

The labels were perfectly collinear with window type — every unsafe run was count-based — so a decision tree would have learned window_type → label and nothing else. And the data is configurations × replicates, with replicates sharing an experiment ID, so a random train/test split puts replicates of the same configuration on both sides. That’s fixed structurally with GroupKFold grouped by experiment ID; contaminated test configurations went from 17 to 0.

The salvage came from a different dependent variable. Detection latency — time-to-open — isn’t saturated the way blast radius is, and it shows a clean monotonic dose-response on both window size and threshold, with time-based climbing much more steeply. Better still, the negative control passes: wait duration should affect recovery and not detection, and it doesn’t move detection at all, while it drives recovery monotonically. Each swept parameter moves exactly the outcome theory predicts and not the other one. That double dissociation is the strongest evidence we have that the instrument is sound.

So the model is retargeted as a regressor on detection latency, and it will be trained once the gateway confound is removed and the sweep re-run. Not before. Training a model on a dataset you already know is compromised produces numbers, not results.

What this project actually taught me

That schema drift and measurement error in this kind of system are silent. The pipeline never crashed. It produced degenerate output on real data — a single-class target, a near-constant regression target, a validation gate reporting PASS because the label had become almost constant rather than because it had become independent. Every failure mode here arrived looking like a finished result.

Built with Spring Boot 3, Resilience4j, Toxiproxy, Gatling, Prometheus + Grafana, Docker Compose, Python, scikit-learn, pandas · Source