The clean result from our first circuit-breaker sweep was that time-based sliding windows produced zero fault propagation. Not low. Zero, across 243 runs, with count-based windows sitting around 19–20% on the same faults.
It is a good headline and it survived about a week.
Two problems, neither of which crashed anything
The first: window size was swept over the values 5, 10 and 20. Resilience4j reads that as a number of calls for a count-based window and a number of seconds for a time-based one. Our load generator issued roughly fifty requests over about two and a half seconds.
A five-second window cannot fill inside a two-and-a-half-second test. The time-based breakers were not outperforming the count-based ones — they were never evaluated at all. The zero was a description of the test envelope, not of the mechanism.
The second: we were measuring the wrong event. The data dictionary defined blast
radius as the share of services that had breached their error-rate SLO. The code
polled /actuator/health and counted services with an OPEN breaker. Those are
close enough to look like the same thing and are not: an open breaker is the
circuit breaker doing its job. Under the metric we had written down, the result
could plausibly have pointed the other way.
Why the pipeline never complained
Every run completed. Every row was well-formed. The summary statistics had the
right number of decimal places. Downstream, the ML pipeline ingested the dataset
without error and produced a decision tree — which turned out to have learned
nothing except window_type, because the label was perfectly collinear with it:
every unsafe run was count-based, without exception.
That collinearity was the tell. A tree that splits once on your treatment variable and stops is not modelling your system, it is telling you your label is your treatment. Later a validation gate reported PASS on the same data, not because the label had become independent of window type but because it had become nearly constant. The gate’s statistic was low for the opposite of the intended reason.
None of this raised an exception anywhere.
What we changed
The load plan is now computed from the configuration under test, so a time-based
window is guaranteed enough wall-clock time to fill, trip and reach half-open.
minimumNumberOfCalls was exposed rather than left at a default of 100 that no
run ever reached.
The metric was rewritten to read the breaker’s own rejected-call and failed-outcome counters, and — the part I’d defend hardest — the raw per-leg failure rates are persisted rather than only the threshold verdict. That means the threshold can be re-swept from the stored CSV instead of re-running the experiment, which turned a blocking design decision into a reversible one. The old metric was kept alongside the new one, because being able to show where two definitions disagree is worth more than quietly replacing one.
The thing I’d tell my earlier self
Ask what the most suspicious number in your results is, and go and try to explain it as an artifact before you try to explain it as a finding. A perfect zero across 243 runs is not a strong effect. In an empirical system with injected faults, network jitter and real timers, a perfect zero almost always means the condition you meant to test never occurred.