What Happens When a Service You Depend On Goes Down?

A service you depend on going down is not an edge case to handle someday, it is a certainty on a long enough timeline. The question is not whether to plan for it, it is whether your service degrades or falls over with it.

The problem

Order service calls inventory service to check stock. Inventory service starts timing out under its own load spike. What should order service do: wait? retry? give up immediately? The answer depends on the failure, and treating every failure the same way is how one struggling dependency takes down a service that has nothing wrong with it.

The naive solution

1@Retryable(maxAttempts = 10, backoff = @Backoff(delay = 100))
2public InventoryResponse checkInventory(String sku) {
3    return inventoryClient.getStock(sku);
4}

Ten retries at a 100ms fixed delay against a service that is already struggling does not help it recover, it adds ten times the load exactly when the dependency can least afford it. This is the retry storm pattern: every caller retries aggressively, the retries become the majority of the traffic, and the dependency never gets a chance to recover.

Why it breaks

• No timeout means "slow" and "down" look identical, and a slow dependency ties up resources indefinitely.
• Unbounded or aggressive retries turn a struggling dependency into a fully down one by amplifying load during the exact window it needs relief.
• No circuit breaker means every caller keeps trying a dependency that has already been failing for the last thirty seconds, instead of failing fast and checking back later.

The production solution

Three separate mechanisms, each solving a different failure mode, composed together with Resilience4j:

• Timeout: bound how long you wait for a single call. Turns "slow" into a definite failure you can act on.
• Retry: a small, bounded number of attempts with backoff, only for failures that are actually likely to be transient (timeouts, connection resets), never for 4xx responses.
• Circuit breaker: after enough failures, stop calling the dependency entirely for a cool-down window, so it gets room to recover and your own threads stop blocking on a call that keeps failing.

1resilience4j:
2  timelimiter:
3    instances:
4      inventory:
5        timeout-duration: 2s
6  retry:
7    instances:
8      inventory:
9        max-attempts: 3
10        wait-duration: 200ms
11        retry-exceptions:
12          - java.io.IOException
13          - java.util.concurrent.TimeoutException
14  circuitbreaker:
15    instances:
16      inventory:
17        sliding-window-size: 20
18        failure-rate-threshold: 50
19        wait-duration-in-open-state: 15s
20        permitted-number-of-calls-in-half-open-state: 5

1@CircuitBreaker(name = "inventory", fallbackMethod = "fallbackInventory")
2@Retry(name = "inventory")
3@TimeLimiter(name = "inventory")
4public CompletableFuture<InventoryResponse> checkInventory(String sku) {
5    return CompletableFuture.supplyAsync(() -> inventoryClient.getStock(sku));
6}
7
8private CompletableFuture<InventoryResponse> fallbackInventory(String sku, Throwable ex) {
9    log.warn("Inventory check failed for sku={}, assuming unavailable", sku, ex);
10    return CompletableFuture.completedFuture(InventoryResponse.unknown(sku));
11}

The fallback method matters as much as the primary one. Returning "assume unavailable" here is a deliberate business decision (better to show a product as possibly out of stock than to let the whole order page fail), and that decision belongs in code review, not buried in a generic exception handler.

Making it reusable

The resilience4j instance names (inventory here) are the reusable unit: every outbound call to a given dependency shares one named configuration, so tuning the circuit breaker for inventory service does not require touching every method that calls it, just the one YAML block.

Testing it

Resilience4j exposes its state machine for testing: force the circuit breaker into OPEN state in a test and assert that calls short-circuit to the fallback without hitting the network at all. That test catches the failure mode that matters most, a misconfigured breaker that never opens and lets a dead dependency keep dragging down every caller.

What I learned

Retries feel like resilience but are often the opposite: a retry without a circuit breaker just delays and amplifies the same failure. The circuit breaker is what actually protects the calling service; the retry only smooths over genuinely transient blips once the breaker confirms the dependency is still basically healthy.