Vasyl’s Dev Notes

TL;DR: Every network call will fail sometimes. Timeouts stop you from waiting forever. Retries fix short glitches. A circuit breaker stops you from hammering a service that is down. And idempotency makes retries safe, so a retried payment doesn’t charge the customer twice. In .NET 8+, Microsoft.Extensions.Http.Resilience gives you the first three in a few lines. The fourth one is your job.

Why this post

“The payment provider was slow for 5 minutes, and our whole API went down.” This is a very common story. One slow dependency, and your app falls over with it.

So here is a short refresh, built around one small refactoring.

Before: a naive HTTP call

builder.Services.AddHttpClient<PaymentClient>(c =>
    c.BaseAddress = new Uri("https://payments.example.com"));

public class PaymentClient(HttpClient http)
{
    public async Task<PaymentResult> ChargeAsync(ChargeRequest req)
    {
        var response = await http.PostAsJsonAsync("/charges", req);
        response.EnsureSuccessStatusCode();
        return (await response.Content.ReadFromJsonAsync<PaymentResult>())!;
    }
}

What happens when the payment API has problems:

• It is slow. Default HttpClient.Timeout is 100 seconds. Every request to our API now waits up to 100 seconds. Threads, connections and memory pile up. Our API becomes slow too.
• It blips. One 503 for 200 ms, and the customer sees an error, even though a second try would work.
• It is down. We keep sending requests, each one waits and fails. We waste our resources and make their recovery harder.

Step 1: timeouts

First rule: never call the network without a timeout. A fast failure is better than a slow one. A slow failure holds resources and spreads the problem.

Choose the timeout from what the user can accept, not from “what if it is slow”. If the page must answer in 3 seconds, the payment call can’t have 100.

Step 2: retries, with backoff and jitter

Many failures are short: a restart, a network blip, a 429 “too many requests”. Trying again often works. But retries done badly make things worse:

• Retry immediately → you hit the service in the same bad second.
• All clients retry at the same moment → a retry storm, the service gets 3x the traffic exactly when it is weak.

So: exponential backoff (wait 200 ms, 400 ms, 800 ms) plus jitter (a bit of random time, so clients don’t sync), and a small number of attempts.

Only retry what is transient: timeouts, 408, 429, 5xx. Never retry 400 or 401. It will fail the same way.

Step 3: circuit breaker

If the service is really down, retries just add load. A circuit breaker watches the failures:

• Closed: all good, calls go through.
• Open: too many failures (say, 50% in 30 seconds). For the next 15 seconds, calls fail immediately, without touching the network.
• Half-open: after the break, let a few calls through to test. Success → closed. Failure → open again.

You fail fast, save your resources, and give the other service time to recover. And you can show a nice fallback (“payments are temporarily unavailable”) instead of a timeout.

Putting it together in .NET

Install Microsoft.Extensions.Http.Resilience. The easy way is one line:

builder.Services.AddHttpClient<PaymentClient>(...)
    .AddStandardResilienceHandler();

This adds a total timeout, retries with backoff and jitter, a circuit breaker and a per-attempt timeout, with reasonable defaults.

If you want control, build the pipeline yourself:

builder.Services.AddHttpClient<PaymentClient>(...)
    .AddResilienceHandler("payments", pipeline =>
    {
        pipeline.AddTimeout(TimeSpan.FromSeconds(10));        // total, all attempts

        pipeline.AddRetry(new HttpRetryStrategyOptions
        {
            MaxRetryAttempts = 3,
            BackoffType = DelayBackoffType.Exponential,
            UseJitter = true,
            Delay = TimeSpan.FromMilliseconds(200)
        });

        pipeline.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
        {
            FailureRatio = 0.5,
            SamplingDuration = TimeSpan.FromSeconds(30),
            MinimumThroughput = 10,
            BreakDuration = TimeSpan.FromSeconds(15)
        });

        pipeline.AddTimeout(TimeSpan.FromSeconds(2));         // per attempt
    });

Order matters. The first strategy wraps everything after it. Here the total timeout covers all retries, and each attempt has its own 2-second timeout. If you put the retry outside the total timeout, a “10 second limit” can turn into 40 seconds.

Step 4: idempotency, the part libraries can’t do for you

Now the dangerous case. We send “charge $100”. The payment API charges the card, but the response is lost (timeout). Our retry sends “charge $100” again. The customer pays $200.

A retry is only safe if the operation is idempotent: doing it twice has the same effect as doing it once.

• GET, PUT, DELETE are idempotent by design.
• POST usually is not.

The standard fix is an idempotency key. The client creates one ID per logical operation, and sends the same ID on every retry:

var request = new HttpRequestMessage(HttpMethod.Post, "/charges")
{
    Content = JsonContent.Create(req)
};
request.Headers.Add("Idempotency-Key", req.PaymentId.ToString());   // same on retry

The server remembers keys it has seen. With a unique index, the database does the hard work:

// Payments table has a UNIQUE index on IdempotencyKey
try
{
    db.Payments.Add(new Payment { IdempotencyKey = key, Amount = req.Amount });
    await db.SaveChangesAsync();
    // ... charge the card, save the result
}
catch (DbUpdateException ex) when (IsUniqueViolation(ex))
{
    var existing = await db.Payments.SingleAsync(p => p.IdempotencyKey == key);
    return Results.Ok(existing);   // second call returns the first result
}

Second request with the same key? No new charge, just the saved result. Stripe and most payment APIs work exactly like this.

Until you have idempotency, don’t retry unsafe methods. Newer versions of the resilience package can disable retries for them (DisableForUnsafeHttpMethods() on the retry options).

Quick checklist

• Every outgoing call has a timeout, and it is much less than 100 seconds.
• Retries: few attempts, exponential backoff, jitter, only transient errors.
• A circuit breaker on each important dependency, with a fallback.
• Retried POST = idempotency key, enforced by a unique constraint.
• Metrics on retries and breaker state. Many retries = a hidden problem.

The 30-second summary

Every remote call can fail, so you need four things. Timeouts, so a slow dependency doesn’t make you slow. Retries with exponential backoff and jitter, only for transient errors, with a small number of attempts to avoid retry storms. A circuit breaker, which stops calling a failing service for a while, so you fail fast and let it recover. And idempotency, so retries are safe: the client sends an idempotency key, and the server uses a unique constraint to make sure the operation runs only once. In .NET 8+, Microsoft.Extensions.Http.Resilience handles the first three, idempotency is on you.

That’s it. If you have a good retry storm story, tell me in comments. 🙂