Vasyl’s Dev Notes

My admin login page allows 5 wrong passwords per minute. After that, it should block you.

I tried 9. It never blocked me.

I checked the other rate limits in my app, and they did not work either. While I was fixing that, I found a second problem: my app saw every user in production as the same IP address. Nobody noticed, because the limiter was off. If I had only fixed the first bug, the limiter would have started to work with one counter for the whole internet.

Why the limiter did nothing

In ASP.NET Core, every request goes through a chain of middleware, one step after another, and the order matters.

One step is routing. It decides “this request is for the login endpoint”. The rate limiter needs this answer, because every endpoint has its own limit.

In my app, the rate limiter was placed before routing. So it asked “which endpoint is this?” too early, got no answer, and let the request go. There was no error and nothing in the logs.

The fix was to move one line:

app.UseRouting();
app.UseRateLimiter(); // must come after routing

(If you never call UseRouting() yourself, ASP.NET adds it at the start for you, so you can’t get this bug. I call it myself, later in the chain, and the limiter ended up above it.)

After this fix the limits started to work, and three more problems showed up. They were there all the time, but they could not hurt anyone while the limiter did nothing.

Every user had the same IP

My limits count requests per IP address. But my app sits behind Cloudflare and nginx, so the app never talks to the user directly. The user’s IP comes in a header, X-Forwarded-For.

nginx builds this header like this:

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

$proxy_add_x_forwarded_for takes the header it received and adds the address of whoever connected to nginx. In my setup that is cloudflared, running on the same machine. So every request arrived like this (the user IP here is an example):

X-Forwarded-For: 203.0.113.42, 127.0.0.1
                 real user      cloudflared, added by nginx

before the fix, my app used:  127.0.0.1
after the fix, my app uses:   203.0.113.42

By default, ASP.NET’s forwarded headers middleware reads only one entry, the last one. And the last one was always 127.0.0.1. So my “3 guest sign-ups per IP” limit would have been 3 guest sign-ups for everybody.

There was a second problem in the same place. My config also trusted this header from anyone. In production the first problem hid it, but without that extra nginx hop, a client could write any IP in the header and get a fresh counter. I checked this in my test setup, and it worked.

This is the fix for both, a bit shortened:

builder.Services.Configure<ForwardedHeadersOptions>(o =>
{
    o.ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
    o.ForwardLimit = null; // read the whole list, not only the last entry

    // trust only my own hops: loopback and private networks
    o.KnownIPNetworks.Clear();
    o.KnownIPNetworks.Add(System.Net.IPNetwork.Parse("127.0.0.0/8"));
    o.KnownIPNetworks.Add(System.Net.IPNetwork.Parse("::1/128"));
    o.KnownIPNetworks.Add(System.Net.IPNetwork.Parse("10.0.0.0/8"));
    o.KnownIPNetworks.Add(System.Net.IPNetwork.Parse("172.16.0.0/12"));
    o.KnownIPNetworks.Add(System.Net.IPNetwork.Parse("192.168.0.0/16"));
});

Now the middleware reads the list from the end, skips the addresses it trusts (my own proxies), and stops at the first one it doesn’t trust. That is the real user. If a client puts a fake IP into the header, it ends up further left, and the middleware never reaches it.

Ten wrong passwords would lock out everyone

My login limit was not per IP at all. It was one counter for the whole site, 10 tries per minute.

While the limiter did nothing, it did not matter. With the limiter on, anyone could send ten wrong passwords and close login for all users, and password reset too, because it used the same limit. I changed it to one counter per IP. An attacker still gets 10 tries, and everyone else can still log in.

My tests were green because of the bug

When the limits started to work, my integration tests started to fail. They send a lot of requests quickly, and they had passed only because nothing was ever blocked. I made the limits configurable, so CI can use higher numbers.

I also added two tests that check the limiter itself, not the endpoints:

• send one request more than the limit allows, and expect 429 Too Many Requests
• send requests as two different users through the same proxy, and expect two separate counters

Before this, I had never seen my limiter return 429. I think that was the real warning, and I missed it.

Check your own app

Send more requests than your limit allows. Only to your own app, of course:

for i in $(seq 1 11); do
  curl -s -o /dev/null -w "%{http_code}\n" -X POST https://your-app/login
done

You should see 429 at the end. If you see only 200, your limit does not work. Then try again with a different fake X-Forwarded-For header each time. If the 429 goes away, anyone can bypass your limit.

Have you ever checked that your rate limiter really returns 429 in production? I am curious if I am the only one who didn’t.

---

I found this while building TextStack, a reader for technical books that explains hard terms in the book’s context. The Android app is in closed testing, and I need a few more testers, so if you read on your phone, you can join the beta here.