Skip to main content

The bug is old. The way you drive the model through it is the new part.

A mystery bounce on an OTP login, chased end to end: probing the error, building a mock to recreate it, red-bar tests, a two-layer fix, adversarial critic passes, and the CI gate. The three-line fix is the least of it. The discipline around the model is what actually solved it.

Ben Schmidt, PhD · ·11 min read

I logged into my own app on staging last week and it refused to let me in. I typed the one-time code. A second later my inbox got the “you recently logged in” email, so somewhere a server was sure I had made it. And then the page dropped me right back on the login screen, as if I had never started.

So how does a server end up certain I am logged in and certain I am not, in the same half-second? That is the question I want to chase, because chasing it is far more interesting than the three lines that eventually fixed it. This is a textbook bug. What is not textbook is what solving a textbook bug looks like now, when a model writes most of the code and the work that is left over is making sure it wrote the right thing.

The first clue: read the redirect

#

The symptom is a redirect, so the first move is to find who issued it. The login-by-code flow keeps a small server-side stash that means “this person is mid-login.” Submitting the code consumes that stash and rotates the session. The framework guards the confirm step with a decorator that, stripped down, does this:

the framework's login gate (paraphrased)
login_stage = LoginStageController.enter(request, LoginByCodeStage.key)
if not login_stage:
    # no stash: assume this person wandered in mid-flow, send them back
    return HttpResponseRedirect(reverse("account_request_login_code"))

There is the redirect. It fires when the stash is empty. And the stash would be empty if something had already consumed it, which is exactly what a successful login does. The gate cannot tell “you are already done” apart from “you never started.” It reads a finished login as a fresh arrival and throws me back to the door I already walked through.

So now I know what ejects me. I do not know how the stash got emptied under a login that also succeeded, and I cannot make the bounce happen on demand. A clue is not a case.

The suspects: two ways to submit twice

#

The only way the stash gets consumed and then read as empty is if the confirm step ran twice. A double-click, a double-tap of Enter, a password manager that re-posts the form. Two requests, and they can race in two different shapes:

  • Both at once. The two requests arrive while the stash is still there. Both find it, both authenticate, both fire the login signal, and I get two “recently logged in” emails.
  • One just behind the other. The first request finishes, consumes the stash, and rotates the session. The second arrives a beat later, finds nothing, and hits that redirect. This is the bounce, and this is the one I hit on staging.

Two suspects. I have a theory for each. But a theory I cannot reproduce is a guess, and a guess is a thing a language model will happily write a confident, green test around. Before I let it near a fix, I need to make the crime happen on command.

Reproducing it

#

The obvious reproduction does not work. A double-click in a browser test does nothing, because the test runner collapses the second click into the page that is already navigating away. The two events never race. My first honest attempt fires two POSTs in parallel with raw fetch:

attempt two: parallel POSTs
await Promise.all([
  fetch(confirmUrl, {method: 'POST', body: form}),
  fetch(confirmUrl, {method: 'POST', body: form}),
]);

This reproduces the race. It also skips the browser’s own form entirely, which means it sails straight past the client-side guard I have not written yet but am about to. A reproduction that bypasses one of the two layers you are about to build cannot tell you whether that layer works. I did not know it yet, but this shortcut was going to come back and make me redo it.

What reproduces a real double-click is dumber: dispatch two submit events at the same form, back to back, the way a double-click or a re-posting password manager actually does.

attempt three: what a real double-click does
() => {
  const form = document.querySelector('form:has(#id_code)');
  form.dispatchEvent(new Event('submit', {bubbles: true, cancelable: true}));
  form.dispatchEvent(new Event('submit', {bubbles: true, cancelable: true}));
}

Run that against the unfixed code and the test goes red for the right reason:

the red bar
pytest tests/browser/login_code_double_submit.py -v# ...FAILED test_otp_double_submit_still_lands_logged_inE AssertionError: expected exactly one "recently logged in" email; saw 2:E ['You recently logged in.', 'You recently logged in.']

A quick aside, because it is the reason I do this in exactly this order. A model told “fix the double-submit bug” will hand you a fix and a passing test in one breath, and you will have no way to tell whether either one touches reality. Making it produce a failing test first, and making that test fail for the real reason, is how you keep a fast, confident writer honest. The red bar is the one checkpoint the model cannot fake its way past. That discipline deserves its own essay; here it is just the load-bearing habit.

So where are we. I can now trigger the bug on demand, and I have a test that fails when the bug is present. I have the what and the how. I still have no fix, and no proof yet that a fix would hold.

The first critic pass, and the test that lied

#

Before writing the fix I ran the diff past two reviewers at once: one for code, one adversarial for security, both told to assume the tests were lying. The security pass came back with a finding I would not have caught myself.

Claude Code / adversarial review
YouReview this reproduction and its assertion. Assume the test is lying to me. Where does a green bar not mean the bug is gone?
CONCERNCritic"exactly one email" is a proxy, not the fact. The email is downstream of the `user_logged_in` signal. A dedupe or template change in the mail layer greens this test while the signal still fires twice on a double-submit: two audit rows, two device-login records, a doubled login count. Assert on the signal, not on the side effect it happens to trigger.

That is the difference between a test that looks like proof and a test that is proof. Counting emails measures a symptom. The thing I actually care about is the login itself firing once, so the real test watches the signal:

assert the cause, not the symptom
def test_login_signal_fires_exactly_once_on_double_submit(self):
    with mock.patch.object(user_logged_in, "send", wraps=user_logged_in.send) as spy:
        client.post(CONFIRM_URL, {"code": code})
    assert spy.call_count == 1   # the login, not the email it happens to trigger

So where are we now. I have a reproduction, a red bar, and a proof that measures the right thing. What I am still missing is the fix, and I need two of them.

The fix, part one: make a repeat boring

#

The bug was the gate reading “already done” as “never started,” so the cleanest repair is to own that branch instead of letting the framework guess at it. A repeat submit does not mean the user got lost. It means the first one already did the job, so the endpoint should agree instead of erroring.

server: a duplicate submit is success, not a bounce
def dispatch(self, request, *args, **kwargs):
    if request.user.is_authenticated:
        # POST #2 arrived with the fresh post-login cookies: send them home
        return HttpResponseRedirect(settings.LOGIN_REDIRECT_URL)

    login_stage = LoginStageController.enter(request, LoginByCodeStage.key)
    if not login_stage:
        # stash already spent by a sibling submit: idempotent success, not the old bounce
        return HttpResponseRedirect(settings.LOGIN_REDIRECT_URL)

    # a genuine mid-login request: run the login form
    return FormView.dispatch(self, request, *args, **kwargs)

Three branches, and the interesting one is the third exception I kept: a stash that existed but expired mid-request still sends you back for a fresh code, because that genuinely is a restart. Writing these branches is easy. Telling a finished login apart from an expired one is where the judgment sits, and that is the part the model needs a human pointing at.

Seven unit tests pin the branches, including the signal-once assertion the critic asked for. They pass. The browser test is still red, though, and that is not a failure. A whole layer is still missing: the one that stops the second submit before it ever leaves the page.

The fix, part two: stop it at the source

#

Belt, meet suspenders. The server is now honest under a repeat, but a repeat still travels the network and still costs a round trip. The client should refuse to send it in the first place. A flag on the form does it: the first submit sets it, every later submit is dropped.

client: drop the second submit
otpForm() {
  return {
    submitting: false,
    handleSubmit() {
      if (this.submitting) return;   // second event: dropped
      this.submitting = true;
      this.$el.submit();             // native submit fires no submit-event, so no loop
    },
  };
}

Adding this layer is where the parallel-fetch reproduction came back to bite me. It posts around the form, so it would have stayed green whether or not this guard worked. To actually exercise the guard I had to throw that reproduction out and switch to the two-synchronous-submit version above. The test had to change to test the layer I was adding. That is not a detour; it is the point. A test that cannot see your fix is not testing your fix.

There are two more holes the second critic pass found here, and how I handled each is the honest bit. A double-click that lands before the page’s JavaScript boots would fire two submits before the flag exists, so the button also carries a plain HTML disabled attribute the browser respects with no code running at all. And a black-holed request would leave the button stuck forever, so the handler arms a fifteen-second timeout that releases it. Both folded in.

The third hole I did not fix, on purpose. A script firing parallel POSTs still beats a client-side guard, because a client-side guard is a courtesy, not a wall. That race belongs to the server tier, and closing it properly means a session-level lock the browser fix has no business faking. So I filed it as its own follow-up rather than pretend the client layer covered it. Knowing which layer a threat lives in is the difference between a fix and a fig leaf.

So where are we. Two layers, one at the form and one at the endpoint, each with a test that actually exercises it, and the one attacker neither layer catches is written down and filed against the tier that owns it. All that is left is to find out whether it holds.

Proof: run all of it

#

A fix is a claim, and a claim you have not run is a guess wearing a lab coat. So it runs, at every layer:

green, and not just once
pytest tests/browser/login_code_double_submit.py # 3 consecutive runs, greenpytest apps/sso_stytch/tests/ # 385 passedpytest apps/authentication/tests/ # unit layer, green

Then it ran on my machine, the same way I found it, by logging in and watching the bounce not happen. Then it went through CI, which had its own opinion that week: a wave of new dependency advisories was blocking every merge, and the lint image itself needed rebuilding before the checks would even run against the right environment. None of that is glamorous, and all of it is the last gate between “works on my machine” and “true for everyone.”

Green at the form, green at the server, green locally, green in CI. Case closed. Which leaves one question worth writing down.

What solved it

#

Not the three-line fix. Strip the model out of this story and the bug is the same, the two-layer repair is the same, the tests are the same. None of the craft here is new. Idempotent endpoints are older than I am.

The model wrote nearly all of that code, and it wrote it fast, and it wrote it plausibly, which is the trap. Plausible is what a language model is for, and plausible sits one careful step short of correct. Everything that carried weight was the frame that turns plausible into proven, and every piece of that frame is a discipline with a name:

  • Reproduce before you fix, and make the reproduction fail for the right reason, and make sure it exercises the layer you are testing.
  • Red bar first, because it is the one checkpoint the model cannot talk its way past.
  • Fix in layers, and know which layer owns which failure, and file the ones a layer cannot honestly hold.
  • Review adversarially, because a second reader assuming your tests lie is how you catch the test that measures a symptom.
  • Write it down, because this essay only exists because the whole chase was in a journal, and the journal is what lets a fast process stay a legible one.

Each of those is its own piece, and some of them will be. The one about why writing down the decision matters more once a machine is doing the typing already is.

The model is a very fast, very confident junior who has never once been wrong out loud. The test layers, the critics, and the journal are how you find out when it was. I still had to log into my own app to notice the bug at all. That part has not been automated either.

The AI & Learning Field Report

Field notes on AI and how people actually learn.

What is real, what is hype, and what the evidence actually says. The reports land here first; subscribers get them in their inbox. No spam, no funnel theater.