| Joe | |
| Jan 20 2026, 6:30 AM |
| F72487319: Screenshot Capture - 2026-03-04 - 00-24-40.png | |
| Mar 3 2026, 11:33 PM |
| F72447656: Screenshot Capture - 2026-02-27 - 17-17-57.png | |
| Feb 27 2026, 4:21 PM |
| F72440794: Screenshot Capture - 2026-02-26 - 23-54-24.png | |
| Feb 26 2026, 10:58 PM |
| F72440779: Screenshot Capture - 2026-02-26 - 23-52-03.png | |
| Feb 26 2026, 10:58 PM |
After some user reports of being rate-limited as unauthenticated users while having logged in with bot passwords and action=login on the action API, I have verified myself that indeed in that case we generate a couple of wiki-session cookies, but no sessionJWT cookie.
This means that at the edge we're not able to verify sessions correctly.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T399291 Epic: API Rate Limiting Architecture | |||
| Open | None | T412585 Epic: Enforce API rate limits (WE5.1.3c) | |||
| Open | None | T417779 rest gateway: enforce rate limits (stage two) | |||
| Resolved | daniel | T417778 rest gateway: enforce rate limits (stage one) | |||
| Resolved | DAlangi_WMF | T415007 Login with `action=login` and bot password does not create a JWT session cookie | |||
| Resolved | • Tgr | T418999 Remove trailing slash in issuer for bot password JWT cookies |
Change #1244649 had a related patch set uploaded (by Gergő Tisza; author: Derick Alangi):
[mediawiki/core@wmf/1.46.0-wmf.16] Session: Emit JWT cookie in ImmutableSessionProviderWithCookie
Change #1244650 had a related patch set uploaded (by Gergő Tisza; author: Derick Alangi):
[mediawiki/core@wmf/1.46.0-wmf.17] Session: Emit JWT cookie in ImmutableSessionProviderWithCookie
Change #1244684 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):
[mediawiki/extensions/CentralAuth@master] tests: Fix missing JWT issuer for CentralAuthSessionProvider
Change #1244692 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):
[operations/mediawiki-config@master] CommonSettings: Set $wgJwtSessionCookieIssuer for bot passwords
Change #1244684 merged by jenkins-bot:
[mediawiki/extensions/CentralAuth@master] tests: Fix missing JWT issuer for CentralAuthSessionProvider
Change #1244649 merged by jenkins-bot:
[mediawiki/core@wmf/1.46.0-wmf.16] Session: Emit JWT cookie in ImmutableSessionProviderWithCookie
Change #1244650 merged by jenkins-bot:
[mediawiki/core@wmf/1.46.0-wmf.17] Session: Emit JWT cookie in ImmutableSessionProviderWithCookie
Mentioned in SAL (#wikimedia-operations) [2026-02-26T21:34:00Z] <catrope@deploy2002> Started scap sync-world: Backport for [[gerrit:1244649|Session: Emit JWT cookie in ImmutableSessionProviderWithCookie (T415007)]], [[gerrit:1244650|Session: Emit JWT cookie in ImmutableSessionProviderWithCookie (T415007)]]
Mentioned in SAL (#wikimedia-operations) [2026-02-26T21:35:55Z] <catrope@deploy2002> catrope, tgr: Backport for [[gerrit:1244649|Session: Emit JWT cookie in ImmutableSessionProviderWithCookie (T415007)]], [[gerrit:1244650|Session: Emit JWT cookie in ImmutableSessionProviderWithCookie (T415007)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
Mentioned in SAL (#wikimedia-operations) [2026-02-26T21:45:39Z] <catrope@deploy2002> Finished scap sync-world: Backport for [[gerrit:1244649|Session: Emit JWT cookie in ImmutableSessionProviderWithCookie (T415007)]], [[gerrit:1244650|Session: Emit JWT cookie in ImmutableSessionProviderWithCookie (T415007)]] (duration: 11m 39s)
Change #1244692 merged by jenkins-bot:
[operations/mediawiki-config@master] CommonSettings: Set $wgJwtSessionCookieIssuer for bot passwords
Change #1244647 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable JWT session cookie for bot passwords (all wikis)
Mentioned in SAL (#wikimedia-operations) [2026-02-26T22:08:07Z] <catrope@deploy2002> Started scap sync-world: Backport for [[gerrit:1244692|CommonSettings: Set $wgJwtSessionCookieIssuer for bot passwords (T415007)]], [[gerrit:1244647|Enable JWT session cookie for bot passwords (all wikis) (T415007)]]
Mentioned in SAL (#wikimedia-operations) [2026-02-26T22:09:58Z] <catrope@deploy2002> catrope, d3r1ck01: Backport for [[gerrit:1244692|CommonSettings: Set $wgJwtSessionCookieIssuer for bot passwords (T415007)]], [[gerrit:1244647|Enable JWT session cookie for bot passwords (all wikis) (T415007)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
Mentioned in SAL (#wikimedia-operations) [2026-02-26T22:19:56Z] <catrope@deploy2002> Finished scap sync-world: Backport for [[gerrit:1244692|CommonSettings: Set $wgJwtSessionCookieIssuer for bot passwords (T415007)]], [[gerrit:1244647|Enable JWT session cookie for bot passwords (all wikis) (T415007)]] (duration: 11m 48s)
Deployed to production.
Follow-up task (not a blocker): T418475: Session providers have no way to invalidate a session from provideSessionInfo()
As a nice side effect, this tanked the "Session store lookups with no user information" metric, ie. MultiBackendSessionStore now knows bot password sessions are always authenticated.
Session reads went up a bit, I think that's just the reads that used to be on the previous chart getting recategorized:
This might ave broken bot passwords; there are five million JWT validation failed: JWT error: wrong subject log entries since yesterday. No error reports though, so not sure what's going on. (But we will have to fix it for the log volume, if nothing else.)
Probably it's from this line. The central IDs I see in the logs are fine, though, so not sure what's going on.
Change #1245368 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):
[operations/mediawiki-config@master] Revert "Enable JWT session cookie for bot passwords (all wikis)"
Change #1245368 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "Enable JWT session cookie for bot passwords (all wikis)"
Mentioned in SAL (#wikimedia-operations) [2026-02-27T14:27:33Z] <tgr@deploy2002> Started scap sync-world: Backport for [[gerrit:1245368|Revert "Enable JWT session cookie for bot passwords (all wikis)" (T415007)]]
Mentioned in SAL (#wikimedia-operations) [2026-02-27T14:29:28Z] <tgr@deploy2002> tgr: Backport for [[gerrit:1245368|Revert "Enable JWT session cookie for bot passwords (all wikis)" (T415007)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
Mentioned in SAL (#wikimedia-operations) [2026-02-27T14:41:06Z] <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1245368|Revert "Enable JWT session cookie for bot passwords (all wikis)" (T415007)]] (duration: 13m 32s)
Mentioned in SAL (#wikimedia-operations) [2026-02-27T14:45:17Z] <tgr_> emergency deploy for T415007#11658252 done
The logs are not down. I tested and bot password JWTs are disabled, as expected. That would imply that the real issue is rMWa8dd114a4668: Session: Emit JWT cookie in ImmutableSessionProviderWithCookie causing some problem in CookieSessionProvider / CentralAuthSessionProvider.
Change #1245387 had a related patch set uploaded (by Gergő Tisza; author: Derick Alangi):
[mediawiki/extensions/CentralAuth@wmf/1.46.0-wmf.17] tests: Fix missing JWT issuer for CentralAuthSessionProvider
Change #1245387 merged by jenkins-bot:
[mediawiki/extensions/CentralAuth@wmf/1.46.0-wmf.17] tests: Fix missing JWT issuer for CentralAuthSessionProvider
Mentioned in SAL (#wikimedia-operations) [2026-02-27T15:31:54Z] <tgr@deploy2002> Started scap sync-world: Backport for [[gerrit:1245381|session: Log stack trace for JWT errors]], [[gerrit:1245387|tests: Fix missing JWT issuer for CentralAuthSessionProvider (T418487 T415007)]], [[gerrit:1245382|session: Log stack trace for JWT errors]]
Mentioned in SAL (#wikimedia-operations) [2026-02-27T15:33:49Z] <tgr@deploy2002> tgr: Backport for [[gerrit:1245381|session: Log stack trace for JWT errors]], [[gerrit:1245387|tests: Fix missing JWT issuer for CentralAuthSessionProvider (T418487 T415007)]], [[gerrit:1245382|session: Log stack trace for JWT errors]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
Mentioned in SAL (#wikimedia-operations) [2026-02-27T15:40:04Z] <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1245381|session: Log stack trace for JWT errors]], [[gerrit:1245387|tests: Fix missing JWT issuer for CentralAuthSessionProvider (T418487 T415007)]], [[gerrit:1245382|session: Log stack trace for JWT errors]] (duration: 08m 11s)
from /srv/mediawiki/php-1.46.0-wmf.17/includes/Session/SessionManager.php(407) #0 /srv/mediawiki/php-1.46.0-wmf.17/includes/Session/JwtSessionCookieHelper.php(158): MediaWiki\Session\SessionManager->validateJwtSubject(array, MediaWiki\User\User) #1 /srv/mediawiki/php-1.46.0-wmf.17/includes/Session/CookieSessionProvider.php(189): MediaWiki\Session\JwtSessionCookieHelper->verifyJwtCookie(MediaWiki\Request\WebRequest, MediaWiki\Session\SessionInfo, array, array) #2 /srv/mediawiki/php-1.46.0-wmf.17/extensions/CentralAuth/includes/session/CentralAuthSessionProvider.php(125): MediaWiki\Session\CookieSessionProvider->provideSessionInfo(MediaWiki\Request\WebRequest) #3 /srv/mediawiki/php-1.46.0-wmf.17/extensions/CentralAuth/includes/session/CentralAuthSessionProvider.php(218): CentralAuthSessionProvider->returnParentSessionInfo(MediaWiki\Request\WebRequest) #4 /srv/mediawiki/php-1.46.0-wmf.17/includes/Session/SessionManager.php(569): CentralAuthSessionProvider->provideSessionInfo(MediaWiki\Request\WebRequest) #5 /srv/mediawiki/php-1.46.0-wmf.17/includes/Session/SessionManager.php(137): MediaWiki\Session\SessionManager->getSessionInfoForRequest(MediaWiki\Request\WebRequest) #6 /srv/mediawiki/php-1.46.0-wmf.17/includes/Request/WebRequest.php(861): MediaWiki\Session\SessionManager->getSessionForRequest(MediaWiki\Request\WebRequest) #7 /srv/mediawiki/php-1.46.0-wmf.17/includes/Setup.php(504): MediaWiki\Request\WebRequest->getSession() #8 /srv/mediawiki/php-1.46.0-wmf.17/includes/WebStart.php(73): require_once(string) #9 /srv/mediawiki/php-1.46.0-wmf.17/api.php(23): require(string) #10 /srv/mediawiki/w/api.php(3): require(string) #11 {main}Change #1245398 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):
[mediawiki/core@master] Session: Validate JWT subject for authenticated users
So apparently this happens for anonymous sessions (in CookieSessionProvider) when there is a JWT cookie. AFAICS the code was essentially the same before rMWa8dd114a4668: Session: Emit JWT cookie in ImmutableSessionProviderWithCookie, but maybe there wasn't an easy way to end up in this situation, and bot passwords somehow changed that?
The important thing we are using anonymous sessions are for is login/signup. I can produce a login error ("There was an unexpected error logging in.") like this:
but it does not produce any session channel log entry... not sure what's going on there. Maybe logs are just getting cut off due to the high volume.
Login volumes seem fine, with no obvious change around the time the patches were deployed (22:10 UTC):
Looking up the affected users, they are all bots. So either this is related to the use of bot passwords, or maybe to the login API more generally. Maybe these are bots which log in with a bot password, but then for some reason lose the bot password cookie without losing the JWT cookie, and then during login get an anonymous session which doesn't work due to the inconsistent JWT cookie? There are about 100/s events, that's way too high for logins though. And some of the API requests clearly aren't logins (most are POST so no way to tell).
Bot password JWTs were disabled today at 14:40 UTC. So most likely the error will stop happening at 18:40 (JWT cookie expiry is 4 hours). Given that and that there's no obvious impact, it's probably best to wait it out. Rolling back the entire set of bot password JWT patches would be error-prone, and trying to write a fix at Friday evening, without really understanding the exact path these errors take, seems like a bad idea.
The log entries didn't fully stop around 18:40 but did drop by a magnitude (maybe more; logs before that were capped by the throttling Logstash applies to events with the same message and channel). The rest is presumably due to some bots (at a glance, maybe just one bot) not respecting cookie expiries.
One way to handle this is T418475: Session providers have no way to invalidate a session from provideSessionInfo() - instead of returning null from provideSessionInfo() which can result in all kinds of things depending on which other session handlers will get involved, just indicate the session is invalid, so cookies (including the JWT) get cleared.
Another is to not invalidate anonymous sessions when there's a conflicting JWT (instead, just refresh them so the JWT cookie gets cleared).
A third would be to create some new mechanism to do cookie adjustments at the end of session validation so the JWT cookie can be set/removed as needed, instead of having to do a full session refresh / invalidation. We might need that for T417833: Set a JWT cookie for OAuth 1 requests and OAuth 2 owner-only requests anyway.
Change #1245398 abandoned by D3r1ck01:
[mediawiki/core@master] Session: Validate JWT subject for authenticated users
Change #1247594 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):
[mediawiki/core@master] Do not invalidate anon sessions with non-anon JWT cookies
Change #1247596 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):
[operations/mediawiki-config@master] Enable JWT session cookie for bot passwords (all wikis) (attempt #2)
Change #1247594 merged by jenkins-bot:
[mediawiki/core@master] Do not invalidate anon sessions with non-anon JWT cookies
Change #1247689 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):
[mediawiki/core@wmf/1.46.0-wmf.17] Do not invalidate anon sessions with non-anon JWT cookies
Change #1247690 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):
[mediawiki/core@wmf/1.46.0-wmf.18] Do not invalidate anon sessions with non-anon JWT cookies
Change #1247689 merged by jenkins-bot:
[mediawiki/core@wmf/1.46.0-wmf.17] Do not invalidate anon sessions with non-anon JWT cookies
Change #1247690 merged by jenkins-bot:
[mediawiki/core@wmf/1.46.0-wmf.18] Do not invalidate anon sessions with non-anon JWT cookies
Change #1247596 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable JWT session cookie for bot passwords (all wikis) (attempt #2)
Mentioned in SAL (#wikimedia-operations) [2026-03-03T22:40:36Z] <tgr@deploy2002> Started scap sync-world: Backport for [[gerrit:1247689|Do not invalidate anon sessions with non-anon JWT cookies (T415007)]], [[gerrit:1247690|Do not invalidate anon sessions with non-anon JWT cookies (T415007)]], [[gerrit:1247596|Enable JWT session cookie for bot passwords (all wikis) (attempt #2) (T415007)]]
Mentioned in SAL (#wikimedia-operations) [2026-03-03T22:42:43Z] <tgr@deploy2002> tgr: Backport for [[gerrit:1247689|Do not invalidate anon sessions with non-anon JWT cookies (T415007)]], [[gerrit:1247690|Do not invalidate anon sessions with non-anon JWT cookies (T415007)]], [[gerrit:1247596|Enable JWT session cookie for bot passwords (all wikis) (attempt #2) (T415007)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
Mentioned in SAL (#wikimedia-operations) [2026-03-03T23:02:23Z] <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1247689|Do not invalidate anon sessions with non-anon JWT cookies (T415007)]], [[gerrit:1247690|Do not invalidate anon sessions with non-anon JWT cookies (T415007)]], [[gerrit:1247596|Enable JWT session cookie for bot passwords (all wikis) (attempt #2) (T415007)]] (duration: 21m 47s)
Deployed again. Resulted in a large but short spike of JWT validation failed: JWT error: wrong subject errors, not sure why. Seems all good now, though.
Other signals (session rates, auth rates, session Logstash channel) look unchanged, other than the same effect as in T415007#11656777. So far, so good.
Change #1247956 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):
[mediawiki/core@master] Session: Remove temporary config for JWT by bot passwords
Change #1247960 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):
[operations/mediawiki-config@master] Remove unused JWT for bot password temporary config
Change #1248000 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):
[operations/mediawiki-config@master] Revert "Enable JWT session cookie for bot passwords (all wikis) (attempt #2)"
Change #1248007 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):
[operations/mediawiki-config@master] Fix $wgJwtSessionCookieIssuer
Change #1248000 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "Enable JWT session cookie for bot passwords (all wikis) (attempt #2)"
Mentioned in SAL (#wikimedia-operations) [2026-03-04T14:19:04Z] <tgr@deploy2002> Started scap sync-world: Backport for [[gerrit:1248000|Revert "Enable JWT session cookie for bot passwords (all wikis) (attempt #2)" (T415007 T418999)]]
Mentioned in SAL (#wikimedia-operations) [2026-03-04T14:21:18Z] <tgr@deploy2002> tgr: Backport for [[gerrit:1248000|Revert "Enable JWT session cookie for bot passwords (all wikis) (attempt #2)" (T415007 T418999)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
Change #1248012 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):
[operations/mediawiki-config@master] Enable JWT session cookie for bot passwords (all wikis) (attempt #3)
Mentioned in SAL (#wikimedia-operations) [2026-03-04T14:26:24Z] <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1248000|Revert "Enable JWT session cookie for bot passwords (all wikis) (attempt #2)" (T415007 T418999)]] (duration: 07m 19s)
Change #1248007 merged by jenkins-bot:
[operations/mediawiki-config@master] Fix $wgJwtSessionCookieIssuer
Mentioned in SAL (#wikimedia-operations) [2026-03-04T20:57:33Z] <tgr@deploy2002> Started scap sync-world: Backport for [[gerrit:1248007|Fix $wgJwtSessionCookieIssuer (T415007 T418999)]]
Mentioned in SAL (#wikimedia-operations) [2026-03-04T20:59:38Z] <tgr@deploy2002> tgr: Backport for [[gerrit:1248007|Fix $wgJwtSessionCookieIssuer (T415007 T418999)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
Mentioned in SAL (#wikimedia-operations) [2026-03-04T21:07:28Z] <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1248007|Fix $wgJwtSessionCookieIssuer (T415007 T418999)]] (duration: 09m 55s)
Change #1248012 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable JWT session cookie for bot passwords (all wikis) (attempt #3)
Mentioned in SAL (#wikimedia-operations) [2026-03-04T21:40:52Z] <tgr@deploy2002> Started scap sync-world: Backport for [[gerrit:1248012|Enable JWT session cookie for bot passwords (all wikis) (attempt #3) (T415007 T418999)]]
Mentioned in SAL (#wikimedia-operations) [2026-03-04T21:43:00Z] <tgr@deploy2002> tgr: Backport for [[gerrit:1248012|Enable JWT session cookie for bot passwords (all wikis) (attempt #3) (T415007 T418999)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
Mentioned in SAL (#wikimedia-operations) [2026-03-04T21:47:56Z] <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1248012|Enable JWT session cookie for bot passwords (all wikis) (attempt #3) (T415007 T418999)]] (duration: 07m 05s)
Undeployed temporarily as change management for T418999: Remove trailing slash in issuer for bot password JWT cookies. No more wrong issuer errors since then, although their cadence was very random in the first place.
Checked various things: authentication metrics and session metrics on Grafana, session writes, the session channel and the authevent channel with loginType:BotPassword in Logstash. All seem normal (although some of them have infrequent events so we need more time to be sure).
Confirmed increase in approved-bot rate limiter hits at the time the last fix was deployed. Seems to be working.
Change #1258714 had a related patch set uploaded (by Fabfur; author: Giuseppe Lavagetto):
[operations/puppet@production] cache::haproxy: remove hotfix for traffic class
Change #1258714 merged by Giuseppe Lavagetto:
[operations/puppet@production] cache::haproxy: remove hotfix for traffic class
Change #1247956 merged by jenkins-bot:
[mediawiki/core@master] Session: Remove temporary config for JWT by bot passwords