[19:13:31] a gadget of mine is running into `badtoken` errors – I could use some help from people who know more about how CSRF tokens work or how often they’re invalidated: T421161 [19:13:31] T421161: ACDC reports "Invalid CSRF token" - https://phabricator.wikimedia.org/T421161 [19:34:29] idk, but we seem to have a higher rate of badtoken erros than usual… click on "badtoken" here to filter: https://grafana.wikimedia.org/d/000000002/mediawiki-action-api-summary?orgId=1&from=now-7d&to=now&timezone=utc&var-Module=1&refresh=5m&viewPanel=panel-41 [19:35:17] hm, interesting [19:35:39] (and that’s a logarithmic y axis) [19:37:00] yeah, it's kind of hard to judge because of that [19:37:22] but i'm looking at the same data in linear scale now and it looks more obviously increasing: https://phabricator.wikimedia.org/F73572593 [19:37:44] anyway, it looks like the tokens live in the session store [19:38:09] that increase time looks suspiciously train-shaped but ACDC is a Commons gadget 🤔 [19:38:19] I suppose it could be DC switchover shaped? [19:38:56] i.e. https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260325T1500 [19:39:15] * lucaswerkmeister afk, will look more later [19:39:19] https://grafana.wikimedia.org/d/000001590/sessionstore?orgId=1&from=now-2d&to=now&timezone=utc&var-dc=000000026&var-site=eqiad&var-prometheus=k8s&var-container_name=$__all [19:39:30] the metrics are all weird due to the switchover indeed [19:40:19] but it may be the cause. looks like the total number of errors is higher, maybe [20:00:44] filed T421168 as a more general task [20:00:45] T421168: Increased rate of badtoken errors / session store issues due to datacenter switchover? - https://phabricator.wikimedia.org/T421168 [22:34:42] I helped fix a thing that was blocking T17583. Now I'm wondering if anyone wants to move T17583 forward. See the task for details. I think I will email wikitech-l too looking for a new champion. [22:34:43] T17583: Enable importing across all Wikimedia projects - https://phabricator.wikimedia.org/T17583