[19:21:59] Hi tech team, I'm on the ACC team at ENWP. I'm seeing a large number of requests in the last hour saying the captcha is not working on account creation, any ideas? [19:42:08] Reedy: ^ per SAL [20:51:15] taavi: A change to a script that is run once a week on a cron job (on a monday)... [20:52:17] Reedy: what info are looking for for T359032? We don't have timestamps of when they would have attempted. We know when they made the request to us and their IP + UA. [20:52:18] T359032: mediawiki_job_generatecaptcha maintenance job fails to run - https://phabricator.wikimedia.org/T359032 [20:53:55] While there is a possibility the script is creating bad captchas (since various recent enough changes), I don't think that is probably the right bug to be reporting these errors on [20:54:35] That's fine - we can use a different or new one [20:55:19] I just don't know what you need from us to be able to help [20:55:19] I similarly can't seemingly replicate the "missing captcha" [20:56:17] Some idea when it potentially started, or at least, it was noticed there was an increase in people struggling [20:56:29] https://www.mediawiki.org/wiki/MediaWiki_1.42/wmf.23/Changelog#ConfirmEdit [20:56:50] there's basically no changes to the extension in this weeks train, so that doesn't seem to be at fault... but that also doesn't mean it wasn't mediawiki core [20:57:13] It also also doesn't mean it's not related to the datacentre switch this week too [20:57:21] So hence, any indication on timing is useful [20:57:49] https://grafana.wikimedia.org/d/000000370/captcha-failure-rates?orgId=1 [20:58:07] That seems to show a large increase in failed captchas as of ~3 hours ago [20:58:25] in fact, to the extent of it showing 100% captcha failure rate [20:59:05] that time lines up with what I'm seeing [20:59:09] To the extent, it looks like when the train ran [20:59:15] >18:16 dancy@deploy1002: rebuilt and synchronized wikiversions files: group2 wikis to 1.42.0-wmf.23 refs T354441 [20:59:15] T354441: 1.42.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T354441 [20:59:52] do you want be to file a blocker asking for a revert? [21:02:46] JJMC89: Rollback is gonna happen, but yeah, a task to associate teh breakage with would be good [21:04:17] T360717 [21:04:17] T360717: CAPTCHA failure rate at 100% - https://phabricator.wikimedia.org/T360717 [22:15:40] RichSmith: JJMC89: Train rolled back, so based on the logs, it'll be fixed for those wikis that are now back on .22 [22:15:45] But likely still broken everywhere else [22:17:28] Reedy: Appreciate the attention :)