[00:53:09] !log tools.guc Add Majavah to https://toolsadmin.wikimedia.org/tools/id/guc [00:53:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.guc/SAL [09:51:22] !log tools.stewardbots Deploy d95dac4ea26 [09:51:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [10:55:21] public service announcement: the bulk of the WMCS team is traveling to a team offsite. This week we may have less than usual time on the keyboard [13:12:28] !log paws move paws to k8s 1.23 b1d0daeb20cfeb6981653d65e44cf6175eded539 T328489 [13:12:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [13:12:33] T328489: Upgrade paws to Kubernetes 1.23 in prod - https://phabricator.wikimedia.org/T328489 [16:17:14] !paws log update extensions domain to public-paws 32d8153012758952e878a199b40ae1f7d6934177 T332081 [16:17:14] T332081: Update PAWS Public Link button to point to new public-paws - https://phabricator.wikimedia.org/T332081 [22:30:39] tgr_ Reedy do you think https://phabricator.wikimedia.org/T321160 could be related to why some tools like Quarry and https://copypatrol.toolforge.org only log you in *sometimes*? It seems multiple tools are having this problem right now, but the task says the fixes were backported [22:30:52] (asking here because so many Cloud Services tools use OAuth) [22:32:07] oddly, some tools like https://global-search.toolforge.org/ seem to have no issues with logging in. All are using OAuth 1.0a, I believe [22:32:31] That bug only affected group0 wikis between last Tue and Thu and group1 wikis between Wed and Thu. Also, OAuth 2 only, and I think it was deterministic. [22:33:13] okay, I asked because these tools I mention have the same symptoms as the Wikipedia Library. You have to try to login X number of times before it actually works. And I see Quarry has the same issue too [22:34:18] Examples with timestamps etc. might help, although in general OAuth logging is in a sad state. But still, better than nothing. [22:34:30] Quarry actually errors out for me. In the case of CopyPatrol, I can confirm the user is always getting redirected to Meta to login, but OAuth apparently isn't sending them back to the provided callback URL [22:35:05] Could be a bug in the tool, in OAuth, in SessionManager, in whatever we infrastructure are using for OAuth nonces these days (memcached?)... [22:35:19] you should be able to repro now using Quarry or CopyPatrol, i.e. https://copypatrol.toolforge.org/oauth/ [22:35:36] it seems odd to me Quarry, CopyPatrol, and The Wikipedia Library all have the same symptoms [22:35:44] hints that this is probably a larger, shared issue [22:36:35] Could be a shared library. TWL devs are looking into mwoauth AIUI. [22:37:11] also pinging bd808. You merged my patch earlier for SlimApp. Now I'm noticing other tools that don't use SlimApp also have issues with logging out! https://global-search.toolforge.org/ being an example [22:37:28] "This request has already been completed and cannot be resubmitted" sounds like an application issue (after it receives the result of the OAuth handshake, it sends the user to log in again). [22:37:53] where do you see that? [22:40:27] the CopyPatrol code for this hasn't changed in many years, so I don't think it's an application issue. We did upgrade the version of SlimApp, though. SlimApp as well as the other tools I've mentioned appear to all use https://github.com/wikimedia/mediawiki-oauthclient-php [22:40:37] not sure about Quarry, though [22:40:45] that's Python so probably a different library [23:08:29] musikanimal: is there a task already? [23:09:10] other than T332349 [23:09:10] T332349: Wikipedia Library /oauth/callback/ Permission denied - https://phabricator.wikimedia.org/T332349 [23:09:32] for CopyPatrol there's https://phabricator.wikimedia.org/T332636. I'm still debugging to confirm if it's a broader issue [23:25:59] ha, I think https://phabricator.wikimedia.org/T332429 is exactly what's happening for CopyPatrol [23:27:02] the JSON response has the `mwoauthdatastore-bad-token` error, with the message "No token was found matching your request." [23:27:08] I'm filing a bug now [23:35:43] if we are getting empty JSON responses in Wikimedia production, we should figure out what's going on. I don't think that's an expected response format. [23:39:17] Hm, that's not true, we do use JSON with HTTP 200 for errors. [23:40:36] https://phabricator.wikimedia.org/T332650