[01:17:02] hm, trying to log into Quarry gives me a "Internal Server Error" [01:20:50] are other people able to log in? [01:20:56] T333043 [01:20:56] T333043: [bug] "Internal Server Error" when logging into Quarry - https://phabricator.wikimedia.org/T333043 [01:31:06] legoktm: left a comment [01:31:10] omg [01:31:19] I just kept clicking and hitting back and got in on the 6th try [01:31:41] ty :D [01:44:49] might be related to similar problems OAuthing in to TWL [07:18:44] I thought the OAuth multidc problems were fixed already [08:43:38] *T332650 is *probably the same bug. [18:52:21] taavi, they are not: https://logstash.wikimedia.org/goto/c8c557ced11daa1919864c6b2f91e69a [18:52:56] ah, just thought that one puppet patch was the fix but apparently not [18:53:32] I also wouldn't bet all my money that they are specifically multi-dc problems, although the timing is suggestive. All I'm sure of is that it's related to replication lag in the main stash. [18:57:37] I'm travelling and probably won't be able to look at it for a few days. I think the next steps would be: 1) validate the patch actually works (ie. that all OAuth handshake related requests go to the primary DC), 2) make the OAuth exception classes implement NormalizedException and plug as much detail into them as easily available (the consumer key, at the very least), then look for patterns, 3) add debug logging to OAuth usage of the main stash (the [18:57:37] lookup_token method IIRC) so we have a clear picture of what's happening. [18:57:55] (But someone who actually understands how replication for the main stash works might have better ideas.)