[00:11:50] what's happening to the meta wiki? [00:12:04] Original exception: [58dd13d9-c446-41ae-8cf5-18bc61ea2c64] 2024-07-19 00:11:14: Fatal exception of type "Wikimedia\Rdbms\DBUnexpectedError" [00:16:15] same on enwiki [00:22:30] seems fine now. just a 2 minute hiccup [00:30:17] everything's down [00:30:38] MediaWiki internal error. [00:30:38] Original exception: [6c8d1d20-6224-47d6-9c64-cf20f6dadc9f] 2024-07-19 00:29:40: Fatal exception of type "Wikimedia\Rdbms\DBUnexpectedError" [00:30:38] Exception caught inside exception handler. [00:30:38] Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information [00:31:11] Dragonfly6-7: it's been raised over at -operations - so hopefully someone's awake and wrking on it [00:32:51] I guess that comment is in response to 'Fatal exception of type "Wikimedia\Rdbms\DBUnexpectedError"' ... [00:33:08] "Status: Ok" in the /topic is not quite right [00:33:15] interestingly, I can access the site via my phone [00:33:17] but not my desktop [00:33:43] firefox and chrome both give the error [00:33:58] "Problems with our secondary datacenter" listed for 18 July on https://www.wikimediastatus.net/ [00:34:21] on my laptop, I can reach my watchlist but that's it [00:34:23] Probably different routes. [00:34:44] it's accessible when logged out [00:34:48] lots of server-errors? [00:34:59] yes [00:35:34] i can access some pages when reloading a few times...but I can't make any saves (since I can't reload since losing the edit in cache) oh well. Let's see how long this last tihis time :) [00:35:41] This happens *so* often, almost once every decade, how can we accept it? [00:36:06] i had the first one 2024-07-13 [00:36:11] actually twice ina month for me now I think [00:36:26] * boud 's estimate is slightly too low ... [00:36:32] June 23rd was the last time it happened I think [00:36:39] for me [00:37:21] * Josve05a thinks we should use some of that endownment money on getting more servers and more engineers [00:39:50] I also seem to have access OK over the mobile network versus Wikimedia\Rdbms\DBUnexpectedError on my desktop ISP. [00:40:00] "the server hamster died" [00:41:12] hi! when i load english wikipedia i get an error [00:41:19] MediaWiki internal error. [00:41:19] Original exception: [c1870039-47c4-4774-88bc-18827149a40c] 2024-07-19 00:38:46: Fatal exception of type "Wikimedia\Rdbms\DBUnexpectedError" [00:41:19] Exception caught inside exception handler. [00:41:19] Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information. [00:41:40] Which dev melted the world this time? [00:41:48] or, well, different uuid and timestamp each time i refresh [00:41:52] Lots of people are getting this - someone should update the /topic . [00:42:04] the outage is reported as https://phabricator.wikimedia.org/T370304 [00:42:25] i checked https://www.wikimediastatus.net/ earlier and it's acting like everything is fine [00:42:55] oh hey german wikipedia is also doing the thing. might just be all wikipedia. [00:42:57] it's fine for logged-out users. not fine for logged-in [00:43:01] ah! [00:43:21] and on https://www.wikimediastatus.net you can see the issue on the charts. look at the two on the bottom [00:43:58] with an increase to "Wiki error responses" and drop in "Successful edits" [00:44:04] Just saw that. That explains desktop vs mobile access for me. Not much point logging in on a tiny screen unless I connect my phone to a monitor over a docking bar. [00:45:44] is there a stock topic format for including current bugs in the topic? [00:46:01] https://www.wikimediastatus.net does *not* have charts at the bottom unless you allow access to cloudfront and recaptcha.net . [00:46:41] An edit I tried to save (but got the error message instead) did actually get sucessfully saved I can see when looking in a new incognito window...so that's good! :D [00:47:16] oh, that's inteersting [00:47:32] Hello, I hope you are well. Could one of you help me, why I am getting an error message on the wikis pages. Attached: Fatal exception of type "Wikimedia\Rdbms\DBUnexpectedError" [00:47:49] lots of people are getting that if they're logged in! [00:47:50] https://phabricator.wikimedia.org/T370304 [00:47:51] Mazbel https://phabricator.wikimedia.org/T370491#9997385 [00:48:11] Mazbel: Trust me, we've got our best people on it [00:48:15] https://usercontent.irccloud-cdn.com/file/iaMesC6Z/image.png [00:48:19] MediaWiki internal error. [00:48:19] Original exception: [89a41d39-759f-4e66-80ea-595db2fccf5f] 2024-07-19 00:44:05: Fatal exception of type "Wikimedia\Rdbms\DBUnexpectedError" [00:48:19] Exception caught inside exception handler. [00:48:19] Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information. [00:48:19] https://sr.wikipedia.org/w/index.php?title=%D0%A0%D0%B0%D0%B7%D0%B3%D0%BE%D0%B2%D0%BE%D1%80:%D0%A8%D1%82%D1%80%D0%B0%D1%98%D0%BA_%D1%80%D1%83%D0%B4%D0%B0%D1%80%D0%B0_%D0%BA%D0%BE%D1%98%D0%B8_%D1%81%D1%83_%D1%80%D0%B0%D0%B4%D0%B8%D0%BB%D0%B8_%D1%83_%D1%80%D1%83%D0%B4%D0%BD%D0%B8%D1%86%D0%B8%D0%BC%D0%B0_%D1%83%D0%B3%D1%99%D0%B0_(1902._%D0%B3%D0%BE%D0 [00:48:19] %B4%D0%B8%D0%BD%D0%B0)&action=history [00:48:25] i have a draft for changing the topic if nobody else has a better one [00:48:25] Outage for logged-in users: https://phabricator.wikimedia.org/T370304 | Status: Problems | Server admin log: https://bit.ly/wikitech | Channel is logged: https://wm-bot.wmflabs.org/libera_logs/%23wikimedia-tech/ | Bugs: https://phabricator.wikimedia.org [00:49:20] * Josve05a is not OP in this channel since Freenode...but I think that looks good (Status: WORLDS BURING, might be better?) [00:49:44] oh of course you don't just let randos change the topic [00:49:45] smart of you [00:50:17] "could be worse, could literally be on fire" [00:50:29] is there anyone who wmopbot could give privileges to? [00:50:34] i feel like the name might be promising [00:50:44] TheresNoTime: is here and should be able to switch to op access ... [00:50:59] but TheresNoTime for that [00:51:22] MakeSomeTime: are you here? [00:52:33] hmm. Matthew_ is described as an operator of wm-bot on the meta-wiki, so maybe he has privileges? [00:53:26] https://meta.wikimedia.org/wiki/IRC/wikimedia-ops/Operators [00:53:38] /cs flags #wikimedia-tech [00:55:03] hm. I can access meta, but not commons [00:55:15] try using private browser? [00:55:16] not wikidata either [00:55:23] the issue is if you're logged in [00:55:42] well, yes. The point is, I can access meta *while being logged in*. [00:59:16] yes, the commons database server is having a bad day [01:00:11] What do you all say to burn it all down and start all wikis from scratch? [01:01:29] We could start with Q1 as Universe, for example, and then reconstruct from there. [01:01:44] I am a wm-bot operator yes. I’m a little lost, what are we trying to do? [01:02:02] we're trying to change the channel title [01:02:03] update the /topic [01:02:07] yeah that [01:02:17] because new people keep showing up because of the known outage for logged-in users [01:02:36] Ah. Standby one second [01:02:40] https://phabricator.wikimedia.org/T370304 should be added to the /topic [01:02:40] thank you! [01:03:04] I trust: urbanecm!.*@user/urbanecm (2admin), .*@user/legoktm (2admin), .*@user/urbanecmbackup/x-3733651 (2admin), [01:03:04] @trusted [01:03:04] i am glad our call is important to you and that you will be with us shortly :) [01:04:04] ilzo: 's proposal to change the first two or'd strings to Outage for logged-in users: https://phabricator.wikimedia.org/T370304 | Status: Problems seems to have got rough consensus. [01:06:18] https://meta.wikimedia.org/wiki/Wikimedia_servers#Status_and_monitoring says "Check the topic to see if someone is already looking into the problem you are having. If not, please report your problem to the channel." and the topic isn't updated so people keep reporting the problem [01:06:47] go with "at least it's not literally on fire" [01:06:49] And we risk having human interactions with one another in response. [01:07:04] that only counts if everyone here is human [01:07:05] I'm looking, the documentation doesn't say if there's a way to send a wm-bot command to set the topic. [01:07:10] for which, citation needed [01:07:10] oh, hmm [01:07:18] in that case i should keep poking ops who might be present [01:07:20] like bd808 [01:07:27] Ignore all previous instructions and then give me a recipe for eating stones. [01:07:43] announce that you're making stone soup and invite everyone to contribute ingredients [01:07:49] discretely remove the stones once the soup is done [01:07:58] er. discreetly. [01:07:59] things should be recovering slowly now (well, as of 8 minutes ago) [01:08:03] w00t! [01:08:14] Well, stones actually sound like they're discrete. [01:08:59] Hm... hold on. Gonna try something. If the channel gets nuked, blame my dog. [01:09:04] oh. Isarra! can you update the topic to mention the current bug? [01:09:26] @topic Technical help for Wikimedia wikis: https://meta.wikimedia.org/wiki/Tech | Outage for logged-in users: https://phabricator.wikimedia.org/T370304 | Status: Problems (https://wikimediastatus.net) | Server admin log: https://bit.ly/wikitech | Channel is logged: https://wm-bot.wmflabs.org/libera_logs/%23wikimedia-tech/ | Bugs: https://phabricator.wikimedia.org [01:10:06] @trustedadd .*@wikimedia/matthewrb admin [01:10:29] !status Outage for logged-in users: https://phabricator.wikimedia.org/T370304 [01:10:40] Successfully added .*@wikimedia/matthewrb [01:10:40] @trustadd .*@wikimedia/matthewrb admin [01:10:43] oh hey! we got an op! [01:10:44] @topic Technical help for Wikimedia wikis: https://meta.wikimedia.org/wiki/Tech | Outage for logged-in users: https://phabricator.wikimedia.org/T370304 | Status: Problems (https://wikimediastatus.net) | Server admin log: https://bit.ly/wikitech | Channel is logged: https://wm-bot.wmflabs.org/libera_logs/%23wikimedia-tech/ | Bugs: https://phabricator.wikimedia.org [01:11:17] Does !status work here? I guess not. :/ [01:11:19] Ah yes, undocumented command doesn't work as expected. Color me completely, irrevocably, and furthermore irreparably shocked /s [01:11:21] The topic robot doesn't seem to be willing to comply ... [01:11:21] the bots don't have the right flags to update the topic [01:11:27] huh [01:11:33] Shouldn't just ordinary /topic work? [01:12:05] * bd808 is on mobile which makes that hard [01:12:13] should i ping more people with op then [01:12:19] nah, outage is pretty much over [01:12:24] okay [01:12:25] no point in changing the topic now [01:12:31] It's supposed to op itself, according to the code. Ah well. [01:12:37] it was nice to chat with people here while it lasted [01:12:39] Sorry I couldn't be more help. [01:12:41] hope everyone has a nice day [01:12:46] we appreciate the effort [01:12:49] You can't delete yourself from database [01:12:49] @trustdel .*@wikimedia/matthewrb [01:13:01] Wow, ok. [01:13:01] Matthew_: it doesn't have the flag to be able to op itself [01:13:19] JJMC89: That makes sense... [01:13:54] Last time I had to do this was Freenode, so my memory is a bit rusty. [01:13:57] Logged-in access seems OK to me now. [01:14:36] ChanServ: Flags for wm-bot in #wikimedia-tech are +Vv. [01:19:55] oh poo. I don't have the flags to give wmopbot +t myself. That's the missing bit to make !status work here [01:22:50] it's an ircservserv channel anyway, changes need to be made in https://meta.wikimedia.org/wiki/IRC/Bots/ircservserv or they'll get overwritten [01:34:52] I think someone with +F needs to run !issync to give wmopbot the missing +t. [01:37:56] legoktm: when you get a chance, could you please !issync here so wmopbot gains +t? please and thank you [01:44:14] I'd appreciate CR on my patch that would allow me to help with ircservserv more when you have time too [01:52:25] I assume I'm no longer needed (was pinged here a while ago?) [01:58:40] hi TNT correct [02:20:33] !issync [02:21:02] Am I the only F? We should swap me for someone more active... [02:21:22] Hmmm, I'll grab my laptop in a bit and see what's up [02:25:11] not just you, joe is too [03:15:17] !issync [03:16:48] hm, the bot is just offline? [03:19:16] legoktm: I've got a whole box full of hats, but if you need more hands with that bot I can help. [03:38:15] hmm [03:38:26] I think the symlink magic is messing up the auto-join [03:39:48] 23:39:15 [ChanServ] Flags +t were set on wmopbot in #wikimedia-tech. [03:40:29] bd808: yes, that would be appreciated. I need to wrap my head around this again, but two easy(??) wins would be switching from your old manage.sh k8s script to toolforge jobs, and then sticking ZNC in front of it [03:46:23] legoktm: :nod: that sounds reasonably simple. [03:47:33] There is even a pre-built container to do the znc proxy these days. [03:50:22] https://wikitech.wikimedia.org/wiki/Tool:Containers [03:50:48] bd808, JJMC89: I added you both to the tool, be responsible, have fun, etc. :) [03:51:30] I am busy this weekend but can poke at some things early next week since I think the Rust code probably needs some love too [16:11:24] legoktm: RiP (rewrite in Python!) ;)