[06:49:32] !log tools.masto-collab Updated from 0aff6ff to 1be3240 [06:49:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.masto-collab/SAL [08:13:48] !log tools.masto-collab Updated from 1be3240 to 8b2bf7c [08:13:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.masto-collab/SAL [14:35:02] hey, qq — is `Database error: Unable to connect to meta.web.db.svc.eqiad.wmflabs` currently expected? [14:35:18] no [14:35:24] that domain is extremely old [14:35:28] I doubt it works? [14:35:37] seeing at https://guc.toolforge.org/?by=date&user=TNTBot [14:35:56] TheresNoTime: isn't it meta_p [14:36:02] *and* at roughly the same time got a poke that https://copypatrol.toolforge.org/en/ is down [14:36:10] When did https://replag.toolforge.org/ break too [14:36:16] That's showing no data [14:36:35] RhinosF1: the _p suffix is not in dns names [14:36:36] arturo: (fwiw that guc tool was working about an hour ago) [14:36:57] and we just merged a change to the DNS config, so that is my #1 suspect [14:37:30] mmm yes [14:37:31] taavi: pretty sure all of wikireplicas is inaccessible [14:37:48] taavi: then that domain is surely expected if one exists for the meta_p database [14:38:08] yes, and? [14:38:33] arturo said they were surprised it even worked [14:39:06] then why are you pinging me about it [14:40:22] TheresNoTime: I guess it eventually should move to the wikimedia.cloud equivalent but it's broken anyway [14:42:05] taavi: worth updating the topic / cloud-announce, I'd imagine many tools are affected by this outage [14:45:08] Done my bare minimum reporting and created T336581, good luck/have fun! [14:45:09] T336581: Unable to connect to wiki replicas - https://phabricator.wikimedia.org/T336581 [14:45:23] issue identified, working on a fix [14:45:33] <3 [14:47:06] tl;dr: we relied on something that was not supposed to work in the first place, and when we fixed that first issue this broke [14:47:39] oh that *never* happens :D /s [14:48:12] especially on Friday nights :P [14:48:28] TheresNoTime, RhinosF1 fixed [14:48:31] aborrero@tools-sgebastion-11:~$ host meta.web.db.svc.eqiad.wmflabs [14:48:31] meta.web.db.svc.eqiad.wmflabs is an alias for s7.web.db.svc.wikimedia.cloud. [14:48:31] s7.web.db.svc.wikimedia.cloud has address 172.16.2.51 [14:49:01] Looks good [14:49:06] Thank you both! ^^ [14:49:10] If there's one thing I've learnt in my time working in tech, no matter where you are, there's always *something* broken, and there's always something else critical which depends on said broken behaviour :D [14:50:09] stw: :) [14:51:02] for anyone curious what broke it: https://gerrit.wikimedia.org/r/c/operations/puppet/+/919341 [14:55:15] now that the thing is fixed [14:55:43] please, folks, if you see something with `wmflabs` in the domain, right a bell, it is likely outdated somehow somewhere [14:55:52] ring a bell* [14:56:12] Half the account-creation-assistance project infra is still on wmflabs :( [14:56:30] I've been meaning to get around to updating it for a while now, but I keep getting sidetracked with other things [14:56:36] so half is in the future, half is outdated :-P [14:57:15] That one's potentially tricky, as I'll need to coordinate with someone to update OAuth redirect urls too [14:57:55] maybe the hackathon happening next week is the right time to do that kind of stuff [14:58:22] stw: you can create a new consumer and get it approved beforehand, and then just swap the used consumer in the configuration when switching traffic, no live coordination required [14:59:29] yeah, I currently lack mental bandwidth for the move, and will do for the next few months at least. [14:59:37] arturo: hackathon is next week? [15:00:03] RhinosF1: it is, yeah https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2023 [15:00:50] I'm basically in firefighting mode around my own infra being in a degraded state due to hardware failure, and trying to keep tabs on what's going on especially around UA deprecation/client hints. [15:01:47] Oh it's not virtual [15:17:50] I think the opening and closing ceremonies will be streamed, and some sessions might end up doing something, but there's no virtual platform this time [15:18:08] dcaro: virtual was fun [15:18:31] I should get to an in person one eventually [15:18:41] yeah, me too [15:19:05] dcaro: in person is expensive [15:19:35] And time off work [15:19:37] getting visas is another issue too (for some) [15:20:21] Visas aren't bad for me [15:22:17] I've not actually left my country since Feb 2020. [15:22:42] I haven't since august 2019 [15:22:49] I'm going away later in the year [15:22:52] But with family [15:22:59] Which reduces cost [15:23:04] yeah, I managed to get away just as things were starting to kick off, which I'm grateful for [15:23:22] And also that's taking like most of my holiday for the year [15:25:19] right now I'm wondering whether my new passport will arrive before I leave for the Hackathon so I could fill out the form to book travel for Wikimania. not sure what to feel about that [15:25:22] xd I lived in 4 countries since 2019, it was not the best time to move around, I was able to keep safe though [15:26:00] dcaro: that's mental [15:26:32] life happened 🤷 [18:38:54] o/ mystery for a Friday: there's a much longer backstory regarding what i thought was going on, but I realized recently that I get different behavior from nginx on a new Cloud-VPS project than on my other ones. specifically, for `hackathon-2023-ml`, if I create a new instance and set up a basic webproxy for it, it hangs forever and times out with a 504 if I try to visit the URL. doing `sudo apt install nginx` and `sudo systemctl start [18:38:54] nginx` doesn't change anything (example: https://nginx-test-please-work-hackathon.wmcloud.org/). if i do the exact same process on a different project though, I get the expected Welcome to nginx page (example: https://nginx-test-please-work.wmcloud.org/). thoughts on what might be going on and how to confirm/fix it? [18:41:51] isaacj: have you configured a security group that permits incoming http traffic on the broken instances? [18:42:44] ahhh...that might be it. i didn't even think to do it because i haven't done it for other projects as far as i recall but also these are old projects so probably somebody had already done it [18:43:02] i'll look at that config on the working projects and see if there's something i can copy over [18:43:09] thanks taavi ! [18:47:39] yeah, was missing 80 and 443 as shown here (https://wikitech.wikimedia.org/wiki/Help:Security_groups#Examples) so that's almost certainly it. many many thanks again -- this one took embarrassingly long for me to realize what was going on [18:53:49] and working! [18:57:44] great! [19:09:23] seeing the above conversation and hoping it doesn't affect travel to hackathon, but if you make it and attend my session T333853, you can know that you helped it be a success (or at least a bit closer to a success). thankfully i'm a bit better with ML than I am with managing cloud vps instances :) [19:09:24] T333853: [Session] Self-hosting ML models on Cloud Services - https://phabricator.wikimedia.org/T333853 [19:28:18] I'm coming, don't worry! Finnish ID cards can be used as travel documents within the EU these days and my old passport is still valid, just need the new one to meet Singapore expiration time requirements [19:29:11] and I'll definitely at least pop by that session, always cool to see what WMCS tools are being used for [19:30:11] oh yay! [19:33:32] taavi: we have to send our old passports back before they'll send a new one [19:48:11] RhinosF1: interesting. I think ours technically get revoked but we still get to keep them. And then there's the very real possibility that it won't get checked at any point [19:48:44] Helsinki-Vantaa generally doesn't check those for Schengen flights [19:49:25] taavi: revoked for us still involves physically cutting the corner of the front few pages [19:49:46] They also are revoked on an proper database [19:49:49] But yes [20:41:13] !log tools.lexeme-forms deployed 15b7403971 (fix stray character) [20:41:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL