[04:42:42] godog: I'm reviewing/refreshing/ improving the docs for debug logging in MW core (/includes/debug/logger/) and came across the seemingly-unused MwlogHandler class which says it emulates udp2log, sneaking in 1 piece of structured information (channel name appended to appname) through an otherwise mostly unstructured string. ref T126989, T205856 [04:42:43] T126989: MediaWiki logging & encryption - https://phabricator.wikimedia.org/T126989 [04:42:43] T205856: Retire udp2log: onboard its producers and consumers to the logging pipeline - https://phabricator.wikimedia.org/T205856 [04:43:15] Is that still desirable or have we basically found a cheap/sustainable way to run udp2log that doesn't warrant a new implementation? Might be able to remove the unused code in that case. [07:48:17] Krinkle: yeah I think that can be removed for now, removing/replacing udp2log is still desirable IMHO and not in any immediate plans/roadmap as far as I'm aware [08:01:41] GitLab maintenance window starts in one hour: 09:00-13:00 UTC. We will switch GitLab from eqiad to codfw. See T345531 [08:01:41] T345531: Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023 - https://phabricator.wikimedia.org/T345531 [08:59:17] GitLab maintenance starting now - T345531 [08:59:18] T345531: Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023 - https://phabricator.wikimedia.org/T345531 [10:12:01] Can I get someone to review https://gerrit.wikimedia.org/r/c/operations/puppet/+/963313 ? We're dcomming ORES, and I want to merge that change before the week is out, but Luca is on PTO. [10:13:21] I'll take a look [10:13:29] thanks! [10:13:41] sure, LGTM [10:14:38] that role doesn't exist [10:15:24] doh, of course, thank you taavi [10:19:17] the decom docs mention a bug form that says to set it to that role, which naturally should be fixed. What's the right thing to do, then? Just delete the entries entirely? [10:19:24] why moving them to a different role at all? [10:19:37] you have to first run the decom cookbook [10:19:44] That is done. [10:20:10] I suspect the spare:: role is meant for machines that will be reused? Not in this case, but... [10:20:26] then if the hosts are not there anymore just "Remove all references from Puppet repository:" [10:20:29] as mentioned in the docs [10:20:50] which docs are you following? [10:20:53] https://phabricator.wikimedia.org/maniphest/task/edit/form/52/ This for still says to use the non-existent role, hence me trying to do so [10:20:58] form* [10:21:09] https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Active_-%3E_Decommissioned [10:23:00] I was following that, but it doesn't contradict the form explicitly, hence my confusion [10:23:30] the form is managed by dcops, in particular r.ob [10:23:42] I will file a bug about it, as mentioned. [10:24:30] Updated the change to make it a removal of those role refs [10:39:12] taavi friendly ping for +1 :) [10:39:53] klausman: now the commit message is not accurate :P [10:40:59] thx [10:41:09] Kiitos :) [11:49:16] Gitlab is back, maintenance done [11:50:42] 🙌 [11:59:35] heads up to all, i have removed the puppetdbquery module. this module is not compatable with puppet 7 further its functionality can now be achived using wmflib::puppetdb_query (a simple wrapper to puppetdb_query) [11:59:42] I need help with parsoid [11:59:59] Latency x5 https://grafana.wikimedia.org/goto/Z7nExiGSk?orgId=1 [12:00:20] unless anything elses is added to the agended for the puppet office hours (today at 14:00 UTC) ill probably talk about how to leverage puppetdb in puppet code and werther its a good idea or not [12:00:29] Seems to be coming from commons https://logstash.wikimedia.org/goto/652b1aad2b109acc3eaadf2cabbd2ae2 [12:01:04] I don't see a bump of requests to commons in superset [12:01:13] So I can only assume it's something internal going on [12:08:52] I'm seeing a jump in slow queries on s4 *before* but not anymore [12:12:24] I'm not seeing anything in SAL around that time [12:14:53] And it now has gone back down under 1s [12:35:25] FYI; lists1001 (the VM running lists.wikimedia.org) will be rebooted in ~ 5m [12:46:42] hnowlan: are you around? the 5xx rate from thumbor is rising in codfw and the haproxy queue is mabye on the long side... [12:47:15] (I have already rolling-restarted the codfw swift frontends as a precaution, but I don't think it's a swift fault) [12:47:55] Emperor: There's a rise in requests since ~1150 [12:48:23] Wait wrong dash [12:50:28] https://grafana.wikimedia.org/goto/GN1TPiGSk?orgId=1 [12:50:43] 5xx rate has been steadily rising for a moment actually [12:51:26] yeah, hence my twitchyness that something is awry with thumbor [12:53:15] Emperor: Seems like it's mostly two pods misbehaving if I'm reading this correctly https://grafana.wikimedia.org/goto/B-aBPiMIk?orgId=1 [12:56:56] looks plausible (but I know little about the internals of thumbor) [12:59:51] I'll delete them, they'll be respawned new and we'll see [13:00:34] Or do you think it can wait until hnowlan answers ? [13:01:40] claime: I'd be inclined to go ahead [13:01:52] a'ight [13:13:56] Emperor: Looks like it's getting better [13:22:03] Back to baseline, going to lunch [13:22:57] TY :) [13:59:46] oops, had an appointment so I missed this - I'll see if I can find out what happened [15:18:08] cdanis: FYI https://github.com/mozilla/standards-positions/issues/99#issuecomment-1746708536 ff may be reconsidering NEL [15:18:35] jbond: FF never implemented it in the first place :) [15:18:41] I've been watching that task [15:18:51] yes they rejected it becaus of privicy concerns [15:18:53] yeah [15:18:55] ack [15:18:58] I might post [15:19:17] some of their ideas I totally agree with (dropping NEL policy immediately if you get served no NEL header for a site you had one stored for) [15:19:29] others would defeat a lot of the utility of NEL (hiding IP addresses) [15:19:47] yes its always tricky the balance between privacy and metricts [15:20:20] request_headers and success reports are questionable, for sure [15:20:54] ah I missed the other task linked there [15:21:33] yes agree [15:21:54] (although we did make good use of success reports, we also had other ways to get that data) [15:22:33] yes success reports are nice to have but like you say we can infer that from other data and not neccesarrily something id want my browser to be shouting about [20:19:07] Anyone have a good way to add an image to a gerrit comment, I could use a random image hosting website, but that seems less than great [20:19:58] drag-and-drop an image to the phabricator front page [20:20:22] interesting, what does that do taavi? [20:21:15] https://phabricator.wikimedia.org/file/upload/ [20:21:47] very cool, thanks folks, I should try drag and dropping images on more random websites ;) [20:21:54] it's just a general purpose file host [20:23:36] I wonder if there is a cli tool that works with our phabricator instance to upload files, would be handy [20:25:19] looks like Arcanist does have support for it [20:25:54] nice thanks AntiComposite