[04:57:49] 10serviceops, 10wikidiff2, 10Better-Diffs-2023, 10Community-Tech (CommTech-Kanban): Deploy wikidiff2 1.14.0 - https://phabricator.wikimedia.org/T340087 (10Legoktm) >>! In T340087#8961668, @tstarling wrote: >>>! In T340087#8958132, @MoritzMuehlenhoff wrote: >> One note: https://github.com/wikimedia/mediawik... [05:49:08] hnowlan: o/ re: rdkafka - nice! I also see that upstream ships librdkafka as their build dep (https://github.com/Blizzard/node-rdkafka/tree/master/deps) so even better for us [05:50:14] 10serviceops, 10ChangeProp, 10WMF-JobQueue: Check if node-rdkafka's version on changeprop can be upgraded from 2.8.1 - https://phabricator.wikimedia.org/T341140 (10elukey) https://github.com/Blizzard/node-rdkafka/tree/master/deps seems to state that `librdkafka` is shipped as build dependency (this is consis... [06:37:15] 10serviceops, 10CX-deployments, 10MinT, 10Language-Team (Language-2023-July-September): Remove Flores key from production - https://phabricator.wikimedia.org/T337284 (10KartikMistry) @jbond @akosiaris Ping. See the above errors happening in the logstash. [08:31:20] 10serviceops, 10CX-deployments, 10MinT, 10Language-Team (Language-2023-July-September): Remove Flores key from production - https://phabricator.wikimedia.org/T337284 (10jbond) >>! In T337284#8992398, @KartikMistry wrote: > @jbond @akosiaris Ping. See the above errors happening in the logstash. @KartikMist... [08:44:31] 10serviceops, 10CX-deployments, 10MinT, 10Language-Team (Language-2023-July-September): Remove Flores key from production - https://phabricator.wikimedia.org/T337284 (10Joe) a:03Joe I think he intended to ping me :) [08:50:34] 10serviceops, 10Observability-Alerting, 10SRE, 10Traffic: Timeouts when talking to phabricator API - https://phabricator.wikimedia.org/T341039 (10fgiunchedi) I have extracted the `maniphest.edit` event duration from phab1004 access log, and on the 29th the operation started to take a whole lot longer: ` 2... [08:51:37] 10serviceops, 10Observability-Alerting, 10SRE, 10Traffic: Timeouts when talking to phabricator API - https://phabricator.wikimedia.org/T341039 (10fgiunchedi) @brennen I saw your updates to phab in SAL, does the above (`maniphest.edit` taking a lot longer to create tasks) ring a bell? [08:56:29] 10serviceops, 10CX-deployments, 10MinT, 10Language-Team (Language-2023-July-September): Remove Flores key from production - https://phabricator.wikimedia.org/T337284 (10Joe) 05Open→03Resolved I removed the key and redeployed cxserver. [10:04:55] 10serviceops, 10Beta-Cluster-Infrastructure, 10wikidiff2, 10Better-Diffs-2023, 10Community-Tech (CommTech-Kanban): Install wikidiff2 1.14.1 deb on deployment-prep & test - https://phabricator.wikimedia.org/T340542 (10TheresNoTime) >>! In T340542#8989804, @MoritzMuehlenhoff wrote: >>>! In T340542#8987623,... [10:14:44] 10serviceops, 10Beta-Cluster-Infrastructure, 10wikidiff2, 10Better-Diffs-2023, 10Community-Tech (CommTech-Kanban): Install wikidiff2 1.14.1 deb on deployment-prep & test - https://phabricator.wikimedia.org/T340542 (10TheresNoTime) `1.14.1` now installed on beta ` samtar@deployment-mediawiki12:~$ php --ri... [10:15:53] claime akosiaris FYI see my latest comment on https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/935089 re: adjusting cirrus cpjobqueue alert [10:24:26] (lunch) [10:49:18] 10serviceops, 10MW-on-K8s: Allow deployers to get a php REPL environment inside the mw-debug pods - https://phabricator.wikimedia.org/T341197 (10Joe) [10:49:39] 10serviceops, 10MW-on-K8s: Allow deployers to get a php REPL environment inside the mw-debug pods - https://phabricator.wikimedia.org/T341197 (10Joe) p:05Triage→03High a:03Joe [11:37:53] 10serviceops, 10MW-on-K8s: Allow deployers to get a php REPL environment inside the mw-debug pods - https://phabricator.wikimedia.org/T341197 (10JMeybohm) [11:38:03] 10serviceops, 10Prod-Kubernetes, 10Toolhub, 10Kubernetes, 10Patch-For-Review: Maintenance environment needed for running one-off commands - https://phabricator.wikimedia.org/T290357 (10JMeybohm) [13:05:37] 10serviceops, 10ChangeProp, 10WMF-JobQueue: Check if node-rdkafka's version on changeprop can be upgraded from 2.8.1 - https://phabricator.wikimedia.org/T341140 (10Ottomata) Pretty sure there is a configurable env var `BUILD_LIBRDKAFKA` that can conditionally disable this. eventgate-wikimedia installs the l... [13:41:05] 10serviceops, 10Maps: Wikimedia is restricting its own access to map tiles - https://phabricator.wikimedia.org/T341226 (10TheDJ) [13:45:55] 10serviceops, 10Maps: Wikimedia is restricting its own access to map tiles - https://phabricator.wikimedia.org/T341226 (10TheDJ) oh wait. these are just throwing 400s of course, but you can't see that because maps can't access itself and throws the 403... regardless, something cross domain is happening here,... [14:37:23] 10serviceops, 10Maps: English Wikipedia maps have error 400 when retrieving the static map image for map with Commons data - https://phabricator.wikimedia.org/T341226 (10TheDJ) [15:17:59] akosiaris, _joe_ - is it ok if I merge https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/935772 ? [15:18:33] <_joe_> elukey: ask the oncall people :D [15:18:35] <_joe_> but, yes [15:18:59] okok :) [15:26:35] 10serviceops, 10Machine-Learning-Team: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471 (10akosiaris) @elukey looks like we are sticking with the old recommendation-api for a while. Should we resolve this? [15:31:10] 10serviceops, 10SRE, 10observability: stop using $::site in description field of service.yaml - https://phabricator.wikimedia.org/T258697 (10akosiaris) Any objections to switching "svc.%{::site}.wmnet" to "discovery.wmnet" ? [15:37:29] 10serviceops, 10Content-Transform-Team-WIP, 10Mobile-Content-Service, 10RESTbase Sunsetting, and 2 others: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10akosiaris) @MSantos, change deployed today. e.g. https://en.wikipedia.org/api/rest_v1/page/mobile-sections now returns a... [15:43:12] 10serviceops, 10SRE, 10observability: stop using $::site in description field of service.yaml - https://phabricator.wikimedia.org/T258697 (10Joe) Is this still relevant? I think we moved all LVS alerts off of icinga by now. But yeah no objection apart from what I stated above. [15:45:50] 10serviceops, 10SRE, 10observability: stop using $::site in description field of service.yaml - https://phabricator.wikimedia.org/T258697 (10akosiaris) No, it's not relevant to icinga so much any more (and it's going to be less and less). It's still an interesting informational thing though and the replaceme... [15:56:41] hnowlan: o/ we don't have node-rdkafka metrics for changeprop right? [16:00:54] elukey: not afaik- even if we emit them they aren't scraped via the statsd gateway [16:01:51] hnowlan: ack I suspected that, it would be nice to have more insights [16:02:37] I don't have a lot of visibility right now, from the changeprop dashboard nothing really changed [16:04:28] anyway, will dig more tomorrow :) [16:05:20] happy to help if there's a way to expose them! [16:06:37] codebase changes are blocked on fixing the build container but that's not a huge deal [16:10:10] 10serviceops, 10Content-Transform-Team-WIP, 10Mobile-Content-Service, 10RESTbase Sunsetting, and 2 others: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10MSantos) 05Open→03Resolved a:03akosiaris >>! In T340036#8994407, @akosiaris wrote: > @MSantos, change deployed tod... [16:10:38] (iff we trust the tests) [17:01:08] 10serviceops, 10Maps: English Wikipedia maps have error 400 when retrieving the static map image for map with Commons data - https://phabricator.wikimedia.org/T341226 (10Pikne) In browser console these 400s have the following response: `Cannot read property 'coordinates' of null`. The map data for all of these... [17:07:56] 10serviceops, 10Page Content Service, 10RESTBase-API: REST PCS not working after REST MCS deprecation - https://phabricator.wikimedia.org/T341248 (10MSantos) [17:15:01] 10serviceops, 10Page Content Service, 10RESTBase-API: REST PCS not working after REST MCS deprecation - https://phabricator.wikimedia.org/T341248 (10akosiaris) 05Open→03Resolved a:03akosiaris Misconfiguration on our side @Brycehughes, thanks for noticing it. Now the 2 URLs above work fine. Resolving, b... [17:16:06] 10serviceops, 10Page Content Service, 10RESTBase-API: REST PCS not working after REST MCS deprecation - https://phabricator.wikimedia.org/T341248 (10Brycehughes) @akosiaris that was fast! Many thanks.