[06:52:17] 10Beta-Cluster-Infrastructure: login verification email from beta never arrives (gmail.com email) - https://phabricator.wikimedia.org/T443183 (10ArielGlenn) 03NEW [06:58:05] 10Beta-Cluster-Infrastructure: login verification email from beta never arrives (gmail.com email) - https://phabricator.wikimedia.org/T443183#12500178 (10VeritasSapientiae) Can confirm with my gmail address. [07:10:58] (03CR) 10Hashar: [C:03+2] "Thank you and welcome to Python 3.10 😊" [integration/config] - 10https://gerrit.wikimedia.org/r/1361150 (https://phabricator.wikimedia.org/T423572) (owner: 10Xqt) [07:12:12] (03Merged) 10jenkins-bot: jjb/zuul: use Python 3.10 instead of 3.9 for deeptest [integration/config] - 10https://gerrit.wikimedia.org/r/1361150 (https://phabricator.wikimedia.org/T423572) (owner: 10Xqt) [07:14:50] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1361150 "use Python 3.10 instead of 3.9 for deeptest" # T423572 [07:14:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:14:54] T423572: Drop support for Python 3.9 - https://phabricator.wikimedia.org/T423572 [07:39:53] 10Beta-Cluster-Infrastructure: login verification email from beta never arrives (gmail.com email) - https://phabricator.wikimedia.org/T443183#12500233 (10ArielGlenn) [07:51:11] (03CR) 10Hashar: "I have fixed the year in the commit message." [integration/config] - 10https://gerrit.wikimedia.org/r/1360519 (owner: 10Jforrester) [07:53:29] 10Beta-Cluster-Infrastructure: login verification email from beta never arrives (gmail.com email) - https://phabricator.wikimedia.org/T443183#12500259 (10Peachey88) [08:32:17] (03PS1) 10Arendpieter: zuul: [mediawiki/extensions/Sanctions] Add Linter dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1361326 [08:38:24] (03PS2) 10Hashar: zuul: [mediawiki/extensions/Sanctions] Add Linter dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1361326 (owner: 10Arendpieter) [08:39:22] (03CR) 10Hashar: [C:03+2] "I have slightly amended the commit message to remove the CI console log since it will vanish after a few days." [integration/config] - 10https://gerrit.wikimedia.org/r/1361326 (owner: 10Arendpieter) [08:40:45] (03Merged) 10jenkins-bot: zuul: [mediawiki/extensions/Sanctions] Add Linter dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1361326 (owner: 10Arendpieter) [08:42:21] (03CR) 10Hashar: [C:03+2] "Deployed and I have done a `recheck` on I5e30aa540e28a94ea039439da70d1622d32b8112" [integration/config] - 10https://gerrit.wikimedia.org/r/1361326 (owner: 10Arendpieter) [08:58:03] 10Beta-Cluster-Infrastructure, 06Growth-Team: Jobs do not run on Beta Cluster since the 2026-10-29 reboots - https://phabricator.wikimedia.org/T443050#12500379 (10Urbanecm_WMF) I just did the cherry pick. I didn't do it on the Friday, because I saw jenkins voted -1 and I was too lazy to take a look. Now that I... [09:00:12] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Replace deployment-zookeeper-3 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436481#12500381 (10Urbanecm_WMF) [09:00:34] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Replace deployment-zookeeper-3 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436481#12500382 (10Urbanecm_WMF) Noting I cherry picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1360972 to beta... [09:01:34] 10Beta-Cluster-Infrastructure, 06Growth-Team (FY2026-27 Q2 Sprint 2): Jobs do not run on Beta Cluster since the 2026-10-29 reboots - https://phabricator.wikimedia.org/T443050#12500403 (10Urbanecm_WMF) p:05Triageβ†’03High a:03Urbanecm_WMF [09:01:54] 10Beta-Cluster-Infrastructure, 06Growth-Team (FY2026-27 Q2 Sprint 2): Jobs do not run on Beta Cluster since the 2026-10-29 reboots - https://phabricator.wikimedia.org/T443050#12500408 (10Urbanecm_WMF) Last processed job is `2026-11-02 09:00:51.686233`. Sounds about right. [09:09:34] 10Beta-Cluster-Infrastructure, 06Growth-Team (FY2026-27 Q2 Sprint 2): Jobs do not run on Beta Cluster since the 2026-10-29 reboots - https://phabricator.wikimedia.org/T443050#12500435 (10Urbanecm_WMF) Triggered a reboot of deployment-zookeeper-3 to make sure it survives [09:10:54] 10Beta-Cluster-Infrastructure, 06Growth-Team (FY2026-27 Q2 Sprint 2): Jobs do not run on Beta Cluster since the 2026-10-29 reboots - https://phabricator.wikimedia.org/T443050#12500440 (10Urbanecm_WMF) 05Openβ†’03Resolved After the reboot: ` urbanecm@deployment-zookeeper-3:~$ sudo service zookeeper statu... [09:12:36] err. I can't access spiderpig right now. Any idea why that might be? I was mid-deploy [09:14:06] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-zookeeper-3 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [09:17:49] okay. we're back in business [09:20:12] I was able to access it ok (log in, see scap backport 1355410 in progress [09:20:24] it says now "waiting 20 seconds for canary traffic" [09:20:37] oh, you got it resolved? [09:20:40] yeah - it's returned. It was down for about 10 mins [09:20:44] huh [09:20:48] (down / unreachable for me) [09:20:51] and a colleague [09:21:32] if it's good for you ima go sleep now (1:20 am) happy deploying! [09:21:40] 10GitLab (Infrastructure), 06Collaboration-Services: Improve error handling in sre.gitlab.upgrade cookbook - https://phabricator.wikimedia.org/T443192 (10Jelto) 03NEW [09:21:48] apergos: is good - thank you! [09:31:19] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown - gerrit1003 - https://phabricator.wikimedia.org/T442798#12500510 (10ABran-WMF) a:03ABran-WMF Found a scraper, added in: https://requestctl.wikimedia.org/commit/68e07110c3e88fc769274442c8d2cd041c3a14a8 JA4H already present in 4 other rules [09:31:42] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown - gerrit1003 - https://phabricator.wikimedia.org/T442798#12500512 (10ABran-WMF) Turnilo query: https://w.wiki/fo7M [09:33:16] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown - gerrit1003 - https://phabricator.wikimedia.org/T442798#12500525 (10ABran-WMF) 05Openβ†’03Resolved same for https://requestctl.wikimedia.org/commit/61378740aae0ef2ed42535517bfd3fbc11ddc18c via the same query, also already present in 4 other... [09:46:25] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown - gerrit1003 - https://phabricator.wikimedia.org/T443160#12500596 (10ABran-WMF) [09:54:49] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown - gerrit1003 - https://phabricator.wikimedia.org/T443160#12500671 (10ABran-WMF) 05Openβ†’03Resolved a:03ABran-WMF Edge case discussed in T428996#12500656 [09:54:57] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown - gerrit1003 - https://phabricator.wikimedia.org/T443160#12500676 (10ABran-WMF) [10:56:16] 10Phabricator, 06DBA: Connection to Phabricator DB fails for phabricator-admins - https://phabricator.wikimedia.org/T443211 (10Urbanecm_WMF) 03NEW [11:00:51] 06Release-Engineering-Team, 10Scap, 06ServiceOps, 10ServiceOps-Datastores, and 2 others: Add etcdmirror status check to scap - https://phabricator.wikimedia.org/T317403#12500947 (10JMeybohm) For etcdmirror there are EtcdReplicationDown and EtcdReplicationLag alerts. We don't have anything for "configuratio... [11:21:39] 10Phabricator, 06Collaboration-Services, 06DBA, 07Regression: Connection to Phabricator DB fails for phabricator-admins - https://phabricator.wikimedia.org/T443211#12501048 (10Aklapper) Indeed, something broke somewhere. ` aklapper@phab1005:/srv/phab/phabricator$ mysql -h m3-slave.eqiad.wmnet -P 3323 -u ph... [11:26:55] 10Phabricator, 06Collaboration-Services, 06DBA, 07Regression: Connection to Phabricator DB fails for phabricator-admins - https://phabricator.wikimedia.org/T443211#12501096 (10Ladsgroup) I think that's {T443208} @CWilliams-WMF I was wrong, that replica is used, we need to fail over to another replica :( bu... [11:35:53] https://development-metrics.wmcloud.org/ is broken it seems [12:34:14] 06Release-Engineering-Team, 10Scap, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12501580 (10KartikMistry) Note that the Recommendation-API node version is no longer used in Production. We can decommission it. CC @SBi... [12:35:06] (03PS4) 10Hslater: Zuul: BlueSpice: drop PHP 8.4/8.5 in initial test [integration/config] - 10https://gerrit.wikimedia.org/r/1355158 [12:37:25] 10Phabricator, 06Collaboration-Services, 06DBA, 07Regression: Connection to Phabricator DB fails for phabricator-admins - https://phabricator.wikimedia.org/T443211#12501599 (10Urbanecm_WMF) > Asked differently: Can the features wait until the hw is fixed? Depends on when that would be. From my end: I was... [12:38:42] 06Release-Engineering-Team, 10Scap, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12501603 (10KartikMistry) [12:39:14] 06Release-Engineering-Team, 10Scap, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12501607 (10KartikMistry) [12:48:36] 10Phabricator, 06Collaboration-Services, 06DBA, 07Regression: Connection to Phabricator DB fails for phabricator-admins - https://phabricator.wikimedia.org/T443211#12501649 (10CWilliams-WMF) > Depends on when that would be. From my end: I was querying the DB for debugging purposes and I'm happy to wait a f... [12:50:49] 06Release-Engineering-Team (Doing 😎), 06WMF-NDA-Requests: Requesting access to WMF-NDA-Requests to get access to instructions for new SSH standards for WMF Staff - https://phabricator.wikimedia.org/T443218#12501660 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Added after verification [12:51:55] 06Release-Engineering-Team (Doing 😎), 06WMF-NDA-Requests: Requesting access to WMF-NDA-Requests to get access to instructions for new SSH standards for WMF Staff - https://phabricator.wikimedia.org/T443218#12501666 (10Aklapper) Note that this also enforces MFA in Phabricator (different thing), see https://... [12:57:15] 06Release-Engineering-Team (Doing 😎), 06WMF-NDA-Requests: Requesting WMF-NDA access for RHo - https://phabricator.wikimedia.org/T443223#12501688 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Added after verification. Note that WMF-NDA access within Phabricator also enforces MFA in Phabricator, see http... [13:36:43] 10Phabricator-Bot-Requests, 10Toolforge, 06tools-platform-team: Bot for Toolforge GitLab automation - https://phabricator.wikimedia.org/T443233 (10Raymond_Ndibe) 03NEW [13:47:00] 10Phabricator-Bot-Requests, 10Toolforge, 06tools-platform-team: Bot for Toolforge GitLab automation - https://phabricator.wikimedia.org/T443233#12501952 (10Raymond_Ndibe) [13:54:17] 10Phabricator-Bot-Requests, 06Release-Engineering-Team (Doing 😎), 10Toolforge, 06tools-platform-team: Bot for Toolforge GitLab automation - https://phabricator.wikimedia.org/T443233#12502032 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Created @gitlab-toolforge-bot. Conduit API Token in non-public... [13:57:19] 10Phabricator: Decouple setting #permanentlyprivate from setting #security and #security-team project tags - https://phabricator.wikimedia.org/T443243 (10Aklapper) 03NEW [13:57:27] 10Phabricator: Decouple setting #permanentlyprivate from setting #security and #security-team project tags - https://phabricator.wikimedia.org/T443243#12502072 (10Aklapper) [14:04:46] (03PS1) 10Hashar: dockerfiles: reparent zuul-cloner on python-all [integration/config] - 10https://gerrit.wikimedia.org/r/1361434 (https://phabricator.wikimedia.org/T335765) [14:07:28] (03CR) 10Jforrester: "Very neat! Yes, let's do it." [integration/config] - 10https://gerrit.wikimedia.org/r/1361434 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [14:09:18] (03PS1) 10Hashar: dockerfiles: phase out Buster images [integration/config] - 10https://gerrit.wikimedia.org/r/1361439 (https://phabricator.wikimedia.org/T335765) [14:10:31] (03CR) 10Hashar: [C:03+2] "I am not sure why I forgot / postponed that move 😊" [integration/config] - 10https://gerrit.wikimedia.org/r/1361434 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [14:10:55] James_F: it only took 3 years and a half to get rid of Buster :b ( https://gerrit.wikimedia.org/r/c/integration/config/+/1361439/1?usp=related-change ) [14:10:57] err [14:10:58] (03CR) 10Jforrester: [C:03+1] "<3" [integration/config] - 10https://gerrit.wikimedia.org/r/1361439 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [14:11:00] https://phabricator.wikimedia.org/T335765 [14:11:11] which makes me wonder whether we have a similar task for Bullseye [14:11:14] 10Phabricator-Bot-Requests, 06Release-Engineering-Team (Doing 😎), 10Toolforge, 06tools-platform-team: Bot for Toolforge GitLab automation - https://phabricator.wikimedia.org/T443233#12502156 (10Raymond_Ndibe) >>! In T443233#12502032, @Aklapper wrote: > Created @gitlab-toolforge-bot. Conduit API Token i... [14:11:29] hashar: First commit, July 2019. Deletion, November 2026. Oops. [14:11:41] Yes, https://phabricator.wikimedia.org/T383337 [14:11:58] E X C E L L E N T [14:12:00] And https://phabricator.wikimedia.org/T397049 for dropping Bookworm. [14:12:06] ++ [14:12:11] (03Merged) 10jenkins-bot: dockerfiles: reparent zuul-cloner on python-all [integration/config] - 10https://gerrit.wikimedia.org/r/1361434 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [14:12:14] No task yet for dropping Trixie, I’m not *that* keen. :-) [14:12:26] by the time we drop Trixie, I will be retired [14:12:27] :b [14:13:17] hashar: BTW, the java11/17 images can’t currently be built, so if we’re also bumping those we’ll have to fix up the images. Can we just bully the relevant teams so we can drop them? [14:14:27] yeah I think I commented on it earlier [14:14:46] 06Release-Engineering-Team (Doing 😎), 06WMF-NDA-Requests: Requesting access to wmf-nda phabricator group for mvolz - https://phabricator.wikimedia.org/T443237#12502163 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Added you after verification (and as your linked LDAP account is a member of `ldap/wmf`).... [14:15:07] we have some production systems still on Java 8 [14:15:10] for Java 11 I have no idea [14:15:45] (03PS1) 10Jforrester: Zuul: Add Punith.nyk to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1361442 [14:16:06] Yeah, Java 8 builds fine. [14:16:12] But 11 and 17 don’t. [14:16:19] (Sigh.) [14:16:27] for java 11 / 17 we might have backports available for newer Debian versions that what ever is currently used [14:16:37] Yeah, we could re-platform I suppose? [14:16:54] (03CR) 10Jforrester: [C:03+2] Zuul: Add Punith.nyk to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1361442 (owner: 10Jforrester) [14:17:49] I have updated the integration-config jobs to use the new zuul-cloner [14:17:54] (03PS1) 10Hashar: jjb: update jobs for zuul-cloner [integration/config] - 10https://gerrit.wikimedia.org/r/1361444 (https://phabricator.wikimedia.org/T335765) [14:18:10] (03Merged) 10jenkins-bot: Zuul: Add Punith.nyk to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1361442 (owner: 10Jforrester) [14:19:00] 10Phabricator, 06Collaboration-Services, 06DBA, 07Regression: Connection to Phabricator DB fails for phabricator-admins - https://phabricator.wikimedia.org/T443211#12502177 (10Marostegui) >>! In T443211#12501094, @Ladsgroup wrote: > I think that's {T443208} @CWilliams-WMF I was wrong, that replica is used,... [14:20:30] (03CR) 10Hashar: [C:03+2] jjb: update jobs for zuul-cloner [integration/config] - 10https://gerrit.wikimedia.org/r/1361444 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [14:21:42] (03Merged) 10jenkins-bot: jjb: update jobs for zuul-cloner [integration/config] - 10https://gerrit.wikimedia.org/r/1361444 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [14:22:04] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all CI jobs from buster to bullseye or later and drop buster testing support - https://phabricator.wikimedia.org/T335765#12502219 (10hashar) [14:22:46] 10Gerrit, 06Traffic, 07Browser-Support-Google-Chrome, 07Wikimedia-production-error: Error 403: Wikimedia Error when viewing Gerrit using Google Chrome - https://phabricator.wikimedia.org/T439314#12502236 (10Meno25) 05Openβ†’03Resolved a:03ssingh >>! In T439314#12491623, @ssingh wrote: > Hi everyone... [14:22:48] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all CI jobs from buster to bullseye or later and drop buster testing support - https://phabricator.wikimedia.org/T335765#12502233 (10hashar) 05Openβ†’03Resolved a:03hashar Success, finally :) Thank... [14:22:48] (03CR) 10Hashar: [C:03+2] dockerfiles: phase out Buster images [integration/config] - 10https://gerrit.wikimedia.org/r/1361439 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [14:24:03] (03Merged) 10jenkins-bot: dockerfiles: phase out Buster images [integration/config] - 10https://gerrit.wikimedia.org/r/1361439 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [14:24:48] 10Phabricator, 06Collaboration-Services, 06DBA, 07Regression: Connection to Phabricator DB fails for phabricator-admins - https://phabricator.wikimedia.org/T443211#12502249 (10CWilliams-WMF) The BIOS and firmware has been updated, so I will be starting MariaDB shortly. [14:45:38] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all CI jobs from Bullseye to Bookworm or later and drop Bullseye testing support - https://phabricator.wikimedia.org/T383337#12502419 (10hashar) [14:56:38] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all CI jobs from Bullseye to Bookworm or later and drop Bullseye testing support - https://phabricator.wikimedia.org/T383337#12502488 (10Jdforrester-WMF) [15:03:06] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-webperf31 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [15:03:12] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-webperf31 on project deployment-prep - https://phabricator.wikimedia.org/T443255 (10wmcs-alerts) 03NEW [15:33:36] (03PS1) 10Jforrester: dockerfiles: Rebuild all images for the ca-certificates update [integration/config] - 10https://gerrit.wikimedia.org/r/1361480 [15:33:48] hashar: ^^^ Bump all the images for you. [15:35:41] Project mediawiki-core-phpmetrics build #905: 04FAILURE in 1 min 39 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-phpmetrics/905/ [15:40:44] James_F: there are some images that are based on bullseye and need to be reparented :) [15:40:52] I listed them on https://phabricator.wikimedia.org/T443253 [15:41:06] buildctl, castor [15:41:10] and python2-build-bullseye can't be updated :b [15:41:14] Yeah, I have patches for those too. [15:41:20] But I didn’t want to conflict. [15:41:53] so your patch doing the whole rebuild will have anything depending on bullseye to fail [15:42:06] they should be excluded from your patch "Rebuild all images for the ca-certificates update [integration/config] - https://gerrit.wikimedia.org/r/1361480" [15:42:42] aka rebuild ci-bookworm/ci-trixie and cascade, excluding I imagine the java images [15:44:31] COPY --from=moby/buildkit@sha256:cf4bce09da1cd0bbe7ce7f3033011d3db44bcbdee8efc58671440a5d64444f8f /usr/bin/buildctl /usr/bin/buildctl [15:44:31] eeek [15:44:36] Yeah. :-( [15:44:49] (03PS1) 10Jforrester: dockerfiles: Rebuild all images for the ca-certificates update [integration/config] - 10https://gerrit.wikimedia.org/r/1361486 [15:45:05] Not sure if ci-bullseye will work, let’s fine out. [15:46:15] (03Abandoned) 10Jforrester: dockerfiles: Rebuild all images for the ca-certificates update [integration/config] - 10https://gerrit.wikimedia.org/r/1361486 (owner: 10Jforrester) [15:46:56] (03PS2) 10Jforrester: dockerfiles: Rebuild all images for the ca-certificates update [integration/config] - 10https://gerrit.wikimedia.org/r/1361480 (https://phabricator.wikimedia.org/T443253) [15:47:23] dduvall: good morning. Do you know whether the releng/buildctl image is still of any use. It is defined at https://gerrit.wikimedia.org/g/integration/config/+/refs/heads/master/dockerfiles/buildctl/ and point to a version of moby/buildkit from 2022. My guess is that it is no more used anywhere and we can remove it :) [15:47:58] dduvall: the image is based on Bullseye which will eventually phase out. If it is still needed we can make it based on Trixie :) [15:50:00] James_F: in https://gerrit.wikimedia.org/r/c/integration/config/+/1361480 the ci-bullseye image will fail due to lack of ca-certificates package [15:50:01] :/ [15:50:24] and the *java* ones can be build independently (they need java update) [15:50:33] Ack, that’s what I pushed on Friday. [15:50:33] (03merge) 10dancy: Add prometheus-atlas-exporter to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/196 (https://phabricator.wikimedia.org/T429672) (owner: 10lizf) [15:50:50] I am also not sure what kind of mess this will cause by upgrading a wide range of packages all other the place :\ [15:51:00] we should use Nix! [15:51:04] Ha. [15:51:18] brb [15:56:40] πŸ‘‹ Can somebody help me figure out why this build is stuck? https://gerrit.wikimedia.org/r/c/operations/software/tegola/+/1361483 [15:58:26] More specifically this `trigger-tegola-pipeline-publish` is not run [15:59:26] Yippee, build fixed! [15:59:26] Project mediawiki-core-phpmetrics build #906: 09FIXED in 5 min 8 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-phpmetrics/906/ [16:00:41] nemo-yiannis: looks like it worked and CI merged the change? [16:01:03] yes but the docker image publishing is stuck for quite some time [16:01:04] there was probably a bit of backlog in CI which delayed the run [16:01:10] ok [16:01:48] nemo-yiannis: the post merge job can be seen at https://integration.wikimedia.org/zuul/#q=1361483 [16:01:54] it has a lower precedence than everything else [16:01:57] but will eventually be run [16:02:09] ok sounds good,thanks [16:25:33] 06Release-Engineering-Team (Radar), 07Essential-Work, 10Test Kitchen (Experiment Platform Sprint 32): Determine if we can expand TK experiment deployment windows - https://phabricator.wikimedia.org/T440612#12503370 (10JVanderhoop-WMF) Thanks @DLynch @Jdlrobson-WMF, it sounds like we want the same thing: bein... [16:34:15] 06Release-Engineering-Team, 10dev-images, 10MediaWiki-Docker: dev/trixie-php-jobrunner and dev/trixie-apache2 latest images are broken, breaking docker - https://phabricator.wikimedia.org/T443035#12503437 (10brennen) 05Openβ†’03Invalid Guessing that status change was unintended. [16:36:14] 06Release-Engineering-Team (Doing 😎), 06Collaboration-Services, 06Infrastructure-Foundations, 13Patch-For-Review: Replace Go templates with Rego policy DSL in the JWT authorizer - https://phabricator.wikimedia.org/T346331#12503452 (10LSobanski) p:05Mediumβ†’03Triage [16:37:02] 10GitLab (Infrastructure), 06Collaboration-Services: Improve error handling in sre.gitlab.upgrade cookbook - https://phabricator.wikimedia.org/T443192#12503468 (10LSobanski) p:05Triageβ†’03Medium [17:02:50] 06Release-Engineering-Team, 10Development-Metrics: development-metrics.wmcloud.org is down - https://phabricator.wikimedia.org/T443279 (10taavi) 03NEW [17:36:28] 06Release-Engineering-Team, 10dev-images, 10Catalyst (Luka Ijo Pimeja Jan): Upgrade apache2 images in Catalyst to Trixie - https://phabricator.wikimedia.org/T443286 (10brennen) 03NEW [17:43:25] Does anyone know if the active bullseye instances at https://os-deprecation.toolforge.org/bullseye/deployment-prep.html are false positives because someone did an in-place OS upgrade? [17:47:18] 10Phabricator, 07Epic: Make sure anti-vandalism features are up to snuff - https://phabricator.wikimedia.org/T84#12504060 (10Aklapper) [17:50:47] 10Beta-Cluster-Infrastructure: login verification email from beta never arrives (gmail.com email) - https://phabricator.wikimedia.org/T443183#12504076 (10ArielGlenn) see also T285527 [18:04:42] o/ I have recently added a repo to the trusted runner list but I'm still getting errors on the `publish-image` step. The job I am trying to rerun is from a merge before the repo was added to the list - is that a problem? (https://gitlab.wikimedia.org/repos/sre/miscweb/mobile-app-microsite/-/pipelines/242197) [18:09:17] hnowlan: You'll need to start a fresh pipeline [18:09:42] Grand, thank you! [18:35:49] (03open) 10dancy: spiderpig: Close database sessions before sending the response [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1396 [18:35:50] (03update) 10dancy: spiderpig: Close database sessions before sending the response [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1396 [18:36:29] (03update) 10dancy: spiderpig: Close database sessions before sending the response [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1396 [18:37:29] (03PS3) 10Hashar: jjb: reenable Catalys based timed jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1359927 (https://phabricator.wikimedia.org/T442664) [18:39:13] (03merge) 10dancy: spiderpig: Close database sessions before sending the response [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1396 [18:41:51] (03open) 10dancy: Release 4.310.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1397 [18:45:22] (03merge) 10dancy: Release 4.310.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1397 [18:45:27] !log Set set_min_part_hours to 6 for all Swift rings - T436476 [18:45:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:45:31] T436476: Replace Swift hosts: deployment-ms-(be0[78]|fe04) (Bullseye deprecation) - https://phabricator.wikimedia.org/T436476 [18:45:52] !log Remove deployment-ms-be07 and deployment-ms-be08 from all Swift rings - T436476 [18:45:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:58:51] 10Scap: scap deploy: Say whether or not service restart happened - https://phabricator.wikimedia.org/T443314 (10dancy) 03NEW [19:04:06] err, what is the procedure to deploy a change to a blubber.yaml file for a service? I mean let's say I +2 it, then what? feel free to point me to reading material, I didn't find the right thing but I may not have known how to look [19:06:06] 10GitLab: Errors while browsing GitLab (web) - https://phabricator.wikimedia.org/T443318 (10mpopov) 03NEW [19:07:48] 10GitLab (Support), 06Collaboration-Services: Errors while browsing GitLab (web) - https://phabricator.wikimedia.org/T443318#12504822 (10brennen) [19:16:12] 10GitLab (Support), 06Collaboration-Services: Errors while browsing GitLab (web) - https://phabricator.wikimedia.org/T443318#12504910 (10mpopov) Okay… I refreshed and got "please contact the wikimedia SRE team to discuss a less disruptive approach at noc@wikimedia.org." error Then I SHIFT-Refreshed and that a... [19:17:16] 10GitLab (Support), 06Collaboration-Services: Errors while browsing GitLab (web) - https://phabricator.wikimedia.org/T443318#12504911 (10mpopov) That is, whenever I hit an error if I hard refresh that seems to fix the page for me. [19:19:09] apergos: if the repo is wired up for +2 handling with zuul then nothing special should be needed for changing .pipeline/blubber.yaml. [19:21:19] 10Beta-Cluster-Infrastructure: login verification email from beta never arrives (gmail.com email) - https://phabricator.wikimedia.org/T443183#12504962 (10ArielGlenn) Because I have beta access via the cli, I tried edit.php to make an edit as the user, but that wasn't enough to overcome the verification required... [19:21:52] oh? jsut +2 and it goes around? (er, how do I check that the repo is wired up appropriately?) [19:21:59] bd808: [19:22:19] (03CR) 10Vaughn Walters: [C:03+2] "All jobs were tested and updated and ran live and passed. Merging." [integration/config] - 10https://gerrit.wikimedia.org/r/1359927 (https://phabricator.wikimedia.org/T442664) (owner: 10Hashar) [19:24:02] (03Merged) 10jenkins-bot: jjb: reenable Catalys based timed jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1359927 (https://phabricator.wikimedia.org/T442664) (owner: 10Hashar) [19:28:33] 06Release-Engineering-Team (Doing 😎), 06WMF-NDA-Requests: Requesting Access to WMF-NDA Project in Phabricator - https://phabricator.wikimedia.org/T443311#12505070 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Added after verification [19:42:49] 10GitLab, 06Release-Engineering-Team: Create new GitLab project repository: itservices / tools - https://phabricator.wikimedia.org/T443341 (10sbassett) 03NEW [19:52:50] 10GitLab (Support), 06Collaboration-Services: Errors while browsing GitLab (web) - https://phabricator.wikimedia.org/T443318#12505318 (10mpopov) 05Openβ†’03Invalid Closing for now but will reopen if it starts happening again :) [19:53:00] 10GitLab (Support), 06Collaboration-Services: Errors while browsing GitLab (web) - https://phabricator.wikimedia.org/T443318#12505322 (10mpopov) 05Invalidβ†’03Resolved [20:03:27] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all CI jobs from Bullseye to Bookworm or later and drop Bullseye testing support - https://phabricator.wikimedia.org/T383337#12505438 (10hashar) [20:07:51] 10GitLab (Project and group requests), 06Release-Engineering-Team: Create new GitLab project repository: itservices / tools - https://phabricator.wikimedia.org/T443341#12505453 (10brennen) a:03brennen I'll create a `/repos/` subgroup that you all can manage. Is `it-services` ok to fit the general naming patt... [20:11:21] 10GitLab (Project and group requests), 06Release-Engineering-Team: Create new GitLab project repository: itservices / tools - https://phabricator.wikimedia.org/T443341#12505463 (10sbassett) >>! In T443341#12505453, @brennen wrote: > I'll create a `/repos/` subgroup that you all can manage. Is `it-services` ok... [20:12:29] 06Release-Engineering-Team, 10Development-Metrics: development-metrics.wmcloud.org is down - https://phabricator.wikimedia.org/T443279#12505465 (10Aklapper) ` [ac@foo ~]$ ssh nginx.development-metrics.eqiad1.wikimedia.cloud aklapper@nginx:~$ sudo docker ps -a CONTAINER ID IMAGE COMMAND... [20:20:00] 10GitLab (Project and group requests), 06Release-Engineering-Team: Create new GitLab project repository: itservices / tools - https://phabricator.wikimedia.org/T443341#12505504 (10bcampbell) `it-services` works for us, thanks. [20:23:21] 10GitLab (Project and group requests), 06Release-Engineering-Team, 07Essential-Work: Create new GitLab project repository: itservices / tools - https://phabricator.wikimedia.org/T443341#12505522 (10brennen) 05Openβ†’03Resolved Created: https://gitlab.wikimedia.org/repos/it-services You're both owners.... [20:23:36] 06Release-Engineering-Team, 10dev-images, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Upgrade apache2 images in Catalyst to Trixie - https://phabricator.wikimedia.org/T443286#12505530 (10brennen) [20:26:07] (03open) 10dancy: deploy: Report service restarts and skipped targets [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1398 (https://phabricator.wikimedia.org/T443314) [20:27:33] (03update) 10dancy: deploy: Report service restarts and skipped targets [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1398 (https://phabricator.wikimedia.org/T443314) [20:28:09] (03merge) 10dancy: deploy: Report service restarts and skipped targets [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1398 (https://phabricator.wikimedia.org/T443314) [20:30:28] (03open) 10dancy: Release 4.311.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1399 [20:32:21] (03merge) 10dancy: Release 4.311.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1399 [20:36:23] apergos: I guess in the CI config if you haven't used the repo enough to know if you typically need to manually merge after a +2. I can peek if you want to share the repo (I don't see anything obvious stalking you in gerrit). [20:44:38] first time deployer over in there for this service and first time with blubber and etc so yeah, it's all "welp, where do I look for that. repo in question is mediawiki/services/change-propagation and when I look at integration/config/zuul/layout.yaml there is this line right under the changeprop stuff that says [20:44:42] - name: mediawiki/services/change-propagation/deploy [20:44:42] # Though the main repo is tested via the pipeline, it's still manually deployed without CI. [20:44:43] # template: [20:44:43] # - name: archived [20:45:07] but maybe the comments mean that's how it was and now it is via the pipeline? I don't understand the contents well enough to know what's going on [20:45:12] bd808: ^ [20:45:31] and what I'd like to do is be able to understand so I don't have to ask otehr people to do it for me... [20:46:52] what I think is that there are going to be two changes here, one to the blubber file that is the change I want to merge and get around (node 20 -> 26). and then changes to the heml values.yaml files, which I guess would use the new blubber image name. and then those would have to go around $somehow [20:50:37] well the change-propagation/deploy repo was archived since it was part of scap3 deploys of services. Now it's on k8s as I understand and is probably: https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Deploying_with_helmfile [20:52:07] ok well that's more info than I had, excellent (making notes right now) [20:52:12] but mediawiki/services/change-propagation is still hooked up to https://integration.wikimedia.org/ci/job/change-propagation-pipeline-publish/ post-merge (which builds images and pushes to https://docker-registry.wikimedia.org/wikimedia/mediawiki-services-change-propagation/tags/ ) [20:52:56] apergos: In the services I work with, pipelinelib builds new containers as a post-merge job. I lookup the new tag in the Jenkins output, and then I do all the manual dancing to update, merge, and deploy the helmfile changes. [20:52:59] so the first part of that (+2 the blubber change, it auto generates images, I should prep a patch to the values.yaml files) is still good? [20:53:35] https://wikitech.wikimedia.org/wiki/Toolhub.wikimedia.org#How_to_deploy is the dance for Toolhub which should be mostly the same as any other k8s service. [20:53:36] !log Delete deployment-ms-be07 and deployment-ms-be08 - T436476 [20:53:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:53:40] T436476: Replace Swift hosts: deployment-ms-(be0[78]|fe04) (Bullseye deprecation) - https://phabricator.wikimedia.org/T436476 [20:54:42] ok. and last question (maybe): is it ok to have the values.yaml patch sit around for awhile or is it expected that once the blubber change is merged, someone oughta get all the rest done in the next 5 minutes or whatever? [20:56:21] apergos: I wouldn't merge the values.yaml change and then walk away, but the change just sitting in gerrit is fine (or not having been made at all). [20:56:52] right right. ok then I can do part one and get that going, thanks y'all for the assist! [20:57:10] once the changed are merged for helmfile.d they could get applied at any point due to k8s work [20:58:11] uh huh, makes sense [20:59:15] It is really common to have pipelinelib build and store container images that never get used. Most repos build after every merge, but only deploy on some other cadence (once a week, when there are enough things queued up to make it worthwhile, etc) [20:59:39] gtk! [21:00:39] Getting to Continuous Deployment is a dream that is not widely realized here yet [21:02:28] getting to a place where everyone who can deploy knows how to do so for all the things, I'd settle for that :-) [21:11:43] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Replace Swift hosts: deployment-ms-(be0[78]|fe04) (Bullseye deprecation) - https://phabricator.wikimedia.org/T436476#12505858 (10Southparkfan) I have removed the old backend hosts from `profile::swift::storagehosts`... [21:17:30] rats, I am stumped one more time. probably the docs I was looking at aren't current. where do I get the tag for the new image? I looked around in https://integration.wikimedia.org/ci/job/change-propagation-pipeline-test/148/ but don't see anything that looks like 2026-10-29-215614-production for today, in https://docker-registry.wikimedia.org/wikimedia/mediawiki-services-change-propagation/tags/ and I don't see it there either. [21:20:15] for full disclusre I was looking (among other places) at this: https://wikitech.wikimedia.org/wiki/Deployment_pipeline/Tutorial [21:27:13] re-summoning thcipriani because you wrote that doc, mostly [21:33:11] 10Beta-Cluster-Infrastructure, 07Puppet: Cannot rollout Trixie hosts in deployment-prep without rm -rf /var/lib/puppet/ssl - https://phabricator.wikimedia.org/T443373 (10Southparkfan) 03NEW [21:33:51] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 06tools-infrastructure-team, 07Puppet: Cannot rollout Trixie hosts in deployment-prep without rm -rf /var/lib/puppet/ssl - https://phabricator.wikimedia.org/T443373#12506006 (10Southparkfan) [21:50:25] apergos: On the gerrit side, look for the "Post-merge build succeeded." message and the magic numbers you want should be in the CI log there. [21:51:03] in your case I think it is trigger-change-propagation-pipeline-publish jobs that you want. [21:51:36] in https://integration.wikimedia.org/ci/job/change-propagation-pipeline-publish/57/console, the string is "2026-11-02-211032-production" [21:51:38] checking! [21:52:55] yeah I checked the wrong step; I was looking at the result of the gate and submit job :-/ [21:53:06] rewriting my notes now tyvm [21:57:18] the post from pipeline bot hadn't shown up when I was looking :-/ [21:58:06] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance deployment-ms-be07 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:01:56] ^ I'm convinced I hit the delete button in Horizon a long time ago.. [22:02:40] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 06tools-infrastructure-team, 07Puppet: Cannot rollout Trixie hosts in deployment-prep without rm -rf /var/lib/puppet/ssl - https://phabricator.wikimedia.org/T443373#12506104 (10dancy) Thanks for the report @Southparkfan. I'll look into this. [22:02:51] Southparkfan: quite possible. That alert will fire when the instance is dead and the rest of the world hasn't figured out that was on purpose [22:03:58] bd808: Yes, but I'm surprised. Cleaned up PuppetDB as well. [22:04:42] Unless the alert is *not* dependent on an entry in PuppetDB for ms-be07, but is merely firing because ms-be07 is still running [22:04:52] (and is self-reporting the failure) [22:05:28] only taavi knows how that works ;) [22:05:54] I have hit the delete button in Horizon two times now, yet the instances aren't being deleted :) [22:07:39] (03PS1) 10Vaughn Walters: jjb: Remove the AchievementBadges, Cargo and Sanctions daily jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1361573 (https://phabricator.wikimedia.org/T443375) [22:09:46] (03Merged) 10jenkins-bot: jjb: Remove the AchievementBadges, Cargo and Sanctions daily jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1361573 (https://phabricator.wikimedia.org/T443375) (owner: 10Vaughn Walters) [22:18:15] bd808: that alert is based on statistics from the node itself, not from PuppetDB. so it'll fire as long as the instance is both up and visible through the openstack API [22:30:35] thanks taavi [22:40:14] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Replace Swift hosts: deployment-ms-(be0[78]|fe04) (Bullseye deprecation) - https://phabricator.wikimedia.org/T436476#12506284 (10Southparkfan) Tentative plan for replacing `deployment-ms-fe04`: - [] Ensure Puppet ru... [22:51:06] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [23:00:45] (03open) 10catrope: Add a script for checking which security patches can't be backported [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/282 [23:05:40] (03open) 10addshore: Draft: Add remote development environments and background sessions [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/792 [23:06:05] (03update) 10addshore: Draft: Add remote development environments and background sessions [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/792 [23:06:23] (03update) 10addshore: Draft: Add remote development environments and background sessions [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/792 (https://phabricator.wikimedia.org/T443381) [23:08:01] (03open) 10project_16_bot_df05b28c9b5fb77789cdd3cf152eb286: Draft: Migrate CLI logging to slog and standardize output and errors [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/793 [23:09:04] (03update) 10addshore: Draft: Migrate CLI logging to slog and standardize output and errors [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/793 (owner: 10project_16_bot_df05b28c9b5fb77789cdd3cf152eb286) [23:38:06] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance deployment-dancytest02 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:38:06] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance deployment-dancytest02 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources