[07:08:58] Seeing the npm cache corruption issue in a couple of rundoc jobs: https://integration.wikimedia.org/ci/job/mwext-node20-rundoc/15368/console . I'll blow away the cache in a sec [07:12:37] 10Gerrit (Gerrit 3.12): Upgrade Gerrit hosts from Bullseye to Bookworm - https://phabricator.wikimedia.org/T392464#11072890 (10hashar) [07:13:31] 10Gerrit: Switch Gerrit from Java 17 to Java 21 - https://phabricator.wikimedia.org/T392465#11072891 (10hashar) [07:14:02] 10Gerrit (Gerrit 3.11), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Upgrade to Gerrit 3.11 - https://phabricator.wikimedia.org/T379714#11072895 (10hashar) [07:14:13] 10Gerrit (Gerrit 3.12): Upgrade to Gerrit 3.12 - https://phabricator.wikimedia.org/T392448#11072896 (10hashar) [07:14:38] !log integration: integration-castor05: rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwext-node20-rundoc [07:14:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:30:36] 10GitLab (Infrastructure), 06collaboration-services: Troubleshoot GitLab nftables throttling after switchover - https://phabricator.wikimedia.org/T400971#11072912 (10Jelto) Unfortunately one of the Digital Ocean Runner IPs is throttled quite often. I'll test higher thresholds. [07:33:24] (03merge) 10jelto: disable usage_ping [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/77 (https://phabricator.wikimedia.org/T401352) [08:03:30] !log update gitlab-settings on all instances - T401352 [08:03:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:03:32] T401352: Disable GitLab 18.0 data collection - https://phabricator.wikimedia.org/T401352 [08:06:24] 10GitLab (Administration, Settings & Policy), 10gitlab-settings, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Disable GitLab 18.0 data collection - https://phabricator.wikimedia.org/T401352#11072977 (10Jelto) 05Openβ†’03Resolved p:05Triageβ†’03Medium a:03Jelto `usag... [08:21:39] (03CR) 10Hashar: [C:03+2] zuul: [mediawiki/extensions/SocialProfile] Add SpamRegex for phan [integration/config] - 10https://gerrit.wikimedia.org/r/1176525 (owner: 10Jack Phoenix) [08:24:30] (03Merged) 10jenkins-bot: zuul: [mediawiki/extensions/SocialProfile] Add SpamRegex for phan [integration/config] - 10https://gerrit.wikimedia.org/r/1176525 (owner: 10Jack Phoenix) [08:55:48] 10Continuous-Integration-Config, 07ci-test-error: npm cache failure enoent Invalid response body while trying to fetch on Wikibase CI (mwext-node20-rundoc) - https://phabricator.wikimedia.org/T401534#11073063 (10Lucas_Werkmeister_WMDE) Hopefully [fixed](https://sal.toolforge.org/log/vGH7l5gB8tZ8Ohr0MRiB): > <... [09:13:20] (03CR) 10Hashar: [C:03+2] "This has been deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/1176551 (https://phabricator.wikimedia.org/T401451) (owner: 10Ahmon Dancy) [09:15:49] (03Merged) 10jenkins-bot: jjb: Update civicrm to 0.6 [integration/config] - 10https://gerrit.wikimedia.org/r/1176551 (https://phabricator.wikimedia.org/T401451) (owner: 10Ahmon Dancy) [09:29:35] 06Release-Engineering-Team, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: CI build - missing patch & mysqli? - https://phabricator.wikimedia.org/T401451#11073118 (10hashar) The mysqli requirement comes from https://github.com/civicrm/civicrm-core/commit/595f9de96cdfb68f2295715... [09:35:13] (03PS1) 10Hashar: dockerfiles: [civicrm] add php-mysqli [integration/config] - 10https://gerrit.wikimedia.org/r/1177328 (https://phabricator.wikimedia.org/T401451) [09:36:43] (03PS1) 10Hashar: jjb: update civicrm to 0.7 [integration/config] - 10https://gerrit.wikimedia.org/r/1177329 (https://phabricator.wikimedia.org/T401451) [09:42:05] (03CR) 10Hashar: [C:03+2] dockerfiles: [civicrm] add php-mysqli [integration/config] - 10https://gerrit.wikimedia.org/r/1177328 (https://phabricator.wikimedia.org/T401451) (owner: 10Hashar) [09:44:27] (03Merged) 10jenkins-bot: dockerfiles: [civicrm] add php-mysqli [integration/config] - 10https://gerrit.wikimedia.org/r/1177328 (https://phabricator.wikimedia.org/T401451) (owner: 10Hashar) [10:07:05] 06Release-Engineering-Team, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: CI build - missing patch & mysqli? - https://phabricator.wikimedia.org/T401451#11073219 (10hashar) I have added `php-mysqli` to the CI image and update the job to `docker-registry.wikimedia.org/releng/ci... [10:07:18] 10Continuous-Integration-Config, 06Release-Engineering-Team, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build - missing patch & mysqli? - https://phabricator.wikimedia.org/T401451#11073220 (10hashar) [10:07:30] (03CR) 10Hashar: [C:03+2] jjb: update civicrm to 0.7 [integration/config] - 10https://gerrit.wikimedia.org/r/1177329 (https://phabricator.wikimedia.org/T401451) (owner: 10Hashar) [10:09:47] (03Merged) 10jenkins-bot: jjb: update civicrm to 0.7 [integration/config] - 10https://gerrit.wikimedia.org/r/1177329 (https://phabricator.wikimedia.org/T401451) (owner: 10Hashar) [10:10:03] (03CR) 10Hashar: [C:03+2] quibble-bullseye-php81-coverage: Remove unused mwservice-phpunit-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/1176722 (owner: 10Krinkle) [11:37:54] 10Continuous-Integration-Config, 07ci-test-error: npm cache failure enoent Invalid response body while trying to fetch on Wikibase CI (mwext-node20-rundoc) - https://phabricator.wikimedia.org/T401534#11073378 (10Umherirrender) 05Openβ†’03Resolved [11:55:33] 10Phabricator, 10Observability-Logging: Logstash Visualization "Events over time" got overwritten - https://phabricator.wikimedia.org/T401579 (10Aklapper) 03NEW [11:59:10] 10Phabricator, 06Language and Product Localization, 10Observability-Logging: Generic Logstash Visualization "Events over time" got overwritten with "ContentTranslation" - https://phabricator.wikimedia.org/T401579#11073420 (10Aklapper) [12:08:01] (03CR) 10Hashar: [C:03+2] build: Upgrade eslint-config-wikimedia to 0.31.0 [integration/docroot] - 10https://gerrit.wikimedia.org/r/1176751 (owner: 10Jforrester) [12:08:41] (03Merged) 10jenkins-bot: build: Upgrade eslint-config-wikimedia to 0.31.0 [integration/docroot] - 10https://gerrit.wikimedia.org/r/1176751 (owner: 10Jforrester) [12:38:17] (03CR) 10Hashar: [C:03+2] build: upgrade QUnit [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1175475 (owner: 10Hashar) [12:39:13] (03Merged) 10jenkins-bot: build: upgrade QUnit [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1175475 (owner: 10Hashar) [12:49:12] Hi! I am trying to run a release pipeline https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/jobs/582996 (discolytics, a bunch of pyhton scripts to be used via AirFlow/Spark). But that fails due to a 404 when trying to fetch debian packages. Is that a known issues? [13:02:46] (03CR) 10Hashar: [C:04-1] jjb: create specific job for Wikilambda daily tests (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1176507 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [13:04:36] pfischer: in general there have been several issues related to bullseye-backports in here recently, I’m sure someone who knows more will chime in soon [13:04:46] https://phabricator.wikimedia.org/T383557 is one of the related tasks I’m aware of [13:22:50] (03PS1) 10Hashar: jjb: stop triggering Wikilambda end-to-end daily [integration/config] - 10https://gerrit.wikimedia.org/r/1177387 (https://phabricator.wikimedia.org/T388056) [13:22:52] (03PS1) 10Hashar: jjb: wikilambda-catalyst-end-to-end does not use Castor [integration/config] - 10https://gerrit.wikimedia.org/r/1177388 (https://phabricator.wikimedia.org/T388056) [13:31:34] 10Phabricator, 06Language and Product Localization, 10Observability-Logging: Generic Logstash Visualization "Events over time" got overwritten with "ContentTranslation" - https://phabricator.wikimedia.org/T401579#11073658 (10colewhite) 05Openβ†’03Resolved a:03colewhite Restored the panel from backup.... [13:37:08] (03PS1) 10Hashar: dockerfiles: [catalyst] allow setting environment name [integration/config] - 10https://gerrit.wikimedia.org/r/1177393 (https://phabricator.wikimedia.org/T388056) [13:37:10] (03PS1) 10Hashar: jjb: update wikilambda-catalyst-end-to-end image [integration/config] - 10https://gerrit.wikimedia.org/r/1177394 (https://phabricator.wikimedia.org/T388056) [13:45:15] (03PS3) 10Hashar: jjb: create specific job for Wikilambda daily tests [integration/config] - 10https://gerrit.wikimedia.org/r/1176507 (https://phabricator.wikimedia.org/T388056) [13:50:01] (03CR) 10Hashar: jjb: create specific job for Wikilambda daily tests (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1176507 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [14:09:23] 10Phabricator (phabricator-next), 10Wikimedia-Phabricator-Extensions: Update Phabricator translations to 2024.35 release - https://phabricator.wikimedia.org/T399604#11073896 (10Pppery) [14:17:06] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services, 06serviceops: Standardize Debian package builds on GitLab CI - https://phabricator.wikimedia.org/T304491#11073939 (10fnegri) I just found this task while looking for ways to build .deb packages from Gitlab. My understanding... [14:20:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:20:36] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T401596 (10wmcs-alerts) 03NEW [14:40:36] (03PS2) 10Jforrester: jjb: [wikilambda-catalyst-end-to-end] Stop triggering daily [integration/config] - 10https://gerrit.wikimedia.org/r/1177387 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [14:40:36] (03PS2) 10Jforrester: jjb: [wikilambda-catalyst-end-to-end] Don't set up with Castor, unused [integration/config] - 10https://gerrit.wikimedia.org/r/1177388 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [14:40:37] (03PS2) 10Jforrester: dockerfiles: [catalyst] Allow setting environment name [integration/config] - 10https://gerrit.wikimedia.org/r/1177393 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [14:40:38] (03PS2) 10Jforrester: jjb: [wikilambda-catalyst-end-to-end] Update to new Catalyst image [integration/config] - 10https://gerrit.wikimedia.org/r/1177394 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [14:40:40] (03PS4) 10Jforrester: jjb: Create specific job for Wikilambda daily tests [integration/config] - 10https://gerrit.wikimedia.org/r/1176507 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [14:40:55] hashar: LGTM. Should I deploy? Did you want to keep working on them? [15:00:07] (03CR) 10Jforrester: [C:03+2] jjb: [wikilambda-catalyst-end-to-end] Stop triggering daily [integration/config] - 10https://gerrit.wikimedia.org/r/1177387 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [15:00:10] (03CR) 10Jforrester: [C:03+2] jjb: [wikilambda-catalyst-end-to-end] Don't set up with Castor, unused [integration/config] - 10https://gerrit.wikimedia.org/r/1177388 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [15:02:00] (03Merged) 10jenkins-bot: jjb: [wikilambda-catalyst-end-to-end] Stop triggering daily [integration/config] - 10https://gerrit.wikimedia.org/r/1177387 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [15:02:52] (03Merged) 10jenkins-bot: jjb: [wikilambda-catalyst-end-to-end] Don't set up with Castor, unused [integration/config] - 10https://gerrit.wikimedia.org/r/1177388 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [15:06:57] 10Continuous-Integration-Config, 06Release-Engineering-Team, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build - missing patch & mysqli? - https://phabricator.wikimedia.org/T401451#11074229 (10dancy) Thanks @hashar! [15:13:39] (03PS1) 10Jforrester: Docker: Provide ci-trixie [integration/config] - 10https://gerrit.wikimedia.org/r/1177418 (https://phabricator.wikimedia.org/T400931) [15:14:37] (03CR) 10Jforrester: [C:03+2] Docker: Provide ci-trixie [integration/config] - 10https://gerrit.wikimedia.org/r/1177418 (https://phabricator.wikimedia.org/T400931) (owner: 10Jforrester) [15:15:06] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Provide ci-trixie base image, once Wikimedia trixie base Docker image exists - https://phabricator.wikimedia.org/T400931#11074275 (10Jdforrester-WMF) 05Openβ†’03Resolved a:03Jdforrester-WMF [15:16:01] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Migrate all CI jobs from Bookworm to Trixie or later and drop Bookworm testing support - https://phabricator.wikimedia.org/T397049#11074291 (10Jdforrester-WMF) p:05Triageβ†’03Low This is not a current priority. Later, it will become so. [15:16:18] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [YOUR DEVELOPER ACCOUNT USERNAME HERE] - https://phabricator.wikimedia.org/T401601 (10Hearvox) 03NEW [15:16:39] (03Merged) 10jenkins-bot: Docker: Provide ci-trixie [integration/config] - 10https://gerrit.wikimedia.org/r/1177418 (https://phabricator.wikimedia.org/T400931) (owner: 10Jforrester) [15:17:29] !log Docker: Provide ci-trixie, for T400931 [15:17:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:17:32] T400931: Provide ci-trixie base image, once Wikimedia trixie base Docker image exists - https://phabricator.wikimedia.org/T400931 [15:18:32] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T401596#11074340 (10bd808) 05Openβ†’03In progress p:05Triageβ†’03High a:03bd808 The cherrypick of https://gerrit.... [15:20:02] (03CR) 10Jforrester: "Ack. Happy to help if needed!" [integration/config] - 10https://gerrit.wikimedia.org/r/1176721 (https://phabricator.wikimedia.org/T395470) (owner: 10Krinkle) [15:22:11] (03PS1) 10Jforrester: fab: Add trixie to seed list of images [integration/config] - 10https://gerrit.wikimedia.org/r/1177423 [15:22:43] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [YOUR DEVELOPER ACCOUNT USERNAME HERE] - https://phabricator.wikimedia.org/T401601#11074362 (10Hearvox) We are looking for a place to store our repos (user script, dataset, API, etc.) for this project: https://en.... [15:23:23] 10GitLab (Infrastructure), 06collaboration-services: Troubleshoot GitLab nftables throttling after switchover - https://phabricator.wikimedia.org/T400971#11074365 (10Milimetric) I know of a few examples of gitlab packaging jobs failing but here's one that I tried to run and know more about: https://gitlab.wiki... [15:23:23] (03CR) 10Jforrester: [C:03+2] fab: Add trixie to seed list of images [integration/config] - 10https://gerrit.wikimedia.org/r/1177423 (owner: 10Jforrester) [15:24:09] (03CR) 10Jforrester: [C:03+2] dockerfiles: [catalyst] Allow setting environment name [integration/config] - 10https://gerrit.wikimedia.org/r/1177393 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [15:24:52] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T401596#11074367 (10bd808) https://gerrit.wikimedia.org/r/c/operations/puppet/+/1143602/5..6 [15:25:00] (03Merged) 10jenkins-bot: fab: Add trixie to seed list of images [integration/config] - 10https://gerrit.wikimedia.org/r/1177423 (owner: 10Jforrester) [15:25:44] (03Merged) 10jenkins-bot: dockerfiles: [catalyst] Allow setting environment name [integration/config] - 10https://gerrit.wikimedia.org/r/1177393 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [15:26:11] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services, 06serviceops: Standardize Debian package builds on GitLab CI - https://phabricator.wikimedia.org/T304491#11074389 (10MatthewVernon) A couple of thoughts on your desiderata: Building on all branches would require, I think, a... [15:26:28] !log Docker: [catalyst] Allow setting environment name, for T388056 [15:26:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:26:31] T388056: Run Wikilambda daily selenium tests against Catalyst - https://phabricator.wikimedia.org/T388056 [15:29:53] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Phorge setup check caching is misbehaving, leading to many duck-sound=quack requests - https://phabricator.wikimedia.org/T401157#11074409 (10Jelto) a:03Dzahn [15:30:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [15:30:36] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T401605 (10wmcs-alerts) 03NEW [15:33:36] (03CR) 10Jforrester: [C:03+2] jjb: [wikilambda-catalyst-end-to-end] Update to new Catalyst image [integration/config] - 10https://gerrit.wikimedia.org/r/1177394 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [15:35:51] (03Merged) 10jenkins-bot: jjb: [wikilambda-catalyst-end-to-end] Update to new Catalyst image [integration/config] - 10https://gerrit.wikimedia.org/r/1177394 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [15:36:35] (03PS5) 10Jforrester: jjb: Create specific job for Wikilambda daily tests [integration/config] - 10https://gerrit.wikimedia.org/r/1176507 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [15:36:35] (03CR) 10Jforrester: jjb: Create specific job for Wikilambda daily tests (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1176507 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [15:37:29] (03PS6) 10Jforrester: jjb: Create specific job for Wikilambda daily tests [integration/config] - 10https://gerrit.wikimedia.org/r/1176507 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [15:38:03] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T401596#11074440 (10bd808) [15:38:06] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T401605#11074442 (10bd808) β†’14Duplicate dup:03T401596 [15:38:12] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T401596#11074443 (10bd808) 05In progressβ†’03Resolved [15:38:14] (03CR) 10Jforrester: "Deployed: https://integration.wikimedia.org/ci/job/wikilambda-catalyst-end-to-end-daily/" [integration/config] - 10https://gerrit.wikimedia.org/r/1176507 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [15:51:06] (03CR) 10Jforrester: [C:03+2] jjb: Create specific job for Wikilambda daily tests [integration/config] - 10https://gerrit.wikimedia.org/r/1176507 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [15:52:46] (03Merged) 10jenkins-bot: jjb: Create specific job for Wikilambda daily tests [integration/config] - 10https://gerrit.wikimedia.org/r/1176507 (https://phabricator.wikimedia.org/T388056) (owner: 10Hashar) [15:54:41] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/WikiLambda] Add SiteMatrix phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1177429 [16:06:18] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/WikiLambda] Add SiteMatrix phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1177429 (owner: 10Jforrester) [16:07:41] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikiLambda] Add SiteMatrix phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1177429 (owner: 10Jforrester) [16:08:02] !log Zuul: [mediawiki/extensions/WikiLambda] Add SiteMatrix phan dep [16:08:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:09:08] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 07Essential-Work: Jenkins fails to connect to node on releases2003 - https://phabricator.wikimedia.org/T401180#11074600 (10thcipriani) [16:10:32] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 06collaboration-services, 07Essential-Work: dpkg error when deploying Jenkins to releases2003.codfw.wmnet - https://phabricator.wikimedia.org/T400645#11074617 (10thcipriani) [16:11:02] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T396374#11074621 (10thcipriani) [16:12:02] 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for Pols12 - https://phabricator.wikimedia.org/T401444#11074625 (10thcipriani) [16:12:09] 10Phabricator, 10Release-Engineering-Team (Doing 😎), 07Essential-Work: Rename Phabricator account username β€˜R4356th’ to β€˜Redmin’ - https://phabricator.wikimedia.org/T401306#11074626 (10thcipriani) [16:12:31] 10Phabricator (2025-08-05), 06Release-Engineering-Team, 06collaboration-services, 07Essential-Work: Deploy Phabricator/Phorge 2025-08-05 - https://phabricator.wikimedia.org/T401213#11074627 (10thcipriani) [16:12:40] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 07Essential-Work: Puppet failures on integration-agent instances - unable to connect to integration-puppetserver-01 - https://phabricator.wikimedia.org/T401123#11074628 (10thcipriani) [16:13:02] 10Phabricator, 10Release-Engineering-Team (Doing 😎), 06Diffusion-Repository-Administrators, 07Essential-Work: Disable or delete some failing mirrored Diffusion repositories - https://phabricator.wikimedia.org/T401373#11074629 (10thcipriani) [16:13:07] 10Phabricator, 10Release-Engineering-Team (Doing 😎), 07Essential-Work: FR Herald rule H455 failed - https://phabricator.wikimedia.org/T399987#11074631 (10thcipriani) [16:13:23] 10Phabricator, 10Release-Engineering-Team (Doing 😎), 07Essential-Work: Remove remnants of Phatality ID in Phabricator ("error.id" custom field) - https://phabricator.wikimedia.org/T398397#11074633 (10thcipriani) [16:38:25] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Hearvox - https://phabricator.wikimedia.org/T401601#11074761 (10Anoop) [16:53:31] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services, 06serviceops: Standardize Debian package builds on GitLab CI - https://phabricator.wikimedia.org/T304491#11074849 (10fnegri) @MatthewVernon thanks for your quick reply! > Building on all branches would require, I think, a s... [17:09:35] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 07Essential-Work: Puppetize systemd unit for zuul's nodepool - https://phabricator.wikimedia.org/T401614 (10thcipriani) 03NEW [17:10:13] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 07Essential-Work: Puppetize systemd unit for zuul's nodepool - https://phabricator.wikimedia.org/T401614#11074928 (10thcipriani) a:03bd808 Assigning to @bd808 per #together t... [17:12:11] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 07Essential-Work: Puppetize systemd unit for zuul's nodepool - https://phabricator.wikimedia.org/T401614#11074937 (10bd808) [17:12:18] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 07Essential-Work: Puppetize systemd unit for zuul's nodepool - https://phabricator.wikimedia.org/T401614#11074940 (10bd808) p:05Triageβ†’03Medium [17:54:31] (03open) 10bd808: tofu: install v1.10.5 from github artifacts [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/48 [18:02:24] (03open) 10dancy: build_image_incr.py: Add label referencing parent image [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/202 [18:02:27] (03update) 10dancy: build_image_incr.py: Add label referencing parent image [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/202 [18:04:33] (03update) 10dancy: build_image_incr.py: Add label referencing parent image [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/202 [19:13:13] 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for Hearvox - https://phabricator.wikimedia.org/T401601#11075314 (10thcipriani) 05Openβ†’03Resolved a:03thcipriani Hi @Hearvox I approved your account just now and you should be able to login. [19:20:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [19:20:37] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T401624 (10wmcs-alerts) 03NEW [19:25:32] (03open) 10dancy: kubernetes.py: prune_local_images, don't remove parent images [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/967 [19:25:36] (03update) 10dancy: kubernetes.py: prune_local_images, don't remove parent images [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/967 [19:46:13] (03approved) 10dduvall: build_image_incr.py: Add label referencing parent image [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/202 (owner: 10dancy) [19:48:21] 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for Hearvox - https://phabricator.wikimedia.org/T401601#11075515 (10Hearvox) I’m in. Much thanks, BG -- https://hearingvoices.com/ https://iffy.news/ [19:57:52] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T401624#11075539 (10bd808) 05Openβ†’03In progress p:05Triageβ†’03High a:03bd808 Caused by https://gerrit.wikimedi... [20:08:41] (03merge) 10bd808: tofu: install v1.10.5 from github artifacts [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/48 [20:10:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [20:10:33] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T401632 (10wmcs-alerts) 03NEW [20:11:45] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T401624#11075614 (10bd808) [20:11:52] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T401632#11075616 (10bd808) β†’14Duplicate dup:03T401624 [20:13:06] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T401624#11075621 (10bd808) 05In progressβ†’03Resolved https://gerrit.wikimedia.org/r/c/operations/puppet/+/114360... [21:23:53] 10Beta-Cluster-Infrastructure, 07Wikimedia-production-error: DBQueryError "The MariaDB server is running with the --read-only option" fails MainStash in Beta Cluster - https://phabricator.wikimedia.org/T401227#11075882 (10Ladsgroup) DBAs don't own databases in beta cluster. I stay subscribed in case I could pr... [22:24:14] dancy: tested scap update-interwiki-cache now, it does work. Thanks. [23:54:36] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.45.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T396375#11076279 (10SecurityPatchBot)