[00:25:46] 10Release-Engineering-Team (Doing 😎), 05Release, 05Train Deployments: 1.44.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T382362#10446566 (10thcipriani) 05Openβ†’03Resolved [00:42:27] (03PS2) 10Subramanya Sastry: Add retry scripts to simplify retrying failures [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1109503 (https://phabricator.wikimedia.org/T383255) [00:42:28] (03PS1) 10Subramanya Sastry: Poor man's puppetization of visual diffing vm and services [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1109519 (https://phabricator.wikimedia.org/T295907) [10:29:14] (03PS3) 10Dreamy Jazz: Zuul: [mediawiki/extensions/CheckUser] Add IPInfo dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1109456 (https://phabricator.wikimedia.org/T380954) [10:30:25] (03PS4) 10Dreamy Jazz: Zuul: [mediawiki/extensions/CheckUser] Add IPInfo dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1109456 (https://phabricator.wikimedia.org/T380954) [10:30:36] (03PS5) 10Dreamy Jazz: Zuul: [mediawiki/extensions/CheckUser] Add IPInfo dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1109456 (https://phabricator.wikimedia.org/T380954) [10:40:23] 10GitLab (Pipeline Services Migration🐀), 06collaboration-services, 10Wikidata, 10Wikidata Query UI, and 3 others: move commons-query.wikimedia.org and query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10447519 (10Gehel) [10:55:44] GitLab needs a short maintenance break at 12:00 UTC (in one hour) [10:56:03] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing 😎): Zuul/Jenkins: Investigate caching of build results for MediaWiki testsuite jobs - https://phabricator.wikimedia.org/T383243#10447672 (10hashar) The `tree` should do yes, nice finding! From the doc Dan pointed to, one can get the sh... [10:58:19] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing 😎): Jenkins: In the integration project in WMCS, test out larger workers for CI - https://phabricator.wikimedia.org/T383254#10447693 (10ArthurTaylor) [11:01:59] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing 😎): Jenkins: In the integration project in WMCS, test out larger workers for CI - https://phabricator.wikimedia.org/T383254#10447707 (10ArthurTaylor) @brennen yeah - that should be pretty easy. I've created T383389 to track this. In case someon... [11:14:05] (03CR) 10Hashar: "+1!" [integration/config] - 10https://gerrit.wikimedia.org/r/1109440 (owner: 10Jforrester) [11:15:35] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/CheckUser] Add IPInfo dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1109456 (https://phabricator.wikimedia.org/T380954) (owner: 10Dreamy Jazz) [11:16:54] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/CheckUser] Add IPInfo dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1109456 (https://phabricator.wikimedia.org/T380954) (owner: 10Dreamy Jazz) [11:17:34] Dreamy_Jazz: I have deployed your CI config change! [11:17:46] Thanks for the quick review! [11:58:48] (03CR) 10Jforrester: [C:03+2] fab: Add bookworm to seed list of images (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1109440 (owner: 10Jforrester) [12:02:43] (03open) 10marostegui: deployments-calendar.yaml: Update list of nicks [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/143 [12:05:01] (03update) 10marostegui: deployments-calendar.yaml: Update list of nicks [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/143 [13:31:35] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [13:41:35] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [14:28:13] 06Release-Engineering-Team, 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, and 3 others: Generate a dumps-enabled mediawiki image - https://phabricator.wikimedia.org/T381473#10448251 (10Gehel) [14:30:03] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 06Java-Scala-Standardization: Add CI_RELEASE_TOKEN secret for {name}-maven-release jobs in jenkins - https://phabricator.wikimedia.org/T379203#10448278 (10Gehel) [14:36:31] (03PS3) 10Subramanya Sastry: Add retry scripts to simplify retrying failures [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1109503 (https://phabricator.wikimedia.org/T383255) [14:36:37] 10Fresnel: Automate finding forced layouts - https://phabricator.wikimedia.org/T219495#10448369 (10Peter) [14:50:08] 10MediaWiki-Releasing, 07Documentation, 05MW-1.43-notes, 05MW-1.43-release: Finish up RELEASE-NOTES for 1.43.0 - https://phabricator.wikimedia.org/T372325#10448392 (10Jdforrester-WMF) 05Openβ†’03Resolved [14:51:12] 10MediaWiki-Releasing, 05MW-1.43-release: Write and send release announcement for 1.43.0 - https://phabricator.wikimedia.org/T372328#10448396 (10Jdforrester-WMF) 05Openβ†’03Resolved a:03MSantos Sent on 2024-12-21: https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/UYZUOQE... [14:51:30] 10MediaWiki-Releasing, 05MW-1.43-notes, 05MW-1.43-release, 13Patch-For-Review, 05Release: Release MediaWiki 1.43.0 - https://phabricator.wikimedia.org/T372295#10448400 (10Jdforrester-WMF) 05Openβ†’03Resolved [15:18:56] 06Release-Engineering-Team, 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, and 4 others: Generate a dumps-enabled mediawiki image - https://phabricator.wikimedia.org/T381473#10448513 (10Gehel) [15:26:35] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 06Java-Scala-Standardization: Add CI_RELEASE_TOKEN secret for {name}-maven-release jobs in jenkins - https://phabricator.wikimedia.org/T379203#10448566 (10Gehel) [15:35:58] (03PS4) 10Subramanya Sastry: Add retry scripts to simplify retrying failures [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1109503 (https://phabricator.wikimedia.org/T383255) [15:39:11] 10Beta-Cluster-Infrastructure: Multiple kafka Cergen certs expired in beta cluster - https://phabricator.wikimedia.org/T383096#10448715 (10elukey) I've added TLS support to deployment kafka main, so kafka mirror should now work correctly. [15:40:04] (03open) 10btullis: Add data-engineering/refinery to the trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/104 (https://phabricator.wikimedia.org/T383417) [15:45:52] (03update) 10dancy: gitlab-runner: bump image version to alpine-v17.5.5 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/416 (https://phabricator.wikimedia.org/T383263) (owner: 10jelto) [15:46:30] (03merge) 10dancy: gitlab-runner: bump image version to alpine-v17.5.5 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/416 (https://phabricator.wikimedia.org/T383263) (owner: 10jelto) [15:48:05] 10Beta-Cluster-Infrastructure: Multiple kafka Cergen certs expired in beta cluster - https://phabricator.wikimedia.org/T383096#10448795 (10elukey) ` elukey@deployment-puppetserver-1:~$ sudo puppetserver ca clean --certname kafka_jumbo-deployment-prep_broker Certificate for kafka_jumbo-deployment-prep_broker has... [15:49:24] !log Updating gitlab-cloud-runners (staging) to v17.5.5 [15:49:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:53:15] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap: "scap deploy"'s config-deploy should check for broken symlinks - https://phabricator.wikimedia.org/T342162#10448825 (10Gehel) [15:53:27] 14Release-Engineering-Team (Now this 🫠), 10Scap, 06Data-Engineering, 07git-lfs: analytics/refinery: Stop using git-fat - https://phabricator.wikimedia.org/T328472#10448826 (10Gehel) [15:55:28] !log Updating gitlab-cloud-runners (prod) to v17.5.5 (T383263) [15:55:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:55:59] 10Beta-Cluster-Infrastructure: Multiple kafka Cergen certs expired in beta cluster - https://phabricator.wikimedia.org/T383096#10448843 (10bd808) 05Openβ†’03Resolved a:03elukey The dashboard at https://prometheus-alerts.wmcloud.org/?q=project%3Ddeployment-prep&q=alertname%3DPuppetCertificateAboutToExpire... [16:04:21] (03approved) 10dancy: Add data-engineering/refinery to the trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/104 (https://phabricator.wikimedia.org/T383417) (owner: 10btullis) [16:13:55] (03open) 10dancy: .gitlab.ci.yml: Don't try to run trusted jobs on forks [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/105 [16:13:58] (03update) 10dancy: .gitlab.ci.yml: Don't try to run trusted jobs on forks [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/105 [16:14:55] (03update) 10dancy: .gitlab.ci.yml: Don't try to run trusted jobs on forks [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/105 [16:16:40] 10Release-Engineering-Team (Radar), 06Java-Scala-Standardization, 03Discovery-Search (Current work): Validate CI integration so that Ci can release Maven artifacts on user's demand - https://phabricator.wikimedia.org/T367403#10448943 (10Gehel) [16:20:58] (03open) 10btullis: Add data-engineering/refinery to the trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/106 (https://phabricator.wikimedia.org/T383417) [16:22:22] (03merge) 10dancy: Add data-engineering/refinery to the trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/106 (https://phabricator.wikimedia.org/T383417) (owner: 10btullis) [16:24:59] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Upstream: Archiva release repository yields 404 for release repository breaking builds - https://phabricator.wikimedia.org/T373352#10449016 (10Gehel) [16:25:37] (03update) 10dancy: .gitlab.ci.yml: Don't try to run trusted jobs on forks [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/105 [16:35:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing 😎): Zuul/Jenkins: Investigate caching of build results for MediaWiki testsuite jobs - https://phabricator.wikimedia.org/T383243#10449117 (10dduvall) >>! In T383243#10447672, @hashar wrote: > The `tree` should do yes, nice finding! > >... [16:42:19] 06Release-Engineering-Team, 06Data-Engineering, 03Discovery-Search (Current work), 13Patch-For-Review: SonarQube build are failing with Java 11 - https://phabricator.wikimedia.org/T355122#10449176 (10Gehel) [17:04:01] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10[DEPRECATED] wdwb-tech, 10API Platform, and 15 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953#10449301 (10Gehel) [17:08:46] (03open) 10dancy: spiderpig: CAS auth integration [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/625 [17:08:49] (03update) 10dancy: spiderpig: CAS auth integration [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/625 [17:16:03] (03update) 10dancy: spiderpig: CAS auth integration [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/625 [17:21:43] (03update) 10dancy: spiderpig: CAS auth integration [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/625 [19:03:24] 10Phabricator: Experiencing Phabricator 429 errors - https://phabricator.wikimedia.org/T383435 (10Dominicbm) 03NEW [19:03:55] 10Phabricator: Experiencing Phabricator 429 errors - https://phabricator.wikimedia.org/T383435#10449563 (10Dominicbm) [19:30:04] (03update) 10dancy: mods for deploy02 [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/93 [19:33:26] (03open) 10dancy: port forward 8001 to 8000 in deploy container for spiderpig testing [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/119 [19:33:29] (03update) 10dancy: port forward 8001 to 8000 in deploy container for spiderpig testing [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/119 [19:34:37] (03merge) 10dancy: port forward 8001 to 8000 in deploy container for spiderpig testing [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/119 [19:35:14] (03open) 10dancy: add ldap and auth containers [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/120 [19:35:17] (03update) 10dancy: add ldap and auth containers [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/120 [19:48:34] (03update) 10dancy: add ldap and auth containers [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/120 [19:48:37] (03update) 10dancy: add ldap and auth containers [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/120 [19:50:01] (03update) 10dancy: Add LDAP and CAS services [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/120 [19:50:04] (03update) 10dancy: Add LDAP and CAS services [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/120 [19:58:49] (03update) 10dancy: Add LDAP and CAS services [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/120 [20:00:36] (03update) 10dancy: Add LDAP and CAS services [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/120