[01:21:23] 10Continuous-Integration-Config, 10MediaWiki-Releasing: Have at least one job running REL1_XX with its appropriate vendor repo - https://phabricator.wikimedia.org/T295175#10933406 (10Bugreporter2) [06:20:45] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Make mw-experimental production ready - https://phabricator.wikimedia.org/T396767#10933526 (10jijiki) [06:54:42] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 06DBA: Prepare a database test for m3 - https://phabricator.wikimedia.org/T390034#10933552 (10Marostegui) @brennen I will remove them on Tuesday EU morning, so you'd still have a bit more days. [07:15:33] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Provide an mwdebug functionality on kubernetes (mw-experimental) - https://phabricator.wikimedia.org/T276994#10933568 (10jijiki) [07:15:51] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Provide an mwdebug functionality on kubernetes (mw-experimental) - https://phabricator.wikimedia.org/T276994#10933569 (10jijiki) [07:17:01] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops: mwdebug: people in the "deployment" group should be able to launch 'experimental' instances for testing purposes - https://phabricator.wikimedia.org/T324003#10933570 (10jijiki) [07:17:03] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Provide an mwdebug functionality on kubernetes (mw-experimental) - https://phabricator.wikimedia.org/T276994#10933571 (10jijiki) [07:17:24] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops: mwdebug: people in the "deployment" group should be able to launch 'experimental' instances for testing purposes - https://phabricator.wikimedia.org/T324003#10933576 (10jijiki) [07:17:27] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Provide an mwdebug functionality on kubernetes (mw-experimental) - https://phabricator.wikimedia.org/T276994#10933577 (10jijiki) [07:18:32] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Provide an mwdebug functionality on kubernetes (mw-experimental) - https://phabricator.wikimedia.org/T276994#10933582 (10jijiki) **Update**: this service is live and ready for testing on **eqiad **only: [[ http... [07:18:43] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Make mw-experimental production ready - https://phabricator.wikimedia.org/T396767#10933584 (10jijiki) **Update**: this service is live and ready for testing on **eqiad **only: [[ https://wikitech.wikimedia.org/w... [07:20:31] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Provide an mwdebug functionality on kubernetes (mw-experimental) - https://phabricator.wikimedia.org/T276994#10933598 (10jijiki) [07:21:33] 06Release-Engineering-Team, 06MediaWiki-Engineering, 06serviceops: Depricate mwdebugXXXX hosts - https://phabricator.wikimedia.org/T397498#10933600 (10jijiki) [07:26:57] 10Phabricator (Upstream), 10Release-Engineering-Team (Doing 😎), 07Performance Issue, 07Upstream: Ferret Fulltext search queries all apps' databases (also those of uninstalled apps) - https://phabricator.wikimedia.org/T397499 (10Aklapper) 03NEW p:05Triage→03Low [07:28:17] paladox: congratulations on getting rid of Polymer. We do not use the zuul-results-summary anymore, I have long converted that to a vanilla JavaScript which relies on the Gerrit Checks API [07:29:16] and beside the Checks API, the vanilla JavaScript code uses the plugin API. I expect it to still be available? :) https://gerrit.wikimedia.org/r/Documentation/pg-plugin-dev.html [07:30:51] 10Phabricator, 10Server-side-upload-request: Consider adding a global Herald rule to comment on #Server-side-upload-request tickets - https://phabricator.wikimedia.org/T397024#10933623 (10A_smart_kitten) @Aklapper slight side note — assuming that PR goes through, please could you unlock the 'tags' field on [[h... [08:09:02] 06Release-Engineering-Team, 06MediaWiki-Engineering, 06serviceops: Deprecate mwdebugXXXX hosts - https://phabricator.wikimedia.org/T397498#10933664 (10Aklapper) [08:20:13] 10Beta-Cluster-Infrastructure, 07Puppet: Puppet failing on deployment-cirrussearch{12,13,14}.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T393924#10933693 (10Gehel) [08:29:50] 10Phabricator, 10Release-Engineering-Team (Doing 😎): Typos in Herald message for good first tasks tag - https://phabricator.wikimedia.org/T397487#10933741 (10Aklapper) 05Open→03Resolved a:03Aklapper Thanks for spotting those! Corrected. [08:49:56] hashar: I haven't got rid of it it yet. [08:50:01] I'm progressing towards it [08:50:19] hashar: I saw two+ of your plugins use Polymer [08:51:22] one sorry! https://github.com/wikimedia/operations-software-gerrit/blob/deploy/wmf/stable-3.10/plugins/wm-pcc.js [08:51:24] I'd rather keep them to use vanilla JavaScript, we don't really use TypeScript beside static analysis [08:52:49] I am not even sure that code requires Polymer strictly [08:52:57] I must have copy pasted from somewhere else :) [09:34:36] donc en gros je me coltine toutes les ITCS [09:34:53] tous les bugs/reviews sur les cinq dernières années [09:35:16] et je vais fouiller pour avoir des points de comparaison. Bref y a du taff :b [09:38:53] 06Release-Engineering-Team, 10Scap (SpiderPig 🕸️): SpiderPig session / login broken - https://phabricator.wikimedia.org/T397457#10933964 (10Lucas_Werkmeister_WMDE) Looks like it unbroke itself now (I had to enter a new OTP but the prompt was no longer broken and now I’m also properly logged in). [09:43:42] 10Scap (SpiderPig 🕸️): Don’t obscure the typed OTP in SpiderPig - https://phabricator.wikimedia.org/T397505 (10Lucas_Werkmeister_WMDE) 03NEW [10:11:30] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:11:45] 10Scap (SpiderPig 🕸️): Don’t obscure the typed OTP in SpiderPig - https://phabricator.wikimedia.org/T397505#10934323 (10A_smart_kitten) [13:12:28] I've converted the zuul plugin you have deployed to lit https://gerrit-review.googlesource.com/c/plugins/zuul/+/485641 hashar [13:12:49] seems to work: https://imgur.com/a/lFyE75G (I faked data) [13:18:57] 10GitLab (Infrastructure), 10Cloud-VPS, 06collaboration-services, 10cloud-services-team (FY2024/2025-Q3-Q4): Volume is stuck to deleted instance in devtools project - https://phabricator.wikimedia.org/T396739#10934468 (10Andrew) I'm still wrestling with gitlab-prod-backup but in the meantime I have duplica... [14:42:50] 10Scap (SpiderPig 🕸️): Don’t obscure the typed OTP in SpiderPig - https://phabricator.wikimedia.org/T397505#10934713 (10A_smart_kitten) [14:43:37] 10Scap (SpiderPig 🕸️): Don’t obscure the typed OTP in SpiderPig - https://phabricator.wikimedia.org/T397505#10934727 (10A_smart_kitten) (apologies @lucas_werkmeister_wmde, I evidently got confused between the two extensions…) [15:09:29] Converted lfs to lit with https://gerrit-review.googlesource.com/c/plugins/lfs/+/485662 [15:24:07] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: Buildkit v0.23.1 released - https://phabricator.wikimedia.org/T397204#10934864 (10dancy) [15:24:23] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cache-upload08 on project deployment-prep - https://phabricator.wikimedia.org/T397424#10934867 (10bd808) `lang=shell-session,counterexample bd808@deployment-cache-text08:~$ sudo -i puppet agent -tv Info: Using environment 'production... [15:26:16] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cache-upload08 on project deployment-prep - https://phabricator.wikimedia.org/T397424#10934871 (10bd808) https://gerrit.wikimedia.org/r/c/operations/puppet/+/1161397 looks to be the cause. [15:28:16] (03approved) 10dancy: image: Simplify remote context handling for bake [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/134 (owner: 10dduvall) [15:31:37] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cache-upload08 on project deployment-prep - https://phabricator.wikimedia.org/T397424#10934889 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/6962afced8ded8cff6b06bd552947a1589287600%5E%21/#F0 ` diff... [15:35:10] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cache-upload08 on project deployment-prep - https://phabricator.wikimedia.org/T397424#10934898 (10bd808) 05Open→03Resolved a:03bd808 This is one that would not have been fixed by Cloud VPS puppet applying role hiera setting... [15:36:10] (03open) 10dancy: LoginPage.vue: Don’t obscure the typed OTP in SpiderPig [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/854 (https://phabricator.wikimedia.org/T397505) [15:36:10] (03update) 10dancy: LoginPage.vue: Don’t obscure the typed OTP in SpiderPig [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/854 (https://phabricator.wikimedia.org/T397505) [15:38:42] (03merge) 10dancy: LoginPage.vue: Don’t obscure the typed OTP in SpiderPig [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/854 (https://phabricator.wikimedia.org/T397505) [15:39:42] (03open) 10dancy: Release 4.181.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/855 [15:40:31] (03PS1) 10Phuedx: Zuul: [mediawiki/extensions/EventLogging] Add CodeEditor Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1162024 (https://phabricator.wikimedia.org/T346540) [15:41:30] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [15:41:35] (03merge) 10dancy: Release 4.181.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/855 [15:42:46] (03update) 10dduvall: image: Simplify remote context handling for bake [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/134 [15:42:51] (03merge) 10dduvall: image: Simplify remote context handling for bake [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/134 [15:52:56] 10Scap (SpiderPig 🕸️), 13Patch-For-Review: Don’t obscure the typed OTP in SpiderPig - https://phabricator.wikimedia.org/T397505#10934936 (10dancy) 05Open→03Resolved p:05Triage→03Low a:03dancy [15:53:08] 10Scap (SpiderPig 🕸️), 13Patch-For-Review: Don’t obscure the typed OTP in SpiderPig - https://phabricator.wikimedia.org/T397505#10934939 (10dancy) Deployed via scap 4.181.0 [15:56:30] RESOLVED: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:02:22] 06Release-Engineering-Team, 10Scap (SpiderPig 🕸️): SpiderPig session / login broken - https://phabricator.wikimedia.org/T397457#10934980 (10thcipriani) Relevant logs: https://logstash.wikimedia.org/goto/1d8446e79445ee0246f2be27f17c603a Can see the API giving 401s checking in with cas redirecting...will need s... [16:29:54] (03update) 10dduvall: make-container-image: introduce webserver-bookworm flavour [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/184 (https://phabricator.wikimedia.org/T378128) (owner: 10swfrench) [16:29:55] (03approved) 10dduvall: make-container-image: introduce webserver-bookworm flavour [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/184 (https://phabricator.wikimedia.org/T378128) (owner: 10swfrench) [17:07:49] (03open) 10bd808: bastion: base64 encode ssh private key [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/9 (https://phabricator.wikimedia.org/T396936) [17:50:31] (03open) 10dduvall: ci: Switch to strict semver tag format and create a release [repos/releng/kokkuri] (review/create-release-7b45) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/136 [17:50:32] (03open) 10dduvall: ci: Refactor pipeline workflows [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/135 [17:50:45] (03update) 10dduvall: ci: Refactor pipeline workflows [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/135 [17:50:49] (03update) 10dduvall: ci: Switch to strict semver tag format and create a release [repos/releng/kokkuri] (review/create-release-7b45) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/136 [17:50:53] (03update) 10dduvall: ci: Switch to strict semver tag format and create a release [repos/releng/kokkuri] (review/create-release-7b45) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/136 [17:51:15] (03update) 10dduvall: ci: Refactor pipeline workflows [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/135 [17:53:06] (03update) 10dduvall: ci: Refactor pipeline workflows [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/135 [17:53:06] (03update) 10dduvall: ci: Switch to strict semver tag format and create a release [repos/releng/kokkuri] (review/create-release-7b45) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/136 [17:53:06] (03update) 10dduvall: ci: Switch to strict semver tag format and create a release [repos/releng/kokkuri] (review/create-release-7b45) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/136 [17:53:11] (03update) 10dduvall: ci: Refactor pipeline workflows [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/135 [17:53:11] (03update) 10dduvall: ci: Switch to strict semver tag format and create a release [repos/releng/kokkuri] (review/create-release-7b45) - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/136 [18:18:20] !log seeing numerous image pull errors in gitlab-cloud-runner cluster [18:18:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:41:07] !log deleted docker-hub-mirror statefulset and admission controller deployment. reapplying production cluster configuration [18:41:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:47:58] dancy: for some strange reason, running a production deployment of gitlab-cloud-runner is not re-creating the dockerhub-mirror resources. have you seen something like that before? [18:56:09] ah, i think it's because of the helm resource state [18:57:03] !log ran `helm --namespace gitlab-runner uninstall docker-hub-mirror` to fix helm state. reapplying production cluster configuration [18:57:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:27:09] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 10Data-Persistence-Backup: Backups for new m1 database: zuul - https://phabricator.wikimedia.org/T396322#10935601 (10Dzahn) Thank you, sounds good. We have just not used it yet, that's why it has no tables so far. [20:46:18] 10Beta-Cluster-Infrastructure: deployment-prep member access for dr0ptp4kt - https://phabricator.wikimedia.org/T397547 (10dr0ptp4kt) 03NEW [20:55:09] dduvall: Sorry for the delay. Did you happen to get a snapshot of what the deployment state of `docker-hub-mirror` was? The helm-check.py script is supposed to fix up helm releases that are in a pending state [20:57:55] dancy: no worries. it was "deployed", i'm guessing because i deleted the statefulset and deployment resources manually [21:18:14] 10Beta-Cluster-Infrastructure: deployment-prep member access for dr0ptp4kt - https://phabricator.wikimedia.org/T397547#10935706 (10bd808) 05Open→03Resolved a:03bd808 `{{Done}}` {F62405575,size=full} [21:30:49] dduvall: that'll do it! [21:41:42] thx bd808 [22:27:27] (03merge) 10bd808: bastion: base64 encode ssh private key [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/9 (https://phabricator.wikimedia.org/T396936) [22:43:57] 10Phabricator (Upstream), 07Upstream: Unclear how to update email address of a Phabricator bot account - https://phabricator.wikimedia.org/T396881#10935869 (10Aklapper) 05Open→03Stalled [22:49:22] (03PS1) 10Krinkle: dockerfiles: Enable pcov in php81-84 base images [integration/config] - 10https://gerrit.wikimedia.org/r/1162106 (https://phabricator.wikimedia.org/T395743) [23:40:54] (03CR) 10Krinkle: "Debugging an issue with docker-pkg not working on macOS / Pyton 3.13." [integration/config] - 10https://gerrit.wikimedia.org/r/1162106 (https://phabricator.wikimedia.org/T395743) (owner: 10Krinkle)