[00:06:53] * bd808 goes to make dinner [01:56:42] 10Beta-Cluster-Infrastructure: [tofu] Add support for service name DNS records in the service module - https://phabricator.wikimedia.org/T436484#12269482 (10bd808) [01:56:43] 10Beta-Cluster-Infrastructure, 07Epic: Use infrastructure as code techniques to rebuild the Beta Cluster - https://phabricator.wikimedia.org/T394316#12269483 (10bd808) [01:57:59] 10Beta-Cluster-Infrastructure: [tofu] Output openstack_compute_instance_v2 instances from service module - https://phabricator.wikimedia.org/T436483#12269485 (10bd808) [01:58:01] 10Beta-Cluster-Infrastructure, 07Epic: Use infrastructure as code techniques to rebuild the Beta Cluster - https://phabricator.wikimedia.org/T394316#12269486 (10bd808) [03:18:02] 10Beta-Cluster-Infrastructure: [tofu] Output openstack_compute_instance_v2 instances from service module - https://phabricator.wikimedia.org/T436483#12269510 (10bd808) 05Open→03Resolved a:03bd808 [03:18:25] 10Beta-Cluster-Infrastructure: [tofu] Add support for service name DNS records in the service module - https://phabricator.wikimedia.org/T436484#12269514 (10bd808) 05Open→03Resolved a:03bd808 [03:22:27] 10Beta-Cluster-Infrastructure, 07Epic: Use infrastructure as code techniques to rebuild the Beta Cluster - https://phabricator.wikimedia.org/T394316#12269520 (10bd808) 05Open→03In progress p:05Triage→03Medium This is being actively used in the work for {T401839}. We are not likely to have all of the Bu... [04:13:52] (03PS1) 10Krinkle: zuul: switch CVNBot repo from dotnet to tox [integration/config] - 10https://gerrit.wikimedia.org/r/1332530 (https://phabricator.wikimedia.org/T327136) [04:13:54] (03PS1) 10Krinkle: jjb: Remove unused dotnet-mono612 job [integration/config] - 10https://gerrit.wikimedia.org/r/1332531 (https://phabricator.wikimedia.org/T327136) [04:15:49] (03CR) 10Krinkle: [C:03+2] zuul: switch CVNBot repo from dotnet to tox [integration/config] - 10https://gerrit.wikimedia.org/r/1332530 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [04:17:28] (03Merged) 10jenkins-bot: zuul: switch CVNBot repo from dotnet to tox [integration/config] - 10https://gerrit.wikimedia.org/r/1332530 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [04:17:30] (03CR) 10Krinkle: [C:03+2] "`" [integration/config] - 10https://gerrit.wikimedia.org/r/1332531 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [04:18:07] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1332530 [04:18:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:19:08] (03Merged) 10jenkins-bot: jjb: Remove unused dotnet-mono612 job [integration/config] - 10https://gerrit.wikimedia.org/r/1332531 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [07:13:09] 10Phabricator: EXCEPTION: (Exception) Policy identifier is an object PHID (''), but no object handle was provided. A handle must be provided for object policies. - https://phabricator.wikimedia.org/T361459#12269726 (10Aklapper) [07:13:50] 10Phabricator (Upstream), 07Upstream: Improve default parameters of default Maniphest global saved search queries - https://phabricator.wikimedia.org/T187587#12269729 (10Aklapper) 05Stalled→03Resolved I'm closing this as good enough IIUC. [07:13:57] 10Phabricator (Upstream), 06Release-Engineering-Team (Priority Backlog 📥), 07Upstream: Maniphest Advanced Search form should display "100" in its "Page Size" field by default - https://phabricator.wikimedia.org/T288915#12269736 (10Aklapper) a:05Aklapper→03None [07:17:08] 10Phabricator (Upstream), 07Upstream: Searching shorter terms ("gerrit", "cors") in global Phabricator search leads to "upstream request timeout" (too many results?) - https://phabricator.wikimedia.org/T258803#12269741 (10Aklapper) [07:24:35] 06Project-Admins, 06Quality-and-Test-Engineering-Team: Archive the #Quality-and-Test-Engineering-Team project(?) - https://phabricator.wikimedia.org/T423518#12269759 (10Aklapper) a:05SLong-WMF→03None Removing inactive assignee [07:24:39] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10QS-Test-Automation: Better prevent changes in MW core from breaking WMF production extensions - https://phabricator.wikimedia.org/T404403#12269761 (10Aklapper) a:05SLong-WMF→03None Removing inactive assignee [07:29:24] 06Release-Engineering-Team, 10MediaWiki-REST-API, 07Epic, 06MediaWiki-API-Platform-Team (MWI-Roadmap), and 2 others: [REST Sandbox] Remove SwaggerUI from MediaWiki Releases - https://phabricator.wikimedia.org/T397470#12269798 (10Aklapper) a:05Atieno→03None Removing inactive assignee - please do so as p... [09:04:00] (03CR) 10Hashar: [C:04-1] "You can have your local git to ignore JetBrains regardless of the repository by using `$XDG_CONFIG_HOME/git/ignore`. Here is mine:" [integration/config] - 10https://gerrit.wikimedia.org/r/1332246 (owner: 10Siebrand) [09:10:54] 10GitLab, 06Collaboration-Services: Internal Server Error when publishing a maven package from GitLab CI - https://phabricator.wikimedia.org/T434834#12270133 (10ABran-WMF) For posterity, here is the distribution after some cleaning up: ` sudo gitlab-psql -c "... [09:14:53] (03CR) 10Hashar: "I think we will want this CI config to be applied first (and thus remove the `Depends-On` header from this change). Then **once deployed*" [integration/config] - 10https://gerrit.wikimedia.org/r/1329282 (https://phabricator.wikimedia.org/T405008) (owner: 10Mszwarc) [09:16:33] !log move cx-server and citoid-beta endpoints in deployment-prep to two new Trixie VMs, update their configs and Docker images and delete the old images. [09:16:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:20:03] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [09:22:04] FIRING: InstanceDown: Project deployment-prep instance deployment-docker-citoid02 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:22:13] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-docker-citoid02 is down - https://phabricator.wikimedia.org/T436507 (10wmcs-alerts) 03NEW [09:25:10] Lucas_WMDE: [09:25:37] wrong command, I wanted to /query ! [09:27:04] RESOLVED: InstanceDown: Project deployment-prep instance deployment-docker-citoid02 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [10:58:03] 10Phabricator, 07Upstream: Pastes don't allow lua syntax highlighting - https://phabricator.wikimedia.org/T100900#12270406 (10valerio.bozzolan) I was browsing this Puppet page: https://gerrit.wikimedia.org/g/operations/puppet/+/production/modules/phabricator/data/fixed_settings.yaml#200 and there was a lin... [11:08:56] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for fengri - https://phabricator.wikimedia.org/T436512 (10Fengrimur) 03NEW [11:47:37] 10Scap, 06SRE Observability: Scap cli statd logging uses inactive statsd endpoint - https://phabricator.wikimedia.org/T436513 (10hnowlan) 03NEW [11:47:54] 10Scap, 06SRE Observability: Scap cli statd logging uses inactive statsd endpoint - https://phabricator.wikimedia.org/T436513#12270518 (10hnowlan) [11:57:03] (03CR) 10Siebrand: "Thanks for the hint, Antoine." [integration/config] - 10https://gerrit.wikimedia.org/r/1332246 (owner: 10Siebrand) [11:57:12] (03Abandoned) 10Siebrand: gitignore: ignore JetBrains *.iml files [integration/config] - 10https://gerrit.wikimedia.org/r/1332246 (owner: 10Siebrand) [12:06:05] 10Phabricator (Upstream), 07Upstream: Headers for some Phab notification emails reference a Herald rule that is currently disabled - https://phabricator.wikimedia.org/T420162#12270578 (10Aklapper) 05Open→03Invalid Today I found out that the `X-Herald-Rules` header in notification emails which reference... [12:35:25] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Missing space between two Diffusion strings / messages - https://phabricator.wikimedia.org/T304205#12270625 (10Aklapper) a:03Aklapper [12:35:35] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Missing space between two Diffusion strings / messages - https://phabricator.wikimedia.org/T304205#12270629 (10Aklapper) [12:38:20] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/UmamiAnalytics] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1332715 [12:38:20] (03CR) 10QChris: [V:03+2 C:03+2] Allow “Gerrit Managers” to import history [extensions/UmamiAnalytics] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1332715 (owner: 10QChris) [12:38:24] (03PS1) 10QChris: Import done. Revoke import grants [extensions/UmamiAnalytics] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1332716 [12:38:25] (03CR) 10QChris: [V:03+2 C:03+2] Import done. Revoke import grants [extensions/UmamiAnalytics] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1332716 (owner: 10QChris) [12:48:59] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Add mouse-over popup tooltips for buttons when viewing images in Phab - https://phabricator.wikimedia.org/T281793#12270671 (10Aklapper) a:03Aklapper [12:51:08] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Add mouse-over popup tooltips for buttons when viewing images in Phab - https://phabricator.wikimedia.org/T281793#12270678 (10Aklapper) [13:08:59] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12270742 (10ABran-WMF) So, this was a wrong direction: >>! In T425441#12247447, @ABran-WMF wrote: > Still open before re-pointing the primary at the CDN... [13:13:04] (03PS1) 10Hashar: tests: fix test catching wrong exception [integration/quibble] - 10https://gerrit.wikimedia.org/r/1332724 [13:15:13] (03CR) 10Hashar: "@adam.wight@wikimedia.de I found a bug from a test you wrote back in 2020 when implementing Parallelism :]" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1332724 (owner: 10Hashar) [13:17:43] (03PS2) 10Hashar: tests: fix test catching wrong exception [integration/quibble] - 10https://gerrit.wikimedia.org/r/1332724 (https://phabricator.wikimedia.org/T436226) [13:20:07] (03open) 10jnuche: patchdemo: install PHP's LDAP client module [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/122 (https://phabricator.wikimedia.org/T435931) [13:20:34] (03merge) 10jnuche: patchdemo: install PHP's LDAP client module [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/122 (https://phabricator.wikimedia.org/T435931) [13:21:28] !log Updating development images on contint primary for T435931 [13:21:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:21:48] (03CR) 10Hashar: "Sorry! I had most of the fixes in https://gerrit.wikimedia.org/r/c/integration/quibble/+/1330357/1 then named *WIP TEST MIGRATION* and th" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1331654 (https://phabricator.wikimedia.org/T436226) (owner: 10Hashar) [13:31:48] (03open) 10phedenskog: Split the castor blocked share into wait and blocked columns [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/53 [13:49:02] (03CR) 10Mszwarc: "I ordered the patches this was so that nothing in CI tries to call `npm run doc` on CheckUser when it doesn't exist, but if you're suggest" [integration/config] - 10https://gerrit.wikimedia.org/r/1329282 (https://phabricator.wikimedia.org/T405008) (owner: 10Mszwarc) [13:49:17] (03PS3) 10Mszwarc: Zuul: [mediawiki/extensions/CheckUser] Publish JS docs post-merge [integration/config] - 10https://gerrit.wikimedia.org/r/1329282 (https://phabricator.wikimedia.org/T405008) [14:14:35] 10Phabricator, 06SRE: Reactivate Sheila Wangari's Phabricator account - https://phabricator.wikimedia.org/T436521#12271149 (10JLam-WMF) [14:16:20] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12271155 (10ABran-WMF) [14:27:57] 10Phabricator, 06SRE: Reactivate Sheila Wangari's Phabricator account - https://phabricator.wikimedia.org/T436521#12271232 (10taavi) 05Open→03Resolved a:03taavi Done. [14:33:27] (03CR) 10STran: [C:03+1] Zuul: [mediawiki/extensions/CheckUser] Publish JS docs post-merge [integration/config] - 10https://gerrit.wikimedia.org/r/1329282 (https://phabricator.wikimedia.org/T405008) (owner: 10Mszwarc) [15:12:06] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:32:44] 10GitLab (Misc): 김정은 똥 먹으라 - https://phabricator.wikimedia.org/T436534 (1020260901t0008) 03NEW [15:36:45] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Back up Zuul encryption keys - https://phabricator.wikimedia.org/T436131#12271597 (10LSobanski) p:05Triage→03Medium [15:36:48] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Back up Zuul encryption keys - https://phabricator.wikimedia.org/T436131#12271599 (10LSobanski) p:05Medium→03Triage [15:39:30] (03merge) 10phedenskog: Split the castor blocked share into wait and blocked columns [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/53 [15:42:51] 10Phabricator, 06Release-Engineering-Team: Establish a workflow that scales for requesting Phab 2FA resets - https://phabricator.wikimedia.org/T306708#12271650 (10LSobanski) [15:46:30] 10GitLab (Infrastructure), 06Collaboration-Services: Create GitLab mirror - https://phabricator.wikimedia.org/T291322#12271691 (10LSobanski) Backups run faster now so we could consider having them run more often. Still a low priority task. [15:46:45] (03open) 10phedenskog: Show p75 next to the median in the core slowest-by-month tables [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/54 [15:47:40] 10Phabricator: Reactivate Sheila Wangari's Phabricator account - https://phabricator.wikimedia.org/T436521#12271700 (10Aklapper) [15:50:11] 10GitLab (Infrastructure), 06Collaboration-Services: Setup alerting for GitLab projects size limits - https://phabricator.wikimedia.org/T316553#12271709 (10LSobanski) p:05Low→03Medium This would have prevented {T434834} Raising the priority of this. Medium because of the recent clean up but could become H... [15:50:14] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537#12271707 (10VRiley-WMF) [15:56:46] 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 10Cloud-VPS (Debian Bullseye Deprecation): Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12271764 (10bd808) I made a [[https://gitlab.wikimedia.org/cloudvps-r... [16:29:49] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537#12271912 (10elukey) Hey folks, I didn't read the whole task since I got aware of it only from the SRE's te... [16:32:49] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537#12271932 (10elukey) @Andrew could you please list here what parameters you modified? It will help me into... [16:33:05] (03merge) 10dancy: deploy-service: Run the check commands of the service catalog [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1296 (https://phabricator.wikimedia.org/T412941) [16:53:20] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537#12271995 (10elukey) Dumping some stuff found on cloudvirt1077 - I see the following BIOS option via Redfis... [17:22:41] (03update) 10dancy: deploy-service: Factor out some types [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1298 (https://phabricator.wikimedia.org/T412941) [17:22:44] (03open) 10dancy: deploy-service: Factor out some types [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1298 (https://phabricator.wikimedia.org/T412941) [17:25:32] (03update) 10dancy: deploy-service: Factor out some types [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1298 (https://phabricator.wikimedia.org/T412941) [17:37:15] (03open) 10dancy: statsd_host now defaults to None [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1299 (https://phabricator.wikimedia.org/T436513) [17:39:04] (03merge) 10dancy: statsd_host now defaults to None [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1299 (https://phabricator.wikimedia.org/T436513) [17:44:53] (03PS3) 10Hashar: tests: replace assertRaises() doing a wild catch [integration/quibble] - 10https://gerrit.wikimedia.org/r/1332724 (https://phabricator.wikimedia.org/T436226) [17:45:32] 10Scap: Include in scap output the number of security patches currently deployed/applied - https://phabricator.wikimedia.org/T434073#12272144 (10dancy) Hi @Reedy. Would you mind taking existing scap output and pasting in here, including an example of what you want to see and where you want to see it? [17:46:32] (03CR) 10Hashar: "I caught a few more `assertRaises(Exception ,msg='whatever')` which would caught any underlying/unrelated exceptions. I replaced them all " [integration/quibble] - 10https://gerrit.wikimedia.org/r/1332724 (https://phabricator.wikimedia.org/T436226) (owner: 10Hashar) [18:04:09] (03update) 10dancy: deploy-promote: Verify the promotion with deployment-info.php [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1280 (https://phabricator.wikimedia.org/T434726) [18:06:39] (03merge) 10dancy: deploy-promote: Verify the promotion with deployment-info.php [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1280 (https://phabricator.wikimedia.org/T434726) [18:09:18] (03update) 10dancy: deploy-service: Factor out some types [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1298 (https://phabricator.wikimedia.org/T412941) [18:09:19] (03update) 10dancy: deploy-service: Add manual check support [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1300 (https://phabricator.wikimedia.org/T412941) [18:09:22] (03update) 10dancy: deploy-service: Factor out some types [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1298 (https://phabricator.wikimedia.org/T412941) [18:09:27] (03open) 10dancy: deploy-service: Add manual check support [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1300 (https://phabricator.wikimedia.org/T412941) [18:09:46] (03update) 10dancy: deploy-service: Run the check commands of the service catalog [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1296 (https://phabricator.wikimedia.org/T412941) [18:12:53] (03update) 10dancy: deploy-service: Add manual check support [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1300 (https://phabricator.wikimedia.org/T412941) [18:13:03] (03update) 10dancy: deploy-service: Add manual check support [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1300 (https://phabricator.wikimedia.org/T412941) [18:13:57] (03update) 10dancy: deploy-service: Add manual check support [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1300 (https://phabricator.wikimedia.org/T412941) [18:22:33] (03merge) 10phedenskog: Show p75 next to the median in the core slowest-by-month tables [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/54 [18:42:23] (03approved) 10swfrench: deploy-service: Factor out some types [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1298 (https://phabricator.wikimedia.org/T412941) (owner: 10dancy) [18:44:14] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Replace deployment-wikifeeds02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436462#12272360 (10Southparkfan) The VM has been decommissioned, but [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+... [19:00:59] bd808: the 503s on the new PHP 8.5 machine, is because php8.5-fpm returns... nothing. [19:02:59] To rule out something in Apache, I've been playing a bit with `cgi-fcgi` on both mediawiki13 and mediawiki15. On mediawiki15, not even HTTP headers are returned. Perfectly fits the `AH01067: Failed to read FastCGI header` error thrown by Apache. [19:10:14] Southparkfan: bd808: want to just move all the deployment-prep hiera from puppet.git to horizon/enc so that it doesn't take someone with ops access to update that? [19:11:25] No objections from my side [19:23:20] (03update) 10thcipriani: Deployment-calendar: Allow event timezones [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/266 (https://phabricator.wikimedia.org/T436187) [19:23:20] (03update) 10thcipriani: Deployment-calendar: Ensure we can run scripts outside cluster [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/265 [19:23:30] (03open) 10thcipriani: Deployment-calendar: Ensure we can run scripts outside cluster [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/265 [19:23:31] (03open) 10thcipriani: Deployment-calendar: Allow event timezones [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/266 (https://phabricator.wikimedia.org/T436187) [19:29:44] (03merge) 10thcipriani: Deployment-calendar: Ensure we can run scripts outside cluster [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/265 [19:29:48] (03update) 10thcipriani: Deployment-calendar: Allow event timezones [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/266 (https://phabricator.wikimedia.org/T436187) [19:43:39] bd808: mediawiki15 and mediawiki16 respond now. EtcdConfig was broken, because the root certificate of etcd05 was not in the trust store. The hash-based symlink for `WMF_TEST_CA.pem` (root CA), `/etc/ssl/certs/46fbbce3.0`, has not been generated during system installation. `openssl rehash` was sufficient. [19:44:58] With "the root certificate of etcd05", I meant the `C = US, ST = California, L = San Francisco, O = "Wikimedia Foundation, Inc", OU = Cloud Services, CN = WMF_TEST_CA` CA. The leaf certificate on etcd05 is issued by this "CA" directly, no intermediates in-between. [19:45:53] Southparkfan: nice work. Thank you :) [19:46:45] taavi: getting all of the deployment-prep hiera out of ops/puppet and into Horizon so we can later put it all in tofu would be really nice. [19:55:52] bd808: well, I tried, but `pymysql.err.DataError: (1406, "Data too long for column 'hiera_data' at row 1")` [19:56:16] oh that's neat [19:56:56] the stupid CDN blocking stuff is probably part of that problem [19:57:09] 06Release-Engineering-Team, 06Collaboration-Services, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Matomo K8s migration: Create OCI (Docker) image - https://phabricator.wikimedia.org/T436005#12272657 (10bking) I have what appears to be a working Matomo OCI image [[ https://gitlab.wiki... [19:57:13] the field is typed as a `TEXT` fwiw, so I guess needs migrating to `MEDIUMTEXT` :/ [19:57:24] bigger is better! [19:57:49] the current enc.yaml is 62k, so rather close to hitting the 65k limit of a TEXT [19:58:12] 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 10Cloud-VPS (Debian Bullseye Deprecation): Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12272658 (10Southparkfan) >>! In T435393#12269425, @bd808 wrote: > Th... [19:59:01] Sometimes, my PKI experience proves useful... ;p [19:59:49] bd808: do you have opinions which task to use to track getting rid of the hiera in puppet.git? T277680? T161675? [19:59:50] T277680: Unduplicate beta cluster hiera keys set both in Horizon and in ops/puppet - https://phabricator.wikimedia.org/T277680 [19:59:50] T161675: Re-think puppet management for deployment-prep - https://phabricator.wikimedia.org/T161675 [20:03:49] taavi: by the way, thanks for your work on this :-) [20:04:50] taavi: dealer's choice :) Maybe T277680 is close to the intent [20:04:51] T277680: Unduplicate beta cluster hiera keys set both in Horizon and in ops/puppet - https://phabricator.wikimedia.org/T277680 [20:05:27] (03merge) 10dancy: deploy-service: Factor out some types [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1298 (https://phabricator.wikimedia.org/T412941) [20:05:28] (03update) 10dancy: deploy-service: Add manual check support [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1300 (https://phabricator.wikimedia.org/T412941) [20:06:38] I'll file a task for a permanent fix on the missing symlinks. Likely going to see this issue again. [20:07:17] that symlink thing feels oddly familiar [20:11:25] bd808: anyway, enc database fixed (T436561) so I was able to push through https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/a376b3ffebaa34ec4d3bc3d17210d77f67489c73%5E%21/#F0 [20:11:25] T436561: Update ENC Hiera value column type to MEDIUMTEXT - https://phabricator.wikimedia.org/T436561 [20:11:31] (03CR) 10Awight: [C:03+2] "These are nice things to actually test, great to see the fix!" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1332724 (https://phabricator.wikimedia.org/T436226) (owner: 10Hashar) [20:15:06] which lets us do https://gerrit.wikimedia.org/r/c/operations/puppet/+/1332804 [20:15:48] (03Merged) 10jenkins-bot: tests: replace assertRaises() doing a wild catch [integration/quibble] - 10https://gerrit.wikimedia.org/r/1332724 (https://phabricator.wikimedia.org/T436226) (owner: 10Hashar) [20:19:41] 10Beta-Cluster-Infrastructure, 10CFSSL-PKI, 06Infrastructure-Foundations: profile::pki::client does not create symlinks using hashes of CAs - https://phabricator.wikimedia.org/T436562 (10Southparkfan) 03NEW [20:30:03] (03open) 10dancy: deploy-service: Name the group of the plan from the plan [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1301 (https://phabricator.wikimedia.org/T412941) [20:31:26] (03update) 10dancy: deploy-service: Factor out some types [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1298 (https://phabricator.wikimedia.org/T412941) [20:32:28] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace deployment-wikifeeds02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436462#12272771 (10Southparkfan) 05In progress→03Resolved [20:36:54] (03update) 10dancy: deploy-service: Name the group of the plan from the plan [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1301 (https://phabricator.wikimedia.org/T412941) [20:36:56] (03update) 10dancy: deploy-service: Name the group of the plan from the plan [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1301 (https://phabricator.wikimedia.org/T412941) [20:37:53] (03update) 10dancy: deploy-service: Name the group of the plan from the plan [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1301 (https://phabricator.wikimedia.org/T412941) [21:24:30] bd808: I have performed a smoke test for mediawiki16 by making it serve non-api traffic for about a minute, and at least Special:Version renders just fine. I have reverted the change to ATS config, so we're back to mediawiki14 now. [21:26:20] 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 10Cloud-VPS (Debian Bullseye Deprecation): Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12272969 (10Southparkfan) As mentioned @IRC: I have performed a smoke... [21:28:33] (03approved) 10swfrench: deploy-service: Add manual check support [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1300 (https://phabricator.wikimedia.org/T412941) (owner: 10dancy) [21:37:33] Southparkfan: cool! thcipriani will be the new bd808 for this stuff starting ... tomorrow :) [21:37:43] thcipriani: meet Southparkfan [21:40:13] (03update) 10dancy: deploy-service: Add manual check support [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1300 (https://phabricator.wikimedia.org/T412941) [21:40:15] (03update) 10dancy: deploy-service: Add manual check support [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1300 (https://phabricator.wikimedia.org/T412941) [21:42:03] (03merge) 10dancy: deploy-service: Add manual check support [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1300 (https://phabricator.wikimedia.org/T412941) [21:44:37] (03open) 10dancy: deploy-service: Reduce rollback prompt verbosity [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1302 (https://phabricator.wikimedia.org/T412941) [21:48:06] bd808: Last day before your sabbatical? [21:48:32] yes. Tomorrow I'm a volunteer for 2 months :) [21:48:55] Alright [21:50:14] Well deserved. Time to discover even more hobbies ;-) [21:51:55] (03merge) 10dancy: deploy-service: Reduce rollback prompt verbosity [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1302 (https://phabricator.wikimedia.org/T412941) [21:55:42] 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 10Cloud-VPS (Debian Bullseye Deprecation): Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12273057 (10Southparkfan) @Krinkle Hi! You have migrated the Beta Clu... [22:57:35] o/ ohai Southparkfan I'm fresh back from sabbatical so bd808 was getting me up to speed with what you all talked about for beta. Sounds like we're aiming for up-to-date and fancified with tofu which seems amazing :) [23:13:02] 06Release-Engineering-Team, 06Collaboration-Services, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Matomo K8s migration: Create OCI (Docker) image - https://phabricator.wikimedia.org/T436005#12273272 (10bking) [23:38:59] (03update) 10thcipriani: Deployment-calendar: Allow event timezones [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/266 (https://phabricator.wikimedia.org/T436187) [23:46:33] (03merge) 10thcipriani: Deployment-calendar: Allow event timezones [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/266 (https://phabricator.wikimedia.org/T436187)