[00:55:06] 10WikimediaDebug, 06MediaWiki-Platform-Team, 06serviceops: X-Wikimedia-Debug cookie not routed correctly in Kubernetes on POST requests - https://phabricator.wikimedia.org/T397439#11048758 (10Krinkle) [00:55:36] 10WikimediaDebug, 06MediaWiki-Platform-Team, 10MW-on-K8s, 06serviceops: X-Wikimedia-Debug cookie not routed correctly in Kubernetes on POST requests - https://phabricator.wikimedia.org/T397439#11048759 (10Krinkle) [08:31:39] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for aaronshaw - https://phabricator.wikimedia.org/T400869 (10Isaac) 03NEW [08:32:58] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for aaronshaw - https://phabricator.wikimedia.org/T400869#11049141 (10Isaac) I can vouch for Aaron - the username was provided to me over Slack by his account and matches [[https://meta.wikimedia.org/wiki/User:Aaronsh... [09:14:54] 10WikimediaDebug, 06MediaWiki-Platform-Team, 10MW-on-K8s, 06serviceops: X-Wikimedia-Debug cookie not routed correctly in Kubernetes on POST requests - https://phabricator.wikimedia.org/T397439#11049177 (10jijiki) Given that the value of the `X-Wikimedia-Debug` header determines whether, and to which `mw-de... [09:15:51] 10WikimediaDebug, 06MediaWiki-Platform-Team: X-Wikimedia-Debug cookie not routed correctly in Kubernetes on POST requests - https://phabricator.wikimedia.org/T397439#11049182 (10jijiki) [09:59:37] 10WikimediaDebug, 06MediaWiki-Platform-Team: X-Wikimedia-Debug cookie not routed correctly in Kubernetes on POST requests - https://phabricator.wikimedia.org/T397439#11049236 (10Tgr) I suppose that's possible (the [[https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/d79e8d4c80ab591cc1c95fb674da3117fc... [10:11:10] 10WikimediaDebug, 06MediaWiki-Platform-Team: X-Wikimedia-Debug cookie not routed correctly in Kubernetes on POST requests - https://phabricator.wikimedia.org/T397439#11049276 (10Tgr) ` $ curl -vso/dev/null -H 'X-Wikimedia-Debug: backend=k8s-mwdebug' 'https://meta.wikimedia.org/w/api.php' |& grep 'server:' < se... [10:13:34] 10WikimediaDebug, 06MediaWiki-Platform-Team: X-Wikimedia-Debug cookie not routed correctly in Kubernetes on POST requests - https://phabricator.wikimedia.org/T397439#11049287 (10Tgr) Oh duh, it has a ` if (req.method != "GET" && req.method != "HEAD") { return (pass); } ` block right above it. [12:32:54] (03open) 10jelto: Draft: remove home_page_url from gitlab-settings [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/76 (https://phabricator.wikimedia.org/T400695) [13:07:28] (03update) 10jelto: remove home_page_url from gitlab-settings [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/76 (https://phabricator.wikimedia.org/T400695) [13:30:33] (03approved) 10arnaudb: remove home_page_url from gitlab-settings [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/76 (https://phabricator.wikimedia.org/T400695) (owner: 10jelto) [13:33:51] (03approved) 10jelto: remove home_page_url from gitlab-settings [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/76 (https://phabricator.wikimedia.org/T400695) [13:33:59] (03merge) 10jelto: remove home_page_url from gitlab-settings [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/76 (https://phabricator.wikimedia.org/T400695) [13:34:44] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Remove home_page_url from gitlab-settings - https://phabricator.wikimedia.org/T400695#11050168 (10Jelto) p:05Triage→03Medium [13:39:58] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Remove home_page_url from gitlab-settings - https://phabricator.wikimedia.org/T400695#11050196 (10Jelto) 05Open→03Resolved This setting is managed by the cookbook now, see changes above. I'll resolve the task. cc @brennen [14:11:33] 06Gerrit-Privilege-Requests, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Offboard Noarave from WMF systems - https://phabricator.wikimedia.org/T399953#11050299 (10taavi) [14:30:37] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work: Buildkit v0.22.0 released - https://phabricator.wikimedia.org/T394931#11050354 (10brennen) 05In progress→03Resolved [15:27:04] 10Continuous-Integration-Infrastructure, 10VisualEditor, 07ci-test-error, 13Patch-For-Review: CI error because installing pngquant-bin NPM library fails with HTTP 429 - https://phabricator.wikimedia.org/T400730#11050645 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF I "fixed" this by j... [15:30:00] (03open) 10bking: Add opensearch-operator as trusted repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/129 (https://phabricator.wikimedia.org/T400295) [15:31:07] (03open) 10dancy: build-images: Use the .next suffix when building next image [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/927 [15:31:09] (03update) 10dancy: build-images: Use the .next suffix when building next image [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/927 [15:31:53] (03update) 10dancy: build-images: Use the .next suffix when building next image [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/927 [15:33:24] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.45.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T396373#11050716 (10brennen) [15:37:06] (03merge) 10dancy: build-images: Use the .next suffix when building next image [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/927 [15:37:57] (03open) 10dancy: Release 4.194.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/928 [15:39:00] Should Gerrit be down right now? [15:39:06] Project mediawiki-core-phpmetrics build #464: 04FAILURE in 0.16 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-phpmetrics/464/ [15:39:17] Oh I see in #wikimedia-operations a restart of Gerrit [15:40:08] 10Beta-Cluster-Infrastructure: Unblock 188.214.8.0/21 for Beta Cluster (Hyperoptic UK) - https://phabricator.wikimedia.org/T395709#11050770 (10Krinkle) [15:40:12] (03merge) 10dancy: Release 4.194.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/928 [15:41:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:41:44] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T400911 (10phaultfinder) 03NEW [15:42:31] 06Release-Engineering-Team, 06collaboration-services: ProbeDown (gerrit1003) - https://phabricator.wikimedia.org/T400911#11050798 (10Jelto) [15:42:48] 06Release-Engineering-Team, 06collaboration-services: ProbeDown (gerrit1003) - https://phabricator.wikimedia.org/T400911#11050800 (10Jelto) [15:43:39] !log gerrit: removed "Gerrit Managers" group as owner of all repositories. Changed to Administrators | https://gerrit.wikimedia.org/r/c/operations/puppet/+/1174669 [15:43:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:44:33] 06Release-Engineering-Team, 06collaboration-services: ProbeDown (gerrit1003) - https://phabricator.wikimedia.org/T400911#11050824 (10Jelto) 05Open→03Resolved p:05Triage→03Medium a:03Jelto A gerrit restart was required to deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/1174669. I'll r... [15:46:31] RESOLVED: [4x] ProbeDown: Service gerrit1003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [16:23:34] dancy: in theory we'd be at ~0 beta cluster puppet hacks now. except, 4 other non-triival patches have popped up since last year. [16:26:08] 10WikimediaDebug, 06MediaWiki-Platform-Team, 06Traffic, 13Patch-For-Review: X-Wikimedia-Debug cookie not routed correctly in Kubernetes on POST requests - https://phabricator.wikimedia.org/T397439#11051132 (10Krinkle) [16:27:05] 10Release-Engineering-Team (Priority Backlog 📥), 07OKR-Work: Publish updated wmf/next container when deploying config backport or security patch - https://phabricator.wikimedia.org/T398875#11051150 (10dancy) 05Open→03Resolved a:03dancy This change is live! The impact of building the next image duri... [16:30:30] !log Upgrading scap to 4.194.1 in beta cluster [16:30:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:05:49] !log Unblock 94.4.0.0/14 (T400926) [17:05:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:35:05] 10WikimediaDebug, 06MediaWiki-Platform-Team, 06Traffic: X-Wikimedia-Debug cookie not routed correctly in Kubernetes on POST requests - https://phabricator.wikimedia.org/T397439#11051473 (10Krinkle) p:05Triage→03Medium a:03Tgr [17:36:44] 10WikimediaDebug, 06MediaWiki-Platform-Team, 06Traffic: X-Wikimedia-Debug cookie not routed correctly in Kubernetes on POST requests - https://phabricator.wikimedia.org/T397439#11051483 (10Krinkle) `lang=irc,name=wikimedia-sre Krinkle: I can simply merge that one if that's fine. (doing now) <... [17:38:38] 10Phabricator, 06SRE, 06Traffic: traffic from Discord and Slack unfurler service is blocked by phabricator.wikimedia.org - https://phabricator.wikimedia.org/T400540#11051487 (10CDobbins) [17:51:02] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Provide ci-trixie base image, once Wikimedia trixie base Docker image exists - https://phabricator.wikimedia.org/T400931 (10Jdforrester-WMF) 03NEW [17:51:09] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Provide ci-trixie base image, once Wikimedia trixie base Docker image exists - https://phabricator.wikimedia.org/T400931#11051536 (10Jdforrester-WMF) [17:52:38] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T400775#11051542 (10bd808) 05Open→03Invalid Networking blip or similar [18:09:32] !log Unblock 152.37.64.0/18 (T400912) [18:09:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:16:21] (03update) 10bking: Add opensearch-operator as trusted repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/129 (https://phabricator.wikimedia.org/T400295) [19:16:29] (03update) 10bking: Add opensearch-operator as trusted repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/129 (https://phabricator.wikimedia.org/T400295) [19:19:43] (03update) 10bking: Add opensearch-operator as trusted repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/129 (https://phabricator.wikimedia.org/T400295) [19:19:49] (03update) 10bking: Add opensearch-operator as trusted repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/129 (https://phabricator.wikimedia.org/T400295) [19:22:29] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.45.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T396373#11051798 (10brennen) [19:23:52] 10Phabricator, 06SRE, 06Traffic: traffic from Discord and Slack unfurler service is blocked by phabricator.wikimedia.org - https://phabricator.wikimedia.org/T400540#11051804 (10ssingh) Hi, thanks for reporting @Novem_Linguae. The issue should be resolved now; I did a quick test but let us know if there is an... [19:28:34] 10Phabricator, 06SRE, 06Traffic: traffic from Discord and Slack unfurler service is blocked by phabricator.wikimedia.org - https://phabricator.wikimedia.org/T400540#11051814 (10Michael) Woohoo! Can confirm! Thank you so much @Novem_Linguae, @ssingh and Traffic Team! 🏆 [19:44:05] (03close) 10bking: Add opensearch-operator as trusted repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/129 (https://phabricator.wikimedia.org/T400295) [19:46:25] (03open) 10bking: Add opensearch-operator as trusted repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/130 (https://phabricator.wikimedia.org/T400295) [20:25:57] 10Continuous-Integration-Infrastructure (Zuul upgrade): Set up new project for Zuulv3+ pre-merge and non-image-build workloads - https://phabricator.wikimedia.org/T396247#11051929 (10bd808) 05In progress→03Resolved The project has existed for quite a while at this point. This and {T396936} are starting t... [20:44:50] 10Phabricator, 06SRE, 06Traffic: traffic from Discord and Slack unfurler service is blocked by phabricator.wikimedia.org - https://phabricator.wikimedia.org/T400540#11051974 (10AntiCompositeNumber) Still no cards on Discord, including brand new tasks. [20:44:51] 10Continuous-Integration-Infrastructure (Zuul upgrade): Investigate monitoring and reporting on the health of the Magnum managed Kubernetes cluster - https://phabricator.wikimedia.org/T400938 (10bd808) 03NEW [21:12:58] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/EmbedSpotify] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1174821 [21:12:58] (03CR) 10QChris: [V:03+2 C:03+2] Allow “Gerrit Managers” to import history [extensions/EmbedSpotify] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1174821 (owner: 10QChris) [21:13:02] (03PS1) 10QChris: Import done. Revoke import grants [extensions/EmbedSpotify] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1174822 [21:13:02] (03CR) 10QChris: [V:03+2 C:03+2] Import done. Revoke import grants [extensions/EmbedSpotify] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1174822 (owner: 10QChris) [21:17:08] 10Release-Engineering-Team (Priority Backlog 📥), 07OKR-Work: Publish updated wmf/next container when deploying config backport or security patch - https://phabricator.wikimedia.org/T398875#11052088 (10dancy) 05Resolved→03Open Looks like the changes caused a deployment problem so I rolled back scap. [21:29:21] 10Phabricator, 06SRE, 06Traffic: traffic from Discord and Slack unfurler service is blocked by phabricator.wikimedia.org - https://phabricator.wikimedia.org/T400540#11052148 (10ssingh) >>! In T400540#11051974, @AntiCompositeNumber wrote: > Still no cards on Discord, including brand new tasks. Sorry about th... [21:47:10] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: allow new zuul executor VMs in prod to talk to cloud VPS - https://phabricator.wikimedia.org/T394838#11052221 (10bd808) >>! In T394838#10924680, @bd808 wrote: > As noted in the Wikitech documentation, we can find another solution... [21:47:23] 10Gerrit, 10Release-Engineering-Team (Priority Backlog 📥): Folks no longer able to merge gerrit dashboard changes - https://phabricator.wikimedia.org/T400829#11052226 (10Umherirrender) @hashar has changed access rights from "ldap/wmf" to "Gerrit Managers" => https://gerrit.wikimedia.org/r/plugins/gitiles/wikim... [22:03:33] (03open) 10dancy: make-container-image: Mention state file and directory [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/198 [22:03:36] (03update) 10dancy: make-container-image: Mention state file and directory [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/198 [22:05:17] (03merge) 10dancy: Add opensearch-operator as trusted repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/130 (https://phabricator.wikimedia.org/T400295) (owner: 10bking) [22:05:21] (03merge) 10dancy: make-container-image: Mention state file and directory [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/198 [22:19:42] (03open) 10dancy: config.py: Disable build_mw_next_container_image [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/929 (https://phabricator.wikimedia.org/T398875) [22:19:44] (03update) 10dancy: config.py: Disable build_mw_next_container_image [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/929 (https://phabricator.wikimedia.org/T398875) [22:22:40] (03merge) 10dancy: config.py: Disable build_mw_next_container_image [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/929 (https://phabricator.wikimedia.org/T398875) [22:23:23] (03open) 10dancy: Release 4.194.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/930 [22:25:29] (03merge) 10dancy: Release 4.194.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/930 [22:29:09] (03open) 10dancy: JobLog.vue: Increase scrollback to 10000 lines [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/931 [22:29:13] (03update) 10dancy: JobLog.vue: Increase scrollback to 10000 lines [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/931 [22:42:19] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.45.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T396373#11052452 (10brennen) 05Open→03Resolved Stable on all wikis. [23:00:00] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: allow new zuul executor VMs in prod to talk to cloud VPS - https://phabricator.wikimedia.org/T394838#11052522 (10bd808) @thcipriani pointed out that in my tests from T394838#11052219 my direct access from my laptop also had a 403... [23:08:28] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: allow new zuul executor VMs in prod to talk to cloud VPS - https://phabricator.wikimedia.org/T394838#11052527 (10bd808) `name=hieradata/common/profile/installserver/proxy.yaml,lang=yaml profile::installserver::proxy::ssl_ports:...