[00:18:02] <wikibugs>	 10Traffic, 10Infrastructure-Foundations, 10SRE, 10vm-requests: eqiad: 1 VM request for ncmonitor - https://phabricator.wikimedia.org/T356710 (10Dzahn) ` 23:55 <+logmsgbot> !log dzahn@cumin1002 START - Cookbook sre.hosts.reimage for host ncmonitor1001.eqiad.wmnet with OS bookworm 00:01 <+logmsgbot> !log dza...
[08:33:48] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney)
[08:35:29] <wikibugs>	 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10cmooney) 05Open→03Resolved a:03cmooney Closing - thanks all for the help!
[09:05:28] <wikibugs>	 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A5 from asw-a5-codfw to lsw1-a5-codfw - https://phabricator.wikimedia.org/T355864 (10ops-monitoring-bot) Draining ganeti2023.codfw.wmnet of running VMs
[09:38:16] <wikibugs>	 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A5 from asw-a5-codfw to lsw1-a5-codfw - https://phabricator.wikimedia.org/T355864 (10ops-monitoring-bot) Draining ganeti2024.codfw.wmnet of running VMs
[10:24:41] <wikibugs>	 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A5 from asw-a5-codfw to lsw1-a5-codfw - https://phabricator.wikimedia.org/T355864 (10klausman)
[10:34:23] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney)
[11:17:52] <wikibugs>	 10Traffic, 10MW-on-K8s, 10Release-Engineering-Team, 10SRE, 10serviceops: Move 50% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T357507 (10Clement_Goubert)
[11:20:16] <wikibugs>	 10Traffic, 10MW-on-K8s, 10Release-Engineering-Team, 10SRE, 10serviceops: Move 50% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T357507 (10Clement_Goubert) p:05Triage→03High
[11:20:59] <wikibugs>	 10Traffic, 10MW-on-K8s, 10Release-Engineering-Team, 10SRE, 10serviceops: Move 60% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T357508 (10Clement_Goubert)
[11:21:43] <wikibugs>	 10Traffic, 10MW-on-K8s, 10Release-Engineering-Team, 10SRE, 10serviceops: Move 60% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T357508 (10Clement_Goubert)
[11:23:53] <wikibugs>	 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert)
[11:25:33] <wikibugs>	 10Traffic, 10MW-on-K8s, 10Release-Engineering-Team, 10SRE, 10serviceops: Move 60% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T357508 (10Clement_Goubert) 05Open→03Stalled p:05Triage→03High We'll stay at 50% or under because we want scap to check canaries before...
[11:26:55] <wikibugs>	 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert)
[12:51:56] <wikibugs>	 10Domains, 10Fundraising-Backlog, 10SRE, 10serviceops: Request donatewiki redirect - https://phabricator.wikimedia.org/T357436 (10AKanji-WMF) Thank you @RLazarus ! The fundraising team would like the redirect to be active for two years - considering the life cycle of the podcast and surrounding marketing c...
[14:59:51] <wikibugs>	 10Domains, 10Fundraising-Backlog, 10SRE, 10serviceops: Request donatewiki redirect - https://phabricator.wikimedia.org/T357436 (10AKanji-WMF) Tagging you @Dwisehaupt as Jeff is out this AM.
[15:31:48] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team, 10User-aborrero: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184 (10aborrero)
[15:31:58] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team, 10User-aborrero: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184 (10aborrero) 05Stalled→03Open In a 2024-02-14 network sync meeting we decided to continue moving older cloudvirts into the new single NI...
[15:51:32] <wikibugs>	 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A5 from asw-a5-codfw to lsw1-a5-codfw - https://phabricator.wikimedia.org/T355864 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=9a43620e-deca-432c-aa1f-5d6e939b51bc) set by cmooney@cumin...
[16:06:47] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10cloud-services-team, 10User-aborrero: clouddb: evaluate moving them into cloud-private - https://phabricator.wikimedia.org/T357543 (10aborrero)
[16:07:31] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10cloud-services-team, 10User-aborrero: clouddb: evaluate moving them into cloud-private - https://phabricator.wikimedia.org/T357543 (10aborrero) p:05Triage→03Medium
[16:07:48] <wikibugs>	 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A5 from asw-a5-codfw to lsw1-a5-codfw - https://phabricator.wikimedia.org/T355864 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=ec1ab967-b8f5-4bfd-914e-e76afe369468) set by cmooney@cumin...
[16:14:59] <wikibugs>	 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A5 from asw-a5-codfw to lsw1-a5-codfw - https://phabricator.wikimedia.org/T355864 (10cmooney) All links moved and all devices pinging ok again.
[16:16:11] <wikibugs>	 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A5 from asw-a5-codfw to lsw1-a5-codfw - https://phabricator.wikimedia.org/T355864 (10ABran-WMF) awesome, will start repooling, thanks @cmooney
[16:36:07] <wikibugs>	 10Traffic, 10netops, 10DBA, 10DC-Ops, and 4 others: Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547 (10jijiki)
[16:42:05] <wikibugs>	 10Traffic, 10Infrastructure-Foundations, 10SRE, 10vm-requests: eqiad: 1 VM request for ncmonitor - https://phabricator.wikimedia.org/T356710 (10BCornwall) 05In progress→03Resolved Thanks!
[16:48:23] <wikibugs>	 10Traffic, 10netops, 10DBA, 10DC-Ops, and 4 others: Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547 (10jijiki)
[17:15:23] <wikibugs>	 10Traffic, 10Security-Team, 10ContentSecurityPolicy, 10Patch-For-Review: Stop sending X-Webkit-CSP and X-Webkit-CSP-Report-Only headers - https://phabricator.wikimedia.org/T357479 (10Fabfur) Thanks @TheDJ for your contribution! The patch has been applied to our hosts!
[18:34:53] <wikibugs>	 10Domains, 10Fundraising-Backlog, 10SRE, 10serviceops: Request donatewiki redirect - https://phabricator.wikimedia.org/T357436 (10Dwisehaupt) @RLazarus Does this really need an apache config patch or just an update to the redirect rules in `hieradata/common/mediawiki.yaml`?
[19:04:18] <wikibugs>	 10Traffic, 10Security-Team, 10ContentSecurityPolicy, 10Patch-For-Review: Stop sending X-Webkit-CSP and X-Webkit-CSP-Report-Only headers - https://phabricator.wikimedia.org/T357479 (10TheDJ) For future reference, the X-Content variant was still in use in IE10-11. IE11 can technically still be used through E...
[19:12:07] <wikibugs>	 10Domains, 10Fundraising-Backlog, 10SRE, 10serviceops: Request donatewiki redirect - https://phabricator.wikimedia.org/T357436 (10RLazarus) Sorry yeah, I was using the term broadly. The goal is to edit the Apache config, but that hieradata file is how you'd do it. :)
[19:19:50] <wikibugs>	 10Traffic, 10netops, 10DBA, 10DC-Ops, and 5 others: Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547 (10lmata)
[19:32:54] <wikibugs>	 10Wikimedia-Apache-configuration, 10Fundraising-Backlog, 10SRE, 10serviceops: Request donatewiki redirect - https://phabricator.wikimedia.org/T357436 (10RLazarus)
[20:10:34] <wikibugs>	 10Traffic, 10Patch-For-Review: Restarting fifo-log-demux should not restart nginx - https://phabricator.wikimedia.org/T355905 (10CodeReviewBot) brett merged https://gitlab.wikimedia.org/repos/sre/fifo-log-demux/-/merge_requests/1  fifo-log-demux: Exit(70) on unable to spawn server
[23:04:27] <wikibugs>	 10Wikimedia-Apache-configuration, 10Fundraising-Backlog, 10SRE, 10fundraising-tech-ops, and 2 others: Request donatewiki redirect - https://phabricator.wikimedia.org/T357436 (10Dwisehaupt)
[23:23:40] <wikibugs>	 10Wikimedia-Apache-configuration, 10serviceops: Investigate restricting match pattern on /wiki RewriteRule - https://phabricator.wikimedia.org/T357595 (10Scott_French)
[23:24:00] <wikibugs>	 10Wikimedia-Apache-configuration, 10serviceops: Investigate restricting match pattern on /wiki RewriteRule - https://phabricator.wikimedia.org/T357595 (10Scott_French) p:05Triage→03Low
[23:28:31] <wikibugs>	 10Wikimedia-Apache-configuration, 10serviceops: Investigate restricting match pattern on /wiki RewriteRule - https://phabricator.wikimedia.org/T357595 (10Scott_French)
[23:56:34] <wikibugs>	 10Wikimedia-Apache-configuration, 10serviceops: Investigate restricting match pattern on /wiki RewriteRule - https://phabricator.wikimedia.org/T357595 (10RLazarus) > the intention was probably for this to match something a bit more restrictive (e.g., matching ^/wiki(/.*)?$)  I'm looking at [[ https://gerrit.wi...