[11:06:33] 10serviceops, 10Kubernetes, 10Patch-For-Review: Configure mesh listeners to allow IPv6 localhost (::) as well as IPv4 (127.0.0.1) - https://phabricator.wikimedia.org/T355686 (10akosiaris) I 've posted changes to revert the hardcoded of localhost to 127.0.0.1. I 've already deployed eventstreams and recommend... [11:17:50] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team, 10SRE, 10Traffic: Move 50% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T357507 (10Clement_Goubert) [11:20:14] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team, 10SRE, 10Traffic: Move 50% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T357507 (10Clement_Goubert) p:05Triage→03High [11:20:57] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team, 10SRE, 10Traffic: Move 60% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T357508 (10Clement_Goubert) [11:21:33] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team, 10SRE, 10Scap: Scap should check errors coming from mw-on-k8s canaries during deployments - https://phabricator.wikimedia.org/T357402 (10Clement_Goubert) [11:21:41] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team, 10SRE, 10Traffic: Move 60% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T357508 (10Clement_Goubert) [11:23:45] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team, 10SRE, 10Scap: Scap should check errors coming from mw-on-k8s canaries during deployments - https://phabricator.wikimedia.org/T357402 (10Clement_Goubert) [11:23:51] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [11:25:31] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team, 10SRE, 10Traffic: Move 60% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T357508 (10Clement_Goubert) 05Open→03Stalled p:05Triage→03High We'll stay at 50% or under because we want scap to check canaries before... [11:26:53] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [11:42:44] 10serviceops, 10MW-on-K8s, 10MediaWiki-Platform-Team (Radar), 10Patch-For-Review: mcrouter daemonset on mw-on-k8s - https://phabricator.wikimedia.org/T346690 (10Clement_Goubert) Todo after testing (just dumping here so we don't forget): - Move configuration to the `mw` namespace in the chart - Amend commen... [11:44:03] 10serviceops, 10MW-on-K8s: Allow php-fpm to read environment variables from the system, not just from the fcgi request - https://phabricator.wikimedia.org/T326705 (10Clement_Goubert) Todo after testing (just dumping here so we don't forget): - Move configuration to the `mw` namespace in the chart - Amend comme... [12:05:09] 10serviceops, 10CX-cxserver, 10Citoid, 10Content-Transform-Team-WIP, and 11 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118 (10Lucas_Werkmeister_WMDE) [12:06:07] 10serviceops, 10Data-Engineering, 10Wikidata, 10Wikidata-Termbox, and 4 others: Migrate Termbox SSR from Node 16 to 18 - https://phabricator.wikimedia.org/T355685 (10Lucas_Werkmeister_WMDE) 05Open→03Resolved a:03Lucas_Werkmeister_WMDE I deployed the update and it’s working as far as I can tell – I th... [12:42:28] 10serviceops, 10Patch-For-Review, 10iPoid-Service (iPoid 1.0): Determine cause of HTTP 503 errors for ~8% of MediaWiki requests to ipoid service - https://phabricator.wikimedia.org/T356766 (10jijiki) >>! In T356766#9537892, @kostajh wrote: >>>! In T356766#9537874, @jijiki wrote: >> * meanwhile, we restarte... [12:51:54] 10serviceops, 10Domains, 10Fundraising-Backlog, 10SRE: Request donatewiki redirect - https://phabricator.wikimedia.org/T357436 (10AKanji-WMF) Thank you @RLazarus ! The fundraising team would like the redirect to be active for two years - considering the life cycle of the podcast and surrounding marketing c... [13:42:44] 10serviceops, 10Patch-For-Review, 10iPoid-Service (iPoid 1.0): Determine cause of HTTP 503 errors for ~8% of MediaWiki requests to ipoid service - https://phabricator.wikimedia.org/T356766 (10jijiki) [14:26:05] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host mw2311.codfw.wmnet with OS bullseye [14:26:10] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host mw2380.codfw.wmnet with OS bullseye [14:26:12] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host mw2379.codfw.wmnet with OS bullseye [14:26:14] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host mw2383.codfw.wmnet with OS bullseye [14:26:18] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host mw2335.codfw.wmnet with OS bullseye [14:28:37] 10serviceops, 10RESTBase Sunsetting, 10Epic, 10Parsoid (Tracking): Prepare amount of workers to handle enwiki traffic for parsoid endpoints - https://phabricator.wikimedia.org/T357504 (10Clement_Goubert) [14:33:18] 10serviceops, 10RESTBase Sunsetting, 10Epic, 10Parsoid (Tracking): Prepare amount of workers to handle enwiki traffic for parsoid endpoints - https://phabricator.wikimedia.org/T357504 (10hnowlan) To limit the impact of this rollout we'll be gradually deploying to restbase servers either in small numbers or... [14:43:05] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host mw2383.codfw.wmnet with OS bullseye executed with errors: - mw2383 (**FAIL**) - Do... [14:43:16] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host mw2379.codfw.wmnet with OS bullseye executed with errors: - mw2379 (**FAIL**) - Do... [14:43:24] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host mw2380.codfw.wmnet with OS bullseye executed with errors: - mw2380 (**FAIL**) - Do... [14:44:37] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host mw2380.codfw.wmnet with OS bullseye [14:45:02] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host mw2379.codfw.wmnet with OS bullseye [14:52:21] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host mw2383.codfw.wmnet with OS bullseye executed with errors: - mw2383 (**FAIL**) - Re... [14:52:25] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host mw2379.codfw.wmnet with OS bullseye executed with errors: - mw2379 (**FAIL**) - Re... [14:52:29] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host mw2380.codfw.wmnet with OS bullseye executed with errors: - mw2380 (**FAIL**) - Re... [14:57:36] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host mw2379.codfw.wmnet with OS bullseye [14:59:49] 10serviceops, 10Domains, 10Fundraising-Backlog, 10SRE: Request donatewiki redirect - https://phabricator.wikimedia.org/T357436 (10AKanji-WMF) Tagging you @Dwisehaupt as Jeff is out this AM. [15:04:02] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host mw2335.codfw.wmnet with OS bullseye completed: - mw2335 (**PASS**) - Downtimed on... [15:06:09] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host mw2379.codfw.wmnet with OS bullseye executed with errors: - mw2379 (**FAIL**) - Re... [15:06:25] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host mw2379.codfw.wmnet with OS bullseye [15:07:55] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host mw2311.codfw.wmnet with OS bullseye completed: - mw2311 (**PASS**) - Downtimed on... [15:09:11] 10serviceops, 10SRE, 10observability: Cert renewal for {appserver,api}.svc.{eqiad,codfw}.wmnet - https://phabricator.wikimedia.org/T304237 (10lmata) 05Open→03Resolved a:03lmata I will boldly resolve this. I discussed this with the team, and we agreed the strategy here is to renew/purchase dates to be m... [15:11:42] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host mw2379.codfw.wmnet with OS bullseye executed with errors: - mw2379 (**FAIL**) - Re... [15:14:12] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host mw2380.codfw.wmnet with OS bullseye [15:21:37] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host mw2380.codfw.wmnet with OS bullseye executed with errors: - mw2380 (**FAIL**) - Re... [15:24:07] 10serviceops, 10ops-codfw: Multiple hosts in codfw fail to PXE boot upon reimage - https://phabricator.wikimedia.org/T357539 (10hnowlan) [15:26:58] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Possible firmware issues reimaging mw2282 - https://phabricator.wikimedia.org/T355333 (10hnowlan) >>! In T355333#9538260, @Jhancock.wm wrote: > idk if this would help, but can we run the provisioning script with the --no-dhcp and --no-user tags. to catch any bios... [15:35:32] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Possible firmware issues reimaging mw2282 - https://phabricator.wikimedia.org/T355333 (10Jhancock.wm) sre.hosts.provision --no-dhcp --no-user [15:36:37] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Possible firmware issues reimaging mw2282 - https://phabricator.wikimedia.org/T355333 (10Volans) >>! In T355333#9542804, @Jhancock.wm wrote: > sre.hosts.provision --no-dhcp --no-user Also `--no-switch` in this case I'd say. [15:52:45] 10serviceops, 10Patch-For-Review, 10iPoid-Service (iPoid 1.0): Determine cause of HTTP 503 errors for ~8% of MediaWiki requests to ipoid service - https://phabricator.wikimedia.org/T356766 (10jijiki) 05Open→03Resolved a:03jijiki Things are progressing well after the last change, please reopen if this r... [15:58:02] 10serviceops, 10Patch-For-Review, 10iPoid-Service (iPoid 1.0): Determine cause of HTTP 503 errors for ~8% of MediaWiki requests to ipoid service - https://phabricator.wikimedia.org/T356766 (10kostajh) Thank you for your help with this. [15:59:43] 10serviceops, 10MW-on-K8s, 10SRE, 10Scap, 10Release-Engineering-Team (Now this 🫠): Scap should check errors coming from mw-on-k8s canaries during deployments - https://phabricator.wikimedia.org/T357402 (10thcipriani) [16:15:07] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Possible firmware issues reimaging mw2282 - https://phabricator.wikimedia.org/T355333 (10hnowlan) This command also fails - but interestingly the host itself appears to have lost network connectivity. `ethtool` reports that the link is up but I can't connect in o... [16:15:37] claime: all those moves are complete in A5 if you want to repool things [16:15:49] fantastic thanks [16:36:00] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Possible firmware issues reimaging mw2282 - https://phabricator.wikimedia.org/T355333 (10cmooney) >>! In T355333#9543167, @hnowlan wrote: > This command also fails - but interestingly the host itself appears to have lost network connectivity. `ethtool` reports th... [16:36:05] 10serviceops, 10DBA, 10DC-Ops, 10Infrastructure-Foundations, and 4 others: Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547 (10jijiki) [16:37:48] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Possible firmware issues reimaging mw2282 - https://phabricator.wikimedia.org/T355333 (10cmooney) The reimage problem may be the firmware issue - link not coming up during the debian installer. @hnowlan if you want to try the reimage again I can take a look at t... [16:39:14] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Possible firmware issues reimaging mw2282 - https://phabricator.wikimedia.org/T355333 (10Jhancock.wm) port shows activity on the server, but the network side is showing as down. Reseating either cable does nothing. but reseating the SFP makes it come back up Pos... [16:39:59] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Possible firmware issues reimaging mw2282 - https://phabricator.wikimedia.org/T355333 (10Jhancock.wm) @cmooney it was me, I was reseating the cable [16:41:32] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Possible firmware issues reimaging mw2282 - https://phabricator.wikimedia.org/T355333 (10cmooney) >>! In T355333#9543231, @Jhancock.wm wrote: > port shows activity on the server, but the network side is showing as down. Reseating either cable does nothing. but re... [16:48:21] 10serviceops, 10DBA, 10DC-Ops, 10Infrastructure-Foundations, and 4 others: Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547 (10jijiki) [16:54:13] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Possible firmware issues reimaging mw2282 - https://phabricator.wikimedia.org/T355333 (10hnowlan) Reimaging fails still after these changes fwiw - however, a reboot has broken network connectivity again?! The host is up and rebooted in the management interface, b... [16:59:27] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Possible firmware issues reimaging mw2282 - https://phabricator.wikimedia.org/T355333 (10Jhancock.wm) replaced the SFP this time. came up. server reboot is causing the port to go down, possibly [17:16:01] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Possible firmware issues reimaging mw2282 - https://phabricator.wikimedia.org/T355333 (10cmooney) >>! In T355333#9543255, @hnowlan wrote: > Reimaging fails still after these changes fwiw - however, a reboot has broken network connectivity again?! The host is up a... [17:31:35] 10serviceops, 10CirrusSearch, 10Discovery-Search (Current work), 10MW-1.42-notes (1.42.0-wmf.17; 2024-02-06), 10Patch-For-Review: High level of backend errors for CirrusSearch jobs in jobrunners - https://phabricator.wikimedia.org/T356526 (10EBernhardson) This looks resolved now, the bi-hourly spikes hav... [17:32:33] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Possible firmware issues reimaging mw2282 - https://phabricator.wikimedia.org/T355333 (10hnowlan) I tried another reimage and it currently proceeding successfully - maybe replacing the SFP did the job? This is all a bit inexplicable. [17:41:33] 10serviceops, 10SRE, 10ops-codfw: Multiple hosts in codfw fail to PXE boot upon reimage - https://phabricator.wikimedia.org/T357539 (10cmooney) All of these are connected to lsw1-a3-codfw (new L3 switch) and they may be the first we've tried to reimage connected to new switch. Investigating if it is related... [17:50:39] 10serviceops, 10SRE, 10ops-codfw: Multiple hosts in codfw fail to PXE boot upon reimage - https://phabricator.wikimedia.org/T357539 (10cmooney) I think what's happening is the new switch is not configured to insert the port information for DHCP requests over the legacy row-wide vlan. Best way forward is to... [18:11:25] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Possible firmware issues reimaging mw2282 - https://phabricator.wikimedia.org/T355333 (10hnowlan) 05Open→03Resolved a:03hnowlan Reimage was successful, networking survived a reboot. All done! [18:12:13] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1002 for host mw2379.codfw.wmnet with OS bullseye [18:34:52] 10serviceops, 10Domains, 10Fundraising-Backlog, 10SRE: Request donatewiki redirect - https://phabricator.wikimedia.org/T357436 (10Dwisehaupt) @RLazarus Does this really need an apache config patch or just an update to the redirect rules in `hieradata/common/mediawiki.yaml`? [18:43:13] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Possible firmware issues reimaging mw2282 - https://phabricator.wikimedia.org/T355333 (10Jhancock.wm) I will mark the SFP I pulled as bad. See if I can test it on a new server. [18:53:35] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1002 for host mw2380.codfw.wmnet with OS bullseye [18:54:20] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1002 for host mw2383.codfw.wmnet with OS bullseye [18:58:29] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1002 for host mw2379.codfw.wmnet with OS bullseye completed: - mw2379 (**WARN**) - Removed from... [19:12:06] 10serviceops, 10Domains, 10Fundraising-Backlog, 10SRE: Request donatewiki redirect - https://phabricator.wikimedia.org/T357436 (10RLazarus) Sorry yeah, I was using the term broadly. The goal is to edit the Apache config, but that hieradata file is how you'd do it. :) [19:19:48] 10serviceops, 10DBA, 10DC-Ops, 10Infrastructure-Foundations, and 5 others: Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547 (10lmata) [19:32:52] 10serviceops, 10Fundraising-Backlog, 10SRE, 10Wikimedia-Apache-configuration: Request donatewiki redirect - https://phabricator.wikimedia.org/T357436 (10RLazarus) [19:35:59] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1002 for host mw2380.codfw.wmnet with OS bullseye completed: - mw2380 (**WARN**) - Removed from... [19:36:27] 10serviceops, 10SRE, 10ops-codfw: Multiple hosts in codfw fail to PXE boot upon reimage - https://phabricator.wikimedia.org/T357539 (10cmooney) 05Open→03Resolved a:03cmooney Yeah the issue here was the hosts being connected to the new switches, but still configured for the legacy vlan. That's fine, bu... [19:37:26] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1002 for host mw2383.codfw.wmnet with OS bullseye completed: - mw2383 (**WARN**) - Removed from... [19:38:15] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10cmooney) >>! In T351074#9543818, @ops-monitoring-bot wrote: >Icinga status is not optimal, downtime not removed This was the ferm service check, seems ferm couldn't resolve a dns... [23:04:25] 10serviceops, 10Fundraising-Backlog, 10SRE, 10Wikimedia-Apache-configuration, and 2 others: Request donatewiki redirect - https://phabricator.wikimedia.org/T357436 (10Dwisehaupt) [23:23:40] 10serviceops, 10Wikimedia-Apache-configuration: Investigate restricting match pattern on /wiki RewriteRule - https://phabricator.wikimedia.org/T357595 (10Scott_French) [23:24:00] 10serviceops, 10Wikimedia-Apache-configuration: Investigate restricting match pattern on /wiki RewriteRule - https://phabricator.wikimedia.org/T357595 (10Scott_French) p:05Triage→03Low [23:28:31] 10serviceops, 10Wikimedia-Apache-configuration: Investigate restricting match pattern on /wiki RewriteRule - https://phabricator.wikimedia.org/T357595 (10Scott_French) [23:56:34] 10serviceops, 10Wikimedia-Apache-configuration: Investigate restricting match pattern on /wiki RewriteRule - https://phabricator.wikimedia.org/T357595 (10RLazarus) > the intention was probably for this to match something a bit more restrictive (e.g., matching ^/wiki(/.*)?$) I'm looking at [[ https://gerrit.wi...