[06:03:56] <wikibugs>	 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Migrate servers in codfw rack B3 from asw-b3-codfw to lsw1-b3-codfw - https://phabricator.wikimedia.org/T355870#9554047 (10Marostegui)
[07:25:19] <wikibugs>	 10Traffic, 10netops, 10DC-Ops, 10Data-Persistence, and 5 others: Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547#9554129 (10Marostegui)
[10:33:53] <jinxer-wm>	 (PuppetZeroResources) firing: Puppet has failed generate resources on cp2029:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[10:36:49] <jinxer-wm>	 (PuppetZeroResources) firing: Puppet has failed generate resources on doh2002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[10:37:49] <jinxer-wm>	 (PuppetZeroResources) firing: Puppet has failed generate resources on cp2036:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[10:38:53] <jinxer-wm>	 (PuppetZeroResources) firing: (2) Puppet has failed generate resources on cp2029:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[10:39:48] <jinxer-wm>	 (PuppetZeroResources) firing: Puppet has failed generate resources on dns2006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[10:40:49] <jinxer-wm>	 (PuppetZeroResources) firing: Puppet has failed generate resources on ncredir2002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[10:48:24] <jinxer-wm>	 (PuppetZeroResources) firing: (2) Puppet has failed generate resources on cp2030:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[10:48:59] <jinxer-wm>	 (PuppetZeroResources) firing: (2) Puppet has failed generate resources on cp2029:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[10:51:48] <jinxer-wm>	 (PuppetZeroResources) firing: (2) Puppet has failed generate resources on doh1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[10:52:49] <jinxer-wm>	 (PuppetZeroResources) firing: (2) Puppet has failed generate resources on cp2030:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[10:53:49] <jinxer-wm>	 (PuppetZeroResources) firing: (3) Puppet has failed generate resources on cp2029:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[10:54:48] <jinxer-wm>	 (PuppetZeroResources) resolved: Puppet has failed generate resources on dns2006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[10:55:57] <jinxer-wm>	 (PuppetZeroResources) firing: Puppet has failed generate resources on dns2004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[10:58:06] <jinxer-wm>	 (PuppetZeroResources) resolved: (2) Puppet has failed generate resources on cp2030:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[11:00:53] <jinxer-wm>	 (PuppetZeroResources) firing: (2) Puppet has failed generate resources on dns2004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[11:01:49] <jinxer-wm>	 (PuppetZeroResources) firing: (3) Puppet has failed generate resources on doh1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[11:01:53] <jinxer-wm>	 (PuppetZeroResources) firing: Puppet has failed generate resources on lvs2014:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[11:03:49] <jinxer-wm>	 (PuppetZeroResources) firing: (3) Puppet has failed generate resources on cp2029:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[11:10:57] <jinxer-wm>	 (PuppetZeroResources) resolved: Puppet has failed generate resources on dns2004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[11:11:31] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team, and 2 others: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184#9554542 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1032.eqiad.wmnet with OS...
[11:11:49] <jinxer-wm>	 (PuppetZeroResources) resolved: Puppet has failed generate resources on lvs2014:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[11:11:57] <jinxer-wm>	 (PuppetZeroResources) firing: (3) Puppet has failed generate resources on doh1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[11:12:02] <jinxer-wm>	 (PuppetZeroResources) firing: Puppet has failed generate resources on cp1103:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[11:18:53] <jinxer-wm>	 (PuppetZeroResources) resolved: (2) Puppet has failed generate resources on cp2029:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[11:20:48] <jinxer-wm>	 (PuppetZeroResources) resolved: Puppet has failed generate resources on ncredir2002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[12:04:02] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team, and 2 others: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184#9554726 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1032.eqiad.wmnet with OS book...
[12:32:05] <wikibugs>	 10Traffic, 10Automoderator, 10Data Products, 10Product-Analytics, and 3 others: Add revision ID to X-Analytics header - https://phabricator.wikimedia.org/T346350#9554819 (10phuedx) > Are there situations where the revision id can be 0 on a page view?  Yes. The patch doesn't add the revision ID of the page...
[12:55:27] <wikibugs>	 10Traffic, 10Automoderator, 10Data Products, 10Product-Analytics, and 3 others: Add revision ID to X-Analytics header - https://phabricator.wikimedia.org/T346350#9554876 (10phuedx) Here's an example demonstrating the behaviour of the patch:  | URL | rev_id | | --- | --- | | https://en.wikipedia.org/wiki/Ig...
[14:20:53] <wikibugs>	 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120#9555136 (10Clement_Goubert)
[16:08:54] <wikibugs>	 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9555525 (10cmooney) p:05Triage→03Low
[16:09:10] <wikibugs>	 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Move lvs2011 from private1-a-codfw (row) to private1-a2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352920#9555526 (10cmooney) p:05Triage→03Low
[16:12:45] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10sre-alert-triage: Alert in need of triage: BGP status (instance cr1-drmrs) - https://phabricator.wikimedia.org/T357389#9555538 (10ayounsi) p:05Triage→03Low a:03ayounsi
[17:16:28] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Update K8S BGP groups eqiad row e-f - https://phabricator.wikimedia.org/T357924#9555767 (10cmooney) p:05Triage→03Medium
[17:16:36] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Update K8S BGP groups eqiad row e-f - https://phabricator.wikimedia.org/T357924#9555777 (10cmooney)
[17:16:44] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: BGP peering from LSW to K8s hosts using loopback IP not IRB - https://phabricator.wikimedia.org/T357619#9555778 (10cmooney)
[17:16:52] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Update K8S BGP groups eqiad row e-f - https://phabricator.wikimedia.org/T357924#9555767 (10cmooney)
[17:21:23] <wikibugs>	 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Move MediaWiki jobs to mw-on-k8s - https://phabricator.wikimedia.org/T349796#9555793 (10hnowlan)
[17:21:31] <wikibugs>	 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Move MediaWiki jobs to mw-on-k8s - https://phabricator.wikimedia.org/T349796#9352065 (10hnowlan)
[17:24:01] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Update K8S BGP groups eqiad row e-f - https://phabricator.wikimedia.org/T357924#9555808 (10cmooney)
[18:28:51] <wikibugs>	 10Wikimedia-Apache-configuration, 10Security-Team, 10WMF-General-or-Unknown, 10Documentation, and 2 others: Add security.txt to Wikimedia sites? (2023 edition) - https://phabricator.wikimedia.org/T337949#9555963 (10mmartorana) a:05mmartorana→03None >>! In T337949#9497580, @MoritzMuehlenhoff wrote: >>>!...
[19:19:22] <wikibugs>	 10Traffic, 10Automoderator, 10Data Products, 10Product-Analytics, and 3 others: Add revision ID to X-Analytics header - https://phabricator.wikimedia.org/T346350#9556093 (10mpopov) Oh that's brilliant! Thanks so much for looking into it and shining light on this @phuedx!
[20:31:51] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Update K8S BGP groups eqiad row e-f - https://phabricator.wikimedia.org/T357924#9556205 (10cmooney) > We will need to arrange a window to push this out to the devices with service ops.  I will discuss with them, but I think the easiest way forward may be to make...