[07:03:11] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10Marostegui) [07:10:33] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10Marostegui) [08:42:38] 10netops, 10Infrastructure-Foundations, 10SRE: Juniper RA receive bug CVE-2023-28981 - https://phabricator.wikimedia.org/T334916 (10cmooney) p:05Triage→03Low [08:52:28] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 3 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Clement_Goubert) [09:50:28] 10netops, 10Infrastructure-Foundations, 10SRE: Juniper RA receive bug CVE-2023-28981 - https://phabricator.wikimedia.org/T334916 (10ayounsi) Agreed, the workaround sgtm! [09:52:11] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10MoritzMuehlenhoff) [09:53:36] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10MoritzMuehlenhoff) [09:56:28] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ArielGlenn) [10:17:29] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10eoghan) [10:23:55] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10MoritzMuehlenhoff) [10:26:43] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10MoritzMuehlenhoff) [10:31:08] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10MoritzMuehlenhoff) [10:39:03] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10jbond) [10:44:40] 10netops, 10Infrastructure-Foundations, 10SRE: Juniper RA receive bug CVE-2023-28981 - https://phabricator.wikimedia.org/T334916 (10Peachey88) [11:51:20] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) jiji@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all active/active services in eqiad: eqiad row D switches upgrade - T333377 started. [11:54:32] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10BTullis) [11:58:18] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10BTullis) [11:58:52] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10hnowlan) [12:21:34] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ssingh) [12:25:43] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) jiji@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all active/active services in eqiad: eqiad row D switches upgrade - T333377 failed. [12:27:36] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) jiji@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all active/active services in eqiad: eqiad row D switches upgrade - T333377 started. [12:27:52] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) jiji@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all active/active services in eqiad: eqiad row D switches upgrade - T333377 compl... [13:08:53] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Codfw:row A/B: rack/cable new switches - https://phabricator.wikimedia.org/T332180 (10Papaul) [13:11:34] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=7fc7ae6f-d3b2-43ed-b030-194ed6367c80) set by cmooney@cumin1001 for 2:00:00 on 270 host(s) and their serv... [13:12:18] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10cmooney) [13:17:11] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=e714b564-285e-4f22-b860-267d7c23208d) set by cmooney@cumin1001 for 2:00:00 on 1 host(s) and their servic... [13:21:51] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10klausman) [13:40:48] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ssingh) [13:42:21] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10Marostegui) dbproxy[1016-1017] reloaded [13:50:46] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10klausman) [13:52:50] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10MoritzMuehlenhoff) [14:01:29] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10cmooney) [14:05:35] 10Traffic, 10SRE-OnFire, 10conftool, 10serviceops, 10Sustainability (Incident Followup): Pybal maintenances break safe-service-restart.py (and thus prevent scap deploys of mediawiki) - https://phabricator.wikimedia.org/T334703 (10MatthewVernon) [14:27:48] 10Traffic, 10SRE, 10Patch-For-Review: Check if it still makes sense to have 8 varnish sockets being used by HAProxy - https://phabricator.wikimedia.org/T333965 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez [14:54:06] 10netops, 10Infrastructure-Foundations, 10SRE-Sprint-Week-Sustainability-March2023, 10Sustainability (Incident Followup): Cr1-eqiad comms problem when moving to 40G row D handoff - https://phabricator.wikimedia.org/T320566 (10ayounsi) [14:54:19] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ayounsi) [15:08:15] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T333377 started. [15:14:37] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10BTullis) [15:38:45] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T333377 failed. [15:39:17] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T333377 started. [15:54:58] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T333377 failed. [15:55:14] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T333377 started. [16:00:25] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T333377 failed. [16:00:41] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T333377 started. [16:02:46] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T333377 failed. [16:04:05] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T333377 started. [16:04:28] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T333377 failed. [16:08:26] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T333377 started. [16:08:47] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T333377 completed. [16:55:16] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw: Q4:rack/setup/install dns200[345] - https://phabricator.wikimedia.org/T326688 (10Jhancock.wm) [17:04:05] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10cmooney) 05Open→03Resolved All works complete, no issues to report. [17:04:25] 10netops, 10Infrastructure-Foundations, 10SRE: eqiad/codfw virtual-chassis upgrades - https://phabricator.wikimedia.org/T327248 (10cmooney) [17:04:37] 10netops, 10Infrastructure-Foundations, 10SRE-Sprint-Week-Sustainability-March2023, 10Sustainability (Incident Followup): Cr1-eqiad comms problem when moving to 40G row D handoff - https://phabricator.wikimedia.org/T320566 (10cmooney) [17:36:00] 10Traffic, 10netops, 10DBA, 10Data-Engineering, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10colewhite) [18:08:10] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10Jclark-ctr) Verified cables for both Servers below are the ports and cable ids @Papaul cloudswift1001 Rack,C8 U35. port {cloudsw1-... [18:23:17] 10Traffic, 10Infrastructure-Foundations, 10SRE: Updating Netbox for LVS hosts in eqiad lvs10(1[789]|20) - https://phabricator.wikimedia.org/T334884 (10cmooney) @ssingh thanks for the heads up. The renamed interfaces are definitely a bit of a headache here. Testing in netbox-next for lvs1020 I see two basi... [18:30:13] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10Papaul) @Jclark-ctr thanks [19:28:40] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10Papaul) @Jclark-ctr there is no network cable connected to both nodes. [20:10:03] 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox, 10Patch-For-Review: Represent sub-interface and bridge device assocations in Netbox - https://phabricator.wikimedia.org/T296832 (10cmooney) @ayounsi has been very helpful with reviewing this patch and it now has a tentative +1 (yay!) In terms of n... [23:02:44] (VarnishHighThreadCount) firing: (14) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [23:07:44] (VarnishHighThreadCount) firing: (17) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [23:17:44] (VarnishHighThreadCount) firing: (17) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [23:27:44] (VarnishHighThreadCount) firing: (16) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [23:47:44] (VarnishHighThreadCount) firing: (28) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount