[01:42:48] 10Traffic, 10SRE: Varnish should allow PURGE requests only from a unix domain socket - https://phabricator.wikimedia.org/T347192 (10Vgutierrez) p:05Triage→03Medium [08:43:30] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+ - https://phabricator.wikimedia.org/T316544 (10dcaro) I'm going to start draining nodes from `D5`: cloudcephosd1011 cloudcephosd1012 cloudcephosd1013 cloudcephosd1014 clou... [08:46:56] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+ - https://phabricator.wikimedia.org/T316544 (10cmooney) >>! In T316544#9214514, @dcaro wrote: > I'm going to start draining nodes from `D5`: @dcaro that's great thanks! L... [09:13:15] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+ - https://phabricator.wikimedia.org/T316544 (10dcaro) [09:18:53] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+ - https://phabricator.wikimedia.org/T316544 (10cmooney) In terms of the other nodes in that rack we have the following cloudvirts, and should consider possibly moving insta... [09:20:59] 10Traffic, 10SRE: Varnish should allow PURGE requests only from a unix domain socket - https://phabricator.wikimedia.org/T347192 (10Fabfur) 05Open→03Resolved Mark this as Resolved, following work on the `purged` side will have dedicated tasks. [09:28:30] 10Traffic, 10SRE: Repackage purged for bullseye and bookworm - https://phabricator.wikimedia.org/T347837 (10Fabfur) [09:56:16] 10Traffic, 10SRE: Repackage purged for bullseye and bookworm - https://phabricator.wikimedia.org/T347837 (10Vgutierrez) p:05Triage→03Medium [10:03:49] 10netops, 10Infrastructure-Foundations, 10SRE: Plan codfw row A/B top-of-rack switch refresh - https://phabricator.wikimedia.org/T327938 (10cmooney) [10:04:50] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Export routes generated from ARP/ND in EVPN - https://phabricator.wikimedia.org/T329369 (10cmooney) 05Open→03Resolved Change merged and pushed out to live devices. No change to announced routes on existing devices, e.g. type 5 routes a... [10:05:03] 10Traffic, 10SRE: Add version flag to purged - https://phabricator.wikimedia.org/T347839 (10Fabfur) [10:05:22] 10Traffic, 10SRE: Add version flag to purged - https://phabricator.wikimedia.org/T347839 (10Fabfur) p:05Triage→03Low [10:40:46] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-tools: Netbox - PuppetDB audit 2021-11 - https://phabricator.wikimedia.org/T295762 (10aborrero) [11:02:30] 10Traffic, 10SRE: Repackage purged for bullseye and bookworm - https://phabricator.wikimedia.org/T347837 (10Fabfur) Tests on cp4040.ulsfo.wmnet shows that the new `purged` version connecting to the proper Varnish socket correctly processes the PURGE requests. An example snippet from `varnishlog`: ` * << Re... [11:17:29] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-tools: Netbox - PuppetDB audit 2021-11 - https://phabricator.wikimedia.org/T295762 (10cmooney) [12:39:57] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10cmooney) [12:51:24] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10cmooney) [13:39:58] 10Traffic, 10SRE, 10Patch-For-Review: Repackage purged for bullseye and bookworm - https://phabricator.wikimedia.org/T347837 (10Fabfur) [14:28:08] 10Traffic, 10SRE, 10Patch-For-Review: Repackage purged for bullseye and bookworm - https://phabricator.wikimedia.org/T347837 (10Fabfur) [14:28:22] 10Traffic, 10SRE, 10Patch-For-Review: Repackage purged for bullseye and bookworm - https://phabricator.wikimedia.org/T347837 (10Fabfur) [15:00:04] 10Traffic, 10SRE: Repackage purged for bullseye and bookworm - https://phabricator.wikimedia.org/T347837 (10Fabfur) [15:01:53] 10Traffic, 10SRE: Repackage purged for bullseye and bookworm - https://phabricator.wikimedia.org/T347837 (10Fabfur) [15:59:10] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+ - https://phabricator.wikimedia.org/T316544 (10aborrero) [15:59:54] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+ - https://phabricator.wikimedia.org/T316544 (10aborrero) [16:08:29] 10Traffic, 10SRE, 10Patch-For-Review: Repackage purged for bullseye and bookworm - https://phabricator.wikimedia.org/T347837 (10Fabfur) [16:24:40] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10VRiley-WMF) cp1106 - B 7. U 27. CableID 5021 port 31 cp1107 - B 7. U 28. CableID 5061 port 37 cp1108 - C 7. U 30. CableID 230304500235 port 36 cp1109 - C 7. U 31. CableID 4780 port 28 c... [16:54:05] vgutierrez: great presentation! [16:54:13] where do you plan on sharing the doc you mentioned? [17:01:16] 10Traffic, 10SRE: Repackage purged for bullseye and bookworm - https://phabricator.wikimedia.org/T347837 (10Fabfur) [17:09:28] topranks: email to sre-at-large as soon as the recording link is available [17:15:35] cool, I will keep an eye out :) [17:52:03] 10Traffic, 10SRE, 10Patch-For-Review: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054 (10ssingh) `ntp.anycast.wmnet` exists and the VIP `10.3.0.2/32` is being announced from all DNS hosts. The next step is to merge http... [18:21:51] 10Traffic, 10SRE, 10Patch-For-Review: Alert on Varnish high thread count - https://phabricator.wikimedia.org/T323723 (10BCornwall) @Vgutierrez thoughts on this? Care to rebut? [19:17:06] 10Traffic, 10SRE, 10Patch-For-Review: Add version flag to purged - https://phabricator.wikimedia.org/T347839 (10Fabfur) 05Open→03In progress