[07:32:15] 10Traffic, 10SRE: reprovision ping VM in esams - https://phabricator.wikimedia.org/T345743 (10ayounsi) [07:32:23] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Do we need ping offload servers at all POPs? - https://phabricator.wikimedia.org/T345809 (10ayounsi) [09:46:13] 10Traffic: Package and deploy ATS 9.2.1 - https://phabricator.wikimedia.org/T339134 (10Vgutierrez) that's not an issue with ATS 9.2.1, problem got fixed by restarting fifo-log-demux@notpurge.service: ` vgutierrez@cp4052:~$ journalctl -u fifo-log-demux@notpurge.service -f -- Journal begins at Sat 2023-08-05 22:39... [09:52:46] 10Traffic: Package and deploy ATS 9.2.1 - https://phabricator.wikimedia.org/T339134 (10Vgutierrez) A quick check with cumin shows several servers impacted: ` vgutierrez@cumin1001:~$ sudo -i cumin 'A:cp' 'journalctl -u fifo-log-demux@notpurge.service --since=-1h | grep Error' 96 hosts will be targeted: cp[2027-20... [12:48:42] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Do we need ping offload servers at all POPs? - https://phabricator.wikimedia.org/T345809 (10BBlack) > to reduce load on LVS hosts My recollection is that it wasn't really about raw load or PPS at the LVSes. It was that our Linux kernel settings ha... [12:52:40] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Do we need ping offload servers at all POPs? - https://phabricator.wikimedia.org/T345809 (10BBlack) The current puppetized tuneables are at: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/8ed59718c7a7603b61d7d42e05726fd11dae5eaa/... [12:57:53] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Do we need ping offload servers at all POPs? - https://phabricator.wikimedia.org/T345809 (10BBlack) Reading into the code above and the history more and self-correcting: the ratelimiter doesn't apply to PTB packets, just some other informational pac... [13:35:59] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Plan codfw row A/B top-of-rack switch refresh - https://phabricator.wikimedia.org/T327938 (10cmooney) @papaul I've done some testing and I'm confident the IP GW moves for the row subnets to the Spines can be done gracefully. I've yet to wo... [13:44:18] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Plan codfw row A/B top-of-rack switch refresh - https://phabricator.wikimedia.org/T327938 (10Papaul) @cmooney thanks for the update. I think we can reuse those the MPO [17:59:35] Hi traffic! [18:01:36] Do you build Wikipedia DNS? [18:01:48] *wikimedia [18:02:53] Guest57: hi, yes, this team does [18:54:49] 10Traffic: Create metrics/monitoring of fifo-log-demux - https://phabricator.wikimedia.org/T345939 (10BCornwall) [18:55:18] 10Traffic: Create metrics/monitoring of fifo-log-demux - https://phabricator.wikimedia.org/T345939 (10BCornwall) p:05Triage→03Medium