[05:17:57] 06Traffic, 10Hiddenparma, 13Patch-For-Review: Requestctl should use x-provenance header - https://phabricator.wikimedia.org/T396621#11063276 (10Joe) 05Resolved→03Open a:05Fabfur→03Joe You missed changing the use of the abuse ACLs and removing the loading of netmaps for X-Public-Cloud and similar stuf... [06:02:28] 06Traffic, 10Hiddenparma, 13Patch-For-Review: Requestctl should use x-provenance header - https://phabricator.wikimedia.org/T396621#11063290 (10Joe) p:05Triage→03High [09:13:25] 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11063973 (10Joe) [09:36:47] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Move pfw1b-codfw to rack F5 - https://phabricator.wikimedia.org/T401297 (10ayounsi) 03NEW [12:46:27] All right back testing late_command on cp2043, I see from /var/log/syslog in d-i's shell: [12:46:30] Aug 6 12:39:02 in-target: NVMe status: Invalid Format: The LBA Format specified [12:46:30] is not supported(0x210a) [12:48:31] from https://github.com/linux-nvme/nvme-cli/issues/2254 it seems that people got around it with a firmware update [12:53:30] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11065013 (10elukey) To keep archives happy, late_command.sh fails. Reporting what I wrote on IRC to the Traffic team: ` All right back testing late_command on... [13:38:32] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Move pfw1b-codfw to rack F5 - https://phabricator.wikimedia.org/T401297#11065301 (10Papaul) Just by reading the task, the first thing we will need before moving the firewall will be a SFP Transceiver 1000Base-LX, 1310nm SMF SingleMode Fiber Opti... [14:51:40] FIRING: VarnishHighThreadCount: Varnish's thread count on cp5027:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=eqsin&var-instance=cp5027 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:06:40] RESOLVED: VarnishHighThreadCount: Varnish's thread count on cp5027:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=eqsin&var-instance=cp5027 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [18:43:42] 06Traffic, 06SRE, 10SRE-SLO: Page on ATS backend errors relative to traffic - https://phabricator.wikimedia.org/T400675#11066418 (10RLazarus) We talked about this in the SLO meeting today -- one possible approach is to keep `ATSBackendErrorsHigh` as a default policy, but keep a list of services to //exclude/...