[13:17:24] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-OnFire, 10Sustainability (Incident Followup): Upgrade POPs asw to Junos 21 - https://phabricator.wikimedia.org/T316532 (10ayounsi) [14:15:08] 10Traffic, 10SRE: Drop the VarnishTrafficDrop and HAProxyEdgeTrafficDrop alerts - https://phabricator.wikimedia.org/T322220 (10fgiunchedi) [14:20:19] 10Traffic, 10Infrastructure-Foundations, 10SRE: netbox-exports git cloning perf issues - https://phabricator.wikimedia.org/T324334 (10ssingh) >>! In T324334#8439896, @Volans wrote: > Sorry for the trouble, that was me indeed, I've fixed the permissions and run the `sre.dns.netbox` cookbook successfully: > >... [14:57:41] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install/decom eqsin: unified decommission task - https://phabricator.wikimedia.org/T323830 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: `cp[5011,5013].eqsin.wmnet` - cp5011.eqsin.w... [14:57:54] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install/decom eqsin: unified decommission task - https://phabricator.wikimedia.org/T323830 (10ssingh) [15:05:02] 10Traffic, 10SRE: Prometheus Varnish exporter alert: add runbook and link to dashboard - https://phabricator.wikimedia.org/T289974 (10fgiunchedi) [15:45:51] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install/decom eqsin: unified decommission task - https://phabricator.wikimedia.org/T323830 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: `cp[5012,5014].eqsin.wmnet` - cp5012.eqsin.w... [15:45:57] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install/decom eqsin: unified decommission task - https://phabricator.wikimedia.org/T323830 (10ssingh) [15:48:18] 10Traffic, 10SRE, 10Patch-For-Review: haproxy: work on systemd unit hardening (cp hosts) - https://phabricator.wikimedia.org/T323944 (10ssingh) We have enabled the hardened haproxy unit on `traffic-cache-bullseye.traffic.eqiad1.wikimedia.cloud` to start with, before rolling it out to the production cp hosts. [16:58:05] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install/decom eqsin: unified decommission task - https://phabricator.wikimedia.org/T323830 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: `cp5015.eqsin.wmnet` - cp5015.eqsin.wmnet (*... [16:58:14] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install/decom eqsin: unified decommission task - https://phabricator.wikimedia.org/T323830 (10ssingh) [17:42:17] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install/decom eqsin: unified decommission task - https://phabricator.wikimedia.org/T323830 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: `cp5016.eqsin.wmnet` - cp5016.eqsin.wmnet (*... [17:42:34] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install/decom eqsin: unified decommission task - https://phabricator.wikimedia.org/T323830 (10ssingh) [21:17:09] 10netops, 10Infrastructure-Foundations, 10SRE: cr2-esams:FPC0 Parity error - https://phabricator.wikimedia.org/T318783 (10ayounsi) 05Resolved→03Open It's back :( ` cr2-esams> show system alarms 1 alarms currently active Alarm time Class Description 2022-12-05 18:15:58 UTC Minor FPC 0 M... [23:08:14] 10Traffic, 10Data-Engineering-Planning, 10Observability-Alerting, 10Shared-Data-Infrastructure: Reduce/eliminate false positives for VarnishKafkaNoMessages alert - https://phabricator.wikimedia.org/T324522 (10BTullis)