[08:01:51] hey folks! I have a change for service.yaml that shouldn't require any pybal restart etc.., https://gerrit.wikimedia.org/r/c/operations/puppet/+/1133848 lemme know if those are ok to be merged [08:02:00] * vgutierrez looking [08:03:11] elukey: can you provide a PCC against low-traffic LVS boxes? [08:04:25] sure yes [08:53:20] 06Traffic, 13Patch-For-Review: Private TLS material (TLS keys) should be stored in volatile storage only - https://phabricator.wikimedia.org/T384227#10716772 (10Fabfur) [09:40:04] elukey: ping? :) [09:42:57] vgutierrez: I was debugging deployment failures till 5 mins ago :) [09:43:45] all right so it seems no diffs for LVS/pybal, gooood [09:43:54] I'll proceed, thanks a lot for the review <3 [10:59:33] 06Traffic, 06Data-Persistence, 06SRE, 10SRE-swift-storage, and 6 others: Change default image thumbnail size - https://phabricator.wikimedia.org/T355914#10717142 (10Ladsgroup) It'd be nice to add this to next week's tech news. Worth mentioning this has been requested 12 years ago (at least) [12:42:24] 10netops, 06Infrastructure-Foundations, 06SRE, 07IPv6, 13Patch-For-Review: WMCS Eqiad: Enable IPv6 in cloud vrf on switches - https://phabricator.wikimedia.org/T389958#10717397 (10cmooney) [12:52:11] 10netops, 06Infrastructure-Foundations, 06SRE, 07IPv6, 13Patch-For-Review: WMCS Eqiad: Enable IPv6 in cloud vrf on switches - https://phabricator.wikimedia.org/T389958#10717450 (10cmooney) 05Open→03Resolved Thankfully all works are now in place for this, after a few little blips on the way. The... [13:24:04] 06Traffic: Upgrade to ATS 9.2.10 - https://phabricator.wikimedia.org/T390912#10717690 (10ssingh) [13:34:05] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: Link down between cr3-ulsfo and cr4-ulsfo - https://phabricator.wikimedia.org/T390731#10717744 (10ayounsi) p:05Low→03High Bumping the priority back up on this one as the interface keeps flapping. {F59004138} {F59004137} @RobH can... [13:36:34] 06Traffic: Upgrade to ATS 9.2.10 - https://phabricator.wikimedia.org/T390912#10717756 (10ssingh) [14:31:17] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: Link down between cr3-ulsfo and cr4-ulsfo - https://phabricator.wikimedia.org/T390731#10717981 (10RobH) Case 01045114 opened just swapped out the info about a bit: > Support, We recently rolled some OS upgrades to our routers and du... [14:45:17] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: Q2:rack/setup E8/F8 new leaf switches - https://phabricator.wikimedia.org/T382017#10718101 (10Jclark-ctr) a:03VRiley-WMF These have been racked and in use is anything else needed for you for ticket? [14:48:39] vgutierrez: I 'll merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1134281, which is returning ATS back to the previous configuration for wikifunctions. Minus the re-used connections per IP config ofc. Any objections? [15:01:45] 10netops, 06Infrastructure-Foundations: Upgrade End Of Support Junos - https://phabricator.wikimedia.org/T390813#10718158 (10ayounsi) p:05Triage→03Medium [15:03:22] 10netops, 06DC-Ops, 06Infrastructure-Foundations: Upgrade management switches to Junos 21.4 - https://phabricator.wikimedia.org/T390814#10718168 (10joanna_borun) p:05Triage→03Low [16:07:39] akosiaris: caught me in the middle of a meeting, looks good but I see you already moved forward [16:09:34] vgutierrez: yup. Triple checked and even tried it in a curl call on the esams host I end up to. Looks good up to now [16:09:38] thanks! [16:21:02] 06Traffic, 13Patch-For-Review: rework ncmonitor's patch submission for ncredir - https://phabricator.wikimedia.org/T390915#10718557 (10BCornwall) 05In progress→03Resolved [18:09:25] FIRING: [2x] SystemdUnitCrashLoop: varnish-frontend.service crashloop on cp7001:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [18:10:40] FIRING: [2x] VarnishPrometheusExporterDown: Varnish Exporter on instance cp7001:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [18:11:00] FIRING: PurgedHighBacklogQueue: Large backlog queue for purged on cp7001:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=magru%20prometheus/ops&var-instance=cp7001 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [18:16:00] RESOLVED: [4x] PurgedHighBacklogQueue: Large backlog queue for purged on cp7001:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [18:17:06] ^varnish crashes are known - those hosts are depooled [18:20:40] FIRING: [2x] VarnishPrometheusExporterDown: Varnish Exporter on instance cp7001:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [18:23:00] FIRING: PurgedHighBacklogQueue: Large backlog queue for purged on cp7002:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=magru%20prometheus/ops&var-instance=cp7002 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [18:28:00] RESOLVED: [2x] PurgedHighBacklogQueue: Large backlog queue for purged on cp7002:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=magru%20prometheus/ops&var-instance=cp7002 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [18:29:25] FIRING: [2x] SystemdUnitCrashLoop: varnish-frontend.service crashloop on cp7001:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [18:30:40] RESOLVED: [2x] VarnishPrometheusExporterDown: Varnish Exporter on instance cp7001:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [18:31:56] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: Q2:rack/setup E8/F8 new leaf switches - https://phabricator.wikimedia.org/T382017#10718936 (10VRiley-WMF) [18:32:00] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: Q2:rack/setup E8/F8 new leaf switches - https://phabricator.wikimedia.org/T382017#10718937 (10VRiley-WMF) 05Open→03Resolved [18:39:25] RESOLVED: [2x] SystemdUnitCrashLoop: varnish-frontend.service crashloop on cp7001:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [23:59:07] Hi, question for you all. Does haproxy receive the real ip as I see you use src? Since for us we are behind cloudflare so have to look for the header cf sends with the real ip.