[07:35:00] 10Traffic, 10SRE, 10observability: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000 (10Vgutierrez) It looks like it's a matter of how we graph the data, please see: https://grafana.wikimedia.org/goto/7xCydjqVk?orgId=1 {F37159719} first panel is the original one using `rat... [08:27:46] 10Traffic, 10SRE, 10observability: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000 (10fgiunchedi) Looking at the raw data https://w.wiki/7AxF there's indeed a counter "reset" e.g. around 16:00 {F37159747} I'm not sure offhand why moving to a smaller period fixes things,... [09:24:46] 10netops, 10Infrastructure-Foundations, 10SRE: Announce new public IPv6 prefix from Amsterdam for knams migration - https://phabricator.wikimedia.org/T343216 (10cmooney) [09:27:18] 10netops, 10Infrastructure-Foundations, 10SRE: Announce new public IPv6 prefix from Amsterdam for knams migration - https://phabricator.wikimedia.org/T343216 (10cmooney) Being announced from all esams/knams routers now, for example: ` cmooney@re0.cr2-esams> show route advertising-protocol bgp 2001:7f8:1:0:a5... [10:48:33] <_joe_> vgutierrez: I'm re-reviewing https://gerrit.wikimedia.org/r/c/operations/puppet/+/941448 then let's merge it? [10:48:56] ok [10:50:51] <_joe_> vgutierrez: I'm disabling puppet everywhere, applying to one host in each cluster in ulsfo [10:50:56] <_joe_> and just check nothing explodes [10:50:59] <_joe_> then reenable [10:51:01] <_joe_> ok? [10:51:02] ack [10:57:06] <_joe_> seems to have reloaded vcl ok [10:57:11] <_joe_> reenabling everywhere [11:01:27] data started to show up: https://w.wiki/7Azv [11:01:45] I've filtered by static_ [11:02:48] this is split by rule: https://w.wiki/7Azw [11:03:40] <_joe_> yeah ua_policy is ofc big [11:04:02] and it should be bigger IMHO [11:04:04] :) [11:04:09] :D [11:15:06] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [12:56:26] 10Traffic, 10Content-Transform-Team-WIP, 10Mobile-Content-Service, 10RESTbase Sunsetting, and 2 others: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10vadim-kovalenko) Hi there! I'm responsible for Kiwix migration to another API, but given the discussion above I'm curious w... [14:25:54] 10Traffic, 10SRE: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10Fabfur) [15:00:08] 10Traffic, 10SRE: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10Fabfur) [15:12:59] 10Traffic, 10SRE: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10Fabfur) Start working on varnishkafka package [15:44:42] (SystemdUnitFailed) firing: anycast-healthchecker.service Failed on dns3002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:49:42] (SystemdUnitFailed) resolved: anycast-healthchecker.service Failed on dns3002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:19:33] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Krinkle) [22:47:55] 10Traffic, 10Observability-Metrics, 10Patch-For-Review: Add prometheus-https load balancer - https://phabricator.wikimedia.org/T326657 (10BCornwall) 05Resolved→03Open