[02:33:48] <wikibugs>	 10Traffic, 10SRE, 10Epic, 10User-notice: Deploy Wikimedia DNS: DNS-over-HTTPS (DoH) and DNS-over-TLS (DoT) public resolver - https://phabricator.wikimedia.org/T252132 (10Shizhao)
[06:18:13] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: scrape ripe atlas data for a few anchors at other large networks - https://phabricator.wikimedia.org/T252890 (10ayounsi) @CDanis  Is that still needed now that we have NEL?
[07:47:45] <jayme>	 hi traffic, I'm going to restart pybals in codfw, eqsin and ulsfo in a bit to change the conf servers they use (https://gerrit.wikimedia.org/r/c/operations/puppet/+/957248/) 
[07:51:18] <wikibugs>	 10Traffic: Move purged repository from Gerrit to GitLab - https://phabricator.wikimedia.org/T346305 (10Fabfur)
[08:25:48] <vgutierrez>	 jayme: ack
[08:41:22] <elukey>	 jayme: good luck :)
[08:42:32] <godog>	 thank you jayme! incidentally that "fixed" the alert I was looking into yesterday where pybal or the alert were confused
[08:58:34] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review: Varnish mobile redirection misses some sites - https://phabricator.wikimedia.org/T344175 (10Fabfur) Regarding the other domains (the ones not part of *.wikimedia.org), only `test.m.wikidata.org` and `m.wikifunctions.org` DNS records are configured.  What should we do wit...
[10:23:12] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10Observability-Metrics, 10SRE Observability (FY2023/2024-Q1): Alert "access port speed less 100mbit" and librenms upgrade - https://phabricator.wikimedia.org/T346317 (10fgiunchedi)
[11:08:05] <wikibugs>	 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert)
[11:09:17] <wikibugs>	 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Direct 5% of all traffic to mw-on-k8s - https://phabricator.wikimedia.org/T341780 (10Clement_Goubert) 05Open→03Resolved We are now serving 5% of global traffic from mw-on-k8s. Resolving.
[14:34:13] <wikibugs>	 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Jdforrester-WMF)
[15:16:59] <jayme>	 vgutierrez: I'd like to undo the lvs change again shortly (restarting them again) - objections?
[15:17:12] <vgutierrez>	 ack, just remember to !log your actions
[15:17:23] <jayme>	 wilco
[15:47:34] <bblack>	 jayme: what lvs change was it?
[15:47:47] <jayme>	 bblack: https://gerrit.wikimedia.org/r/c/operations/puppet/+/957395
[15:48:19] <bblack>	 ah ok, I get it now :)
[15:49:43] <jayme>	 all done from my side and "everything" looks happy again
[15:49:57] <wikibugs>	 10Traffic, 10SRE, 10GitLab (Project Migration): Move purged repository from Gerrit to GitLab - https://phabricator.wikimedia.org/T346305 (10Aklapper) + #gitlab-migration
[15:51:15] <wikibugs>	 10Traffic, 10SRE, 10GitLab (Project Migration): Move purged repository from Gerrit to GitLab - https://phabricator.wikimedia.org/T346305 (10Aklapper)
[16:16:10] <denisse>	 Hello team, I was trying to merge DNS changes in dns1004 and I encountered this issue: fatal: Initial load of zone data failed
[16:18:45] <denisse>	 This is the error I'm getting: https://phabricator.wikimedia.org/P52510
[16:20:02] <topranks>	 denisse: sorry that is due to some work I was doing 
[16:21:22] <topranks>	 fixing up now give me 10 mins - I will re-run the netbox dns cookbook & do authdns update after 
[16:23:37] <denisse>	 Sure, no worries. :)
[18:46:17] <topranks>	 denisse: I did that update back when I said I would but realised I forgot to let you know sry
[18:47:42] <denisse>	 Hi topranks , no worries. I figured it was done and I merged my changes successfully, thank you!! ^^
[18:48:24] <topranks>	 glad to hear! 
[19:08:30] <wikibugs>	 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Do we need ping offload servers at all POPs? - https://phabricator.wikimedia.org/T345809 (10cmooney) Do we have any way to measure it's impact?  I had a quick look at available promethues metrics and didn't see much corresponding to icmp (but may ha...
[19:29:31] <wikibugs>	 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Do we need ping offload servers at all POPs? - https://phabricator.wikimedia.org/T345809 (10BBlack) > some sort of rate-limiting configured on the switch-side for ICMP echo, which was IP-aware and didn't count packets from our own internal systems...
[19:31:55] <wikibugs>	 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Do we need ping offload servers at all POPs? - https://phabricator.wikimedia.org/T345809 (10BBlack) https://grafana.wikimedia.org/d/000000513/ping-offload might be a good starting point (might need some updates/tweaking to get the exact data you wan...
[19:32:39] <wikibugs>	 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Do we need ping offload servers at all POPs? - https://phabricator.wikimedia.org/T345809 (10cmooney) >>! In T345809#9168116, @BBlack wrote: >> some sort of rate-limiting configured on the switch-side for ICMP echo, which was IP-aware and didn't coun...