[01:25:35] <wikibugs>	 10Traffic, 10DNS, 10Patch-For-Review: add wikimedia.social to WMF DNS (was: Update DNS records for mastodon.wikimedia.org) - https://phabricator.wikimedia.org/T337586 (10ssingh) 05Open→03Resolved ` dig wikimedia.social NS +short ns1.wikimedia.org. ns0.wikimedia.org. ns2.wikimedia.org. `  ` dig wikimedia....
[05:10:34] <wikibugs>	 10Traffic, 10DNS, 10Patch-For-Review: add wikimedia.social to WMF DNS (was: Update DNS records for mastodon.wikimedia.org) - https://phabricator.wikimedia.org/T337586 (10Dzahn) Thanks a lot @ssingh :)
[05:17:44] <jinxer-wm>	 (VarnishHighThreadCount) firing: (3) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[05:22:44] <jinxer-wm>	 (VarnishHighThreadCount) firing: (8) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[05:27:44] <jinxer-wm>	 (VarnishHighThreadCount) firing: (8) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[05:32:44] <jinxer-wm>	 (VarnishHighThreadCount) resolved: (8) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[07:29:07] <wikibugs>	 10Traffic, 10Anti-Harassment, 10Data-Engineering, 10SRE, and 2 others: Include User-Agent Client Hints in WebRequest logs - https://phabricator.wikimedia.org/T337947 (10kostajh)
[08:08:07] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review: port 80 paging on scheduled single host maintenance in text@esams - https://phabricator.wikimedia.org/T339898 (10Vgutierrez) `counterexample vgutierrez@prometheus3002:~$ curl -w @- -o /dev/null --resolve www.wikipedia.org:80:91.198.174.192 -s http://www.wikipedia.org <<'...
[09:32:13] <jinxer-wm>	 (VarnishHighThreadCount) firing: (8) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[09:36:51] <jinxer-wm>	 (VarnishHighThreadCount) firing: (8) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[09:41:45] <jinxer-wm>	 (VarnishHighThreadCount) firing: (8) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[09:46:45] <jinxer-wm>	 (VarnishHighThreadCount) resolved: (8) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[12:04:07] <jinxer-wm>	 (VarnishHighThreadCount) firing: (3) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[12:09:16] <jinxer-wm>	 (VarnishHighThreadCount) firing: (7) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[12:12:44] <jinxer-wm>	 (VarnishHighThreadCount) firing: (7) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[12:17:45] <jinxer-wm>	 (VarnishHighThreadCount) resolved: (7) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[12:39:51] <elukey>	 hello folks
[12:40:01] <elukey>	 if you are ok I'd roll out pki to vk instances in codfw
[12:40:14] <elukey>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/931498
[12:41:30] <vgutierrez>	 elukey: please go ahead
[12:42:00] <elukey>	 vgutierrez: mmm role::cache::text for codfw in hiera disappeared :D
[12:42:29] <vgutierrez>	 elukey: yeah, it was consolidated
[12:42:37] <vgutierrez>	 cause right now it's applied globally
[12:42:41] <vgutierrez>	 so no need to tell between DCs
[12:43:10] <vgutierrez>	 you can blame fabfur 
[12:43:20] <vgutierrez>	 that's https://gerrit.wikimedia.org/r/c/operations/puppet/+/929989
[12:43:33] <elukey>	 ahhhh
[12:43:53] <elukey>	 can I recreate it, just to support the migration?
[12:43:57] <vgutierrez>	 of course
[12:46:30] <elukey>	 thanks! patch updated, running pcc again
[12:49:41] <elukey>	 vgutierrez: when you have a moment, could you double check https://gerrit.wikimedia.org/r/c/operations/puppet/+/931498 ?
[12:49:50] <elukey>	 it is trivial but better safe than sorry
[12:50:48] <elukey>	 <3
[12:57:23] <elukey>	 interesting, I didn't expect so much difference between eqiad and codfw for vk traffic
[12:57:51] <elukey>	 I assumed that frontend traffic hitting codfw was more
[13:07:18] <jinxer-wm>	 (VarnishHighThreadCount) firing: (8) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[13:12:14] <jinxer-wm>	 (VarnishHighThreadCount) firing: (11) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[13:15:07] <wikibugs>	 10Traffic, 10Mobile-Content-Service, 10Product-Infrastructure-Team-Backlog-Deprecated, 10SRE, 10serviceops: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10MSantos)
[13:15:45] <wikibugs>	 10Traffic, 10Mobile-Content-Service, 10Product-Infrastructure-Team-Backlog-Deprecated, 10SRE, 10serviceops: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10MSantos) p:05Triage→03High
[13:16:44] <jinxer-wm>	 (VarnishHighThreadCount) firing: (12) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[13:16:55] <wikibugs>	 10Traffic, 10Mobile-Content-Service, 10Product-Infrastructure-Team-Backlog-Deprecated, 10SRE, 10serviceops: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10MSantos)
[13:17:55] <wikibugs>	 10Traffic, 10Mobile-Content-Service, 10Product-Infrastructure-Team-Backlog-Deprecated, 10SRE, 10serviceops: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10MSantos)
[13:19:45] <wikibugs>	 10Traffic, 10Mobile-Content-Service, 10Product-Infrastructure-Team-Backlog-Deprecated, 10RESTbase Sunsetting, and 2 others: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10MSantos)
[13:21:44] <jinxer-wm>	 (VarnishHighThreadCount) firing: (13) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[13:25:25] <wikibugs>	 10Traffic, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: decide on an aggregation function to combine multiple probes into a single measurement - https://phabricator.wikimedia.org/T337318 (10JameelKaisar) You can see Box Plots and Violin Plots of the per-country latency data here:   ` stat1009.e...
[13:26:44] <jinxer-wm>	 (VarnishHighThreadCount) firing: (13) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[13:29:27] <elukey>	 all codfw vk instances migrated!
[13:31:44] <jinxer-wm>	 (VarnishHighThreadCount) firing: (10) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[13:36:44] <jinxer-wm>	 (VarnishHighThreadCount) firing: (8) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[13:38:33] <wikibugs>	 10Traffic, 10Mobile-Content-Service, 10Product-Infrastructure-Team-Backlog-Deprecated, 10RESTbase Sunsetting, and 2 others: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10akosiaris) So, we need something to identify those users. wikiwand, if I understand the usage of the MC...
[13:41:44] <jinxer-wm>	 (VarnishHighThreadCount) resolved: (7) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[13:45:44] <jinxer-wm>	 (VarnishHighThreadCount) firing: (8) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[13:52:58] <wikibugs>	 10Traffic, 10Mobile-Content-Service, 10Product-Infrastructure-Team-Backlog-Deprecated, 10RESTbase Sunsetting, and 2 others: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10MSantos) >>! In T340036#8952804, @akosiaris wrote: > So, we need something to identify those users. wik...
[13:55:29] <wikibugs>	 10Traffic, 10Mobile-Content-Service, 10Product-Infrastructure-Team-Backlog-Deprecated, 10RESTbase Sunsetting, and 2 others: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10akosiaris) >>! In T340036#8952836, @MSantos wrote: > Sounds great to me, no objections.  Cool. Do we ha...
[13:55:44] <jinxer-wm>	 (VarnishHighThreadCount) firing: (8) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[14:00:45] <jinxer-wm>	 (VarnishHighThreadCount) firing: (8) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[14:02:07] <jbond>	 any work going on in codfw we just got a nel report for tcp resets from text-lb.codfw.wikimedia.org (allrady seems to be recovering)
[14:05:45] <jinxer-wm>	 (VarnishHighThreadCount) resolved: (8) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[14:06:58] <vgutierrez>	 jbond: hmmm not besides elukey migrating varnishkafka
[14:07:34] <elukey>	 yep exactly but completed half an hour ago (and touching only varnishkafka)
[14:07:35] <jbond>	 vgutierrez: ack it was only a small peak so could equally be some isp rebooting a cgnat or somethinfg
[14:07:37] <vgutierrez>	 but I'm guessing that PROBLEM - BGP status on cr1-codfw is CRITICAL: BGP CRITICAL - AS1299/IPv4: Connect - Telia https://wikitech.wikimedia.org/wiki/Network_monitoring%23BGP_status could have triggered that
[14:07:53] <elukey>	 yep --^
[14:07:57] <jbond>	 ahh yes thanks
[14:49:15] <wikibugs>	 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10aborrero)
[15:01:32] <hnowlan>	 hey vgutierrez - could you give https://gerrit.wikimedia.org/r/c/operations/puppet/+/929674 another look when you have time please? 
[15:01:46] <vgutierrez>	 yeah.. I was checking it
[15:03:05] <vgutierrez>	 https://www.irccloud.com/pastebin/amDydVKI/
[15:03:16] <vgutierrez>	 endpoint looks healthy now in terms of DNS and TLS
[15:05:15] <brett>	 vgutierrez: You running pcc or shall I?
[15:05:25] <vgutierrez>	 against..? :)
[15:05:32] <brett>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/931947/5
[15:05:59] <vgutierrez>	 brett: I did it for PS4 and PS5 is just a rebase on top of the previous change
[15:06:56] <brett>	 but 4 → 5 seems to be an underscore fix?
[15:07:20] <brett>	 It seems I'm missing something, I'll get out of your way
[15:08:45] <vgutierrez>	 brett: nope, you're right
[15:08:57] <vgutierrez>	 rerunning right now :)
[15:15:27] <vgutierrez>	 hnowlan: happy to help you moving forward with that CR
[15:15:50] <hnowlan>	 vgutierrez: thank you! I am PTO for the next 2 days so I might hold off on merging it until Monday if that's okay 
[15:16:08] <vgutierrez>	 hnowlan: perfect
[19:34:05] <wikibugs>	 10Traffic, 10Infrastructure-Foundations, 10SRE: decide on an aggregation function to combine multiple probes into a single measurement - https://phabricator.wikimedia.org/T337318 (10JameelKaisar) Update:   - Instead of trimming bottom 10 %, we are trimming bottom 5 % only.   - We are plotting Box plots as we...