[01:45:56] 06Traffic, 06SRE, 10SRE-swift-storage, 10Thumbor: Cache thumbs in our caching infrastructure (e.g. ATS) - https://phabricator.wikimedia.org/T345334#9762680 (10tstarling) >>! In T345334#9752173, @Ladsgroup wrote: > That'd work on overall hits, as you said "sort images by popularity". That's not the case her... [02:27:58] 10Wikimedia-Apache-configuration, 10Wikidata, 10Wikimedia-Site-requests, 10[DEPRECATED] wdwb-tech (Freezer πŸ₯Ά), and 2 others: wikidata.org/entity/Q12345 should do content negotiation immediately, instead of redirecting to wikidata.org/wiki/Special:EntityData... - https://phabricator.wikimedia.org/T119536#9762706 [11:15:52] 06Traffic, 10MoveComms-Support, 10MW-on-K8s, 06serviceops, and 2 others: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323#9763469 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1... [11:17:27] 06Traffic, 10MoveComms-Support, 10MW-on-K8s, 06serviceops, and 2 others: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323#9763472 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1... [11:21:29] 06Traffic, 10MoveComms-Support, 10MW-on-K8s, 06serviceops, and 2 others: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323#9763477 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1... [11:21:30] 06Traffic, 10MoveComms-Support, 10MW-on-K8s, 06serviceops, and 2 others: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323#9763478 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1... [11:21:33] 06Traffic, 10MoveComms-Support, 10MW-on-K8s, 06serviceops, and 2 others: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323#9763479 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1... [11:51:59] 06Traffic, 10MoveComms-Support, 10MW-on-K8s, 06serviceops, and 2 others: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323#9763643 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1371.... [11:53:59] 06Traffic, 10MoveComms-Support, 10MW-on-K8s, 06serviceops, and 2 others: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323#9763647 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1409.... [11:56:00] 06Traffic, 10MoveComms-Support, 10MW-on-K8s, 06serviceops, and 2 others: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323#9763650 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1405.... [11:57:27] 06Traffic, 10MoveComms-Support, 10MW-on-K8s, 06serviceops, and 2 others: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323#9763652 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1435.... [12:01:12] 06Traffic, 10MoveComms-Support, 10MW-on-K8s, 06serviceops, and 2 others: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323#9763678 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1399.... [13:59:22] good morning! I have crafted some bespoke shell one-liner monitoring for watching for the first probes towards magru: [13:59:34] πŸ’™cdanis@stat1009.eqiad.wmnet ~ πŸ•™β˜• kafkacat -C -b kafka-jumbo1008.eqiad.wmnet:9092 -G cdanis-kafkacat-networkprobe {eqiad,codfw}.development.network.probe | jq -c 'select(.meta.domain != "canary") | select(.reports[].target_name == "magru")' | head | jq . [14:00:29] cdanis: it's been a while since I've seen a new maxi-oneliner from you :) [14:00:30] cdanis: ha [14:00:40] you could have pasted this without your name and we could have guessed it was you [14:01:40] :) [14:02:26] XioNoX: with older versions of kafkacat you'd have to use https://phabricator.wikimedia.org/P13393 to watch both the eqiad and codfw topics [14:04:19] cdanis: invalid, more than 1 line [14:10:20] πŸ˜‚ [15:24:25] 06Traffic, 06Infrastructure-Foundations: Q4:magru VM tracking task - https://phabricator.wikimedia.org/T364016 (10ssingh) 03NEW [15:31:33] 06Traffic, 06Infrastructure-Foundations: Q4:magru VM tracking task - https://phabricator.wikimedia.org/T364016#9764292 (10MoritzMuehlenhoff) p:05Triageβ†’03High [15:33:24] 06Traffic, 06Infrastructure-Foundations: Q4:magru VM tracking task - https://phabricator.wikimedia.org/T364016#9764294 (10MoritzMuehlenhoff) [15:37:23] 06Traffic, 06Infrastructure-Foundations: Q4:magru VM tracking task - https://phabricator.wikimedia.org/T364016#9764307 (10MoritzMuehlenhoff) [15:44:42] 06Traffic, 06Infrastructure-Foundations: Q4:magru VM tracking task - https://phabricator.wikimedia.org/T364016#9764324 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1002 for host durum7001.magru.wmnet with OS bookworm [15:53:33] 06Traffic, 06Infrastructure-Foundations: Q4:magru VM tracking task - https://phabricator.wikimedia.org/T364016#9764374 (10ayounsi) [16:40:44] 06Traffic, 06Infrastructure-Foundations: Q4:magru VM tracking task - https://phabricator.wikimedia.org/T364016#9764551 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1002 for host durum7001.magru.wmnet with OS bookworm completed: - durum7001 (**WARN**) - Removed from Puppet... [18:16:41] 06Traffic, 06Data-Platform-SRE, 06Infrastructure-Foundations: Investigate why pools.json does not match https://config-master.wikimedia.org/pybal/${datacenter}/${service} T363702 - https://phabricator.wikimedia.org/T364037 (10bking) 03NEW [18:19:08] 06Traffic, 06Data-Platform-SRE: LVS hosts: Monitor/alert on when pooled nodes are outside broadcast domain - https://phabricator.wikimedia.org/T363702#9764939 (10bking) FWiW, I've started work on [[ https://gitlab.wikimedia.org/repos/search-platform/sre/lvs_l2_checker/-/tree/main?ref_type=heads | a simple scri... [18:20:12] 06Traffic, 06Infrastructure-Foundations, 10vm-requests: magru: (2) VMs for ncredir - https://phabricator.wikimedia.org/T363881#9764941 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host ncredir7001.magru.wmnet with OS bookworm [18:22:23] 06Traffic, 06Infrastructure-Foundations, 10vm-requests: magru: (2) VMs for ncredir - https://phabricator.wikimedia.org/T363881#9764943 (10BCornwall) 05Stalledβ†’03In progress [18:41:21] 06Traffic, 06Infrastructure-Foundations: Q4:magru VM tracking task - https://phabricator.wikimedia.org/T364016#9764976 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1002 for host doh7001.wikimedia.org with OS bookworm [18:46:01] cdanis: since you asked, we have pooled the cp's :) [18:55:00] curl -I4 https://measure-magru.wikimedia.org [18:55:05] x-cache: cp7014 int [19:06:26] sukhe: amazing! [19:16:09] I'll attempt to get the patch against WikimediaEvents deployed today [19:17:19] thanks! [19:30:36] sukhe: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1026667 [19:36:30] 06Traffic, 06Infrastructure-Foundations: Q4:magru VM tracking task - https://phabricator.wikimedia.org/T364016#9765147 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1002 for host doh7001.wikimedia.org with OS bookworm completed: - doh7001 (**WARN**) - Removed from Puppet a... [19:37:18] so then we could merge this https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/1021521 [19:39:21] fabfur: it's actually been merged, I'm about to run `scap backport` to put it live [19:39:25] unless you want to do the honors [19:39:32] but isn't it late for you there? [19:39:35] absolutely! (not) [19:39:48] it's late, so if you want to scap, scap! [19:39:53] ok will do [19:39:59] thanks a lot! [19:40:16] but if wm goes down for this, I want the honor [19:43:10] we will have to share it equally [19:43:13] that's still something I'm yet to do here [19:49:09] 06Traffic, 06Infrastructure-Foundations, 10vm-requests: magru: (2) VMs for ncredir - https://phabricator.wikimedia.org/T363881#9765177 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host ncredir7001.magru.wmnet with OS bookworm executed with errors: - ncredir7001 (... [20:01:46] sukhe: fabfur: we have received our first probes for magru [20:03:58] πŸ‘ [20:04:11] there will be a lot of data for us to look at in my morning tomorrow [20:06:19] part of these will be my curl's [20:06:22] :) [20:07:23] fabfur: ahh but I was looking at the analytics event kafka queue [20:07:44] client-side js in mediawiki posts there [20:09:04] 10netops, 06Infrastructure-Foundations, 06SRE, 10MW-1.43-notes (1.43.0-wmf.3; 2024-04-30), 13Patch-For-Review: Add probenet configuration for magru - https://phabricator.wikimedia.org/T362902#9765219 (10CDanis) 05Openβ†’03Resolved [20:09:42] 06Traffic: Craft geo-maps file to create lowest-latency routes from south america - https://phabricator.wikimedia.org/T363722#9765223 (10CDanis) [20:09:44] 06Traffic, 06Infrastructure-Foundations, 06SRE: Slowly ramping up traffic to the Brazil data center (magru) and related geo-maps - https://phabricator.wikimedia.org/T359054#9765224 (10CDanis) [20:09:46] 10netops, 06Infrastructure-Foundations, 06SRE, 10MW-1.43-notes (1.43.0-wmf.3; 2024-04-30), 13Patch-For-Review: Add probenet configuration for magru - https://phabricator.wikimedia.org/T362902#9765222 (10CDanis) [20:14:02] cdanis: woo! [20:14:30] we look forward to the data :) [20:43:21] 06Traffic, 06Infrastructure-Foundations: Q4:magru VM tracking task - https://phabricator.wikimedia.org/T364016#9765289 (10ssingh)