[02:56:12] 06serviceops, 06Community-Tech, 10LPL Essential, 10MinT, 10Community Wishlist (Translations): Caching service request for MinT - https://phabricator.wikimedia.org/T370755#10374431 (10JWheeler-WMF) [05:20:17] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10374516 (10tstarling) The original feature was for Apache servers to make requests to localhost, avoiding the overhead of making a request over the network via the CDN. Maintenan... [05:48:18] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10374541 (10tstarling) >>! In T341553#10370811, @Tgr wrote: > The actual problem is MWHttpRequest [[https://gerrit.wikimedia.org/g/mediawiki/core/+/f9659bc86e80851eb7b575733dca13a... [06:14:04] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10374558 (10Joe) >>! In T341553#10374541, @tstarling wrote: > > You can see it there in the doc comment of the method linked there. "Check if the URL can be served by localhost".... [09:53:02] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10374957 (10ops-monitoring-bot) pool host wikikube-worker1006.eqiad.wmnet by jelto@cumin1002 with reason: None [09:53:05] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10374958 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker1006.eqiad.wmnet comp... [09:53:50] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-eqiad, and 3 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T381268#10374959 (10Jelto) [10:41:49] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10375071 (10ops-monitoring-bot) depool host kubernetes[1019-1020].eqiad.wmnet by jelto@cumin1002 with reason: Renaming nodes [10:43:08] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10375072 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host kubernetes[1019-1020].eqiad.wmnet... [11:45:24] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10375234 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes1019 to wikikube-work... [11:51:12] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10375251 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes1020 to wikikube-work... [11:58:44] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10375271 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker1015.eq... [12:04:53] 06serviceops, 10Wikidata, 06Wikidata Dev Team, 10Wikidata-Termbox, 10wmde-wikidata-tech: [SW] [GENERAL] Simplify Termbox SSR test release - https://phabricator.wikimedia.org/T355955#10375278 (10karapayneWMDE) [12:06:39] 06serviceops, 10Wikidata, 06Wikidata Dev Team, 10Wikidata-Termbox, 10wmde-wikidata-tech: [SW] [GENERAL] Simplify Termbox SSR test release - https://phabricator.wikimedia.org/T355955#10375294 (10karapayneWMDE) [12:09:34] 06serviceops, 10Wikidata, 06Wikidata Dev Team, 10Wikidata-Termbox, 10wmde-wikidata-tech (DOT by Project): [SW] [GENERAL] Simplify Termbox SSR test release - https://phabricator.wikimedia.org/T355955#10375302 (10karapayneWMDE) [12:32:46] 06serviceops: kafka-main100[6789] and kafka-main1010 implementation tracking - https://phabricator.wikimedia.org/T363214#10375387 (10jijiki) [12:37:05] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10375398 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker1015.eqiad.wmnet with OS bookworm... [12:37:51] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10375399 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker1016.eqiad.wmnet with OS book... [13:15:37] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10375613 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker1016.eqiad.wmnet with OS bookworm... [13:28:24] BTW, for peeps doing reimages, I have the first version of a roll-reimage cookbook working: https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1094494/4 . If you use specifically PS4, I won't break it :D (cc jelto?) [13:29:13] (Currently t has hard-coded-ish arguments and the CLI syntax is a bit awkward, I'll be fixing that, but still, it seems a Lot more convenient than running multiple reimages separately) [14:13:39] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10375818 (10ops-monitoring-bot) pool host wikikube-worker[1015-1016].eqiad.wmnet by jelto@cumin1002 with reason: None [14:13:40] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10375819 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker[1015-1016].eqiad.wmn... [14:14:15] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-eqiad, and 3 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T381268#10375822 (10Jelto) [15:13:40] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10376008 (10ops-monitoring-bot) depool host kubernetes[1021-1022].eqiad.wmnet by jelto@cumin1002 with reason: Renaming nodes [15:17:36] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10376011 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host kubernetes[1... [15:38:26] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10376106 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes1021 to wikikube-worker1034 completed: - ku... [15:48:45] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10376184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes1022 to wikikube-worker1035 completed: - ku... [15:49:28] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: Decommission mc-gp200[1-3].codfw.wmnet - https://phabricator.wikimedia.org/T381174#10376196 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [15:51:53] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10376253 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker1034.eqiad.wmnet with OS book... [16:26:22] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 10MW-1.44-notes (1.44.0-wmf.4; 2024-11-19), and 3 others: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10376451 (10akosiaris) >>! In T374683#1037167... [16:27:45] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 10MW-1.44-notes (1.44.0-wmf.4; 2024-11-19), and 3 others: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10376465 (10HCoplin-WMF) Fabulous! Thank you... [16:50:01] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10376558 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker1034.eqiad.wmnet with OS bookworm... [16:51:20] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10376578 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker1034.eqiad.wmnet with OS book... [17:05:20] jelto: please leave some hosts for me :-D (testing my roll-reimage cookbook, which you are also welcome to use) [17:07:49] OTOH, if you finish all the ones that need a rename, I won't have to add that functionality to the cookbook! :D [17:30:18] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10376796 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker1034.eqiad.wmnet with OS bookworm... [17:32:52] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10376799 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker1035.eqiad.wmnet with OS book... [18:01:19] I'll take a look at the cookbook tomorrow. That's great new. And don't worry, there are plenty of hosts left (codfw 21.2% done and eqiad 14.6% done) [18:02:58] jelto: ok, thanks :-) I'm about to commit a newer version (that may or may not work :D) in case you want to also look at the code [18:11:17] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10376971 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker1035.eqiad.wmnet with OS bookworm... [18:12:38] 06serviceops, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10376930 (10Scott_French) [18:24:23] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-eqiad, and 3 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T381268#10377086 (10Jelto) [18:35:48] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10377150 (10ops-monitoring-bot) pool host wikikube-worker[1034-1035].eqiad.wmnet by jelto@cumin1002 with reason: None [18:35:49] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10377151 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker[1034-1035].eqiad.wmn... [19:05:29] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10377310 (10ops-monitoring-bot) Started 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10377321 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1278.eqiad.wmnet with OS boo... [19:30:41] \o hi service-ops, I was allocating new VIPs for some new wdqs lvs services last night and noticed that `mw-parsoid` is likely missing entries in the netbox UI for its VIP [19:31:03] TLDR `mw-parsoid` has PTR and A records for `10.2.2.92/32` and `10.2.1.92/32` but is missing a corresponding entry in the netbox UI, see https://wikitech.wikimedia.org/wiki/DNS/Netbox#How_to_manually_allocate_a_special_purpose_IP_address_in_Netbox for how that's done [19:53:45] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10377474 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1278.eqiad.wmnet with OS bookwor... [19:55:42] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10377477 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1279.eqiad.wmnet with OS boo... [20:09:00] 06serviceops, 10ContentTranslation, 10Recommendation-API: Caching service request for recommendation api - https://phabricator.wikimedia.org/T381438 (10SBisson) 03NEW [20:13:52] 06serviceops, 10ContentTranslation, 10LPL Hypothesis, 10Recommendation-API: Caching service request for recommendation api - https://phabricator.wikimedia.org/T381438#10377521 (10SBisson) [20:38:42] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10377614 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1279.eqiad.wmnet with OS bookwor... [20:47:33] 06serviceops, 10Cassandra: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#10377649 (10Eevans) >>! In T368096#10046564, @Eevans wrote: > @Cparle how hard would it be to create some functional tests for the extension? Something we could run (even manually/ad... [21:07:11] ryankemper: ah, good catch! I'm a bit surprised you're the first to run into that ... I'll take a look at backfilling those this afternoon. [21:09:56] swfrench-wmf: cool! fwiw I suspect the reason I'm the first is because I'm launching 2 services not 1 and wanted the IPs to be sequential. so I skipped over 4 spots where only one IP was available [21:10:27] and landed on `.92` which must be highly desirable real estate ;) [21:13:15] ah, I see - so .92 was the first "hole" of size at least 2 :)