[07:35:30] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9836599 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin2002 for host mc2049.codfw.wmnet with OS bookworm [07:51:20] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9836654 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin2002 for host mc2049.codfw.wmnet with OS bookworm executed with errors: - mc2049 (**FAIL... [07:52:00] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9836656 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin2002 for host mc2049.codfw.wmnet with OS bookworm [08:03:15] 06serviceops, 10ChangeProp, 06Content-Transform-Team, 10Lift-Wing, and 5 others: Selectively disable changeprop functionality that is no longer used - https://phabricator.wikimedia.org/T361483#9836666 (10dcausse) @achou except expert search users explicitly searching for topics (which I suspect are rare) t... [08:22:13] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9836693 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin2002 for host mc2049.codfw.wmnet with OS bookworm executed with errors: - mc2049 (**FAIL... [09:35:45] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9836955 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin2002 for host mc2049.codfw.wmnet with OS bookworm [10:21:35] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9837065 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin2002 for host mc2049.codfw.wmnet with OS bookworm completed: - mc2049 (**WARN**) - Dow... [12:12:56] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#9837454 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by hnowlan@cumin1002 from kubernetes2032 to wikikube-worker2002 completed:... [12:19:09] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#9837464 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host wikikube-worker2002.codfw.wmnet with OS... [12:19:10] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#9837465 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host wikikube-worker2002.codfw.wmnet with OS bul... [13:04:55] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#9837693 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host wikikube-worker2002.codfw.wmnet with OS bullseye [13:35:16] 06serviceops, 10Data Products (Data Products Sprint 13): Service Ops Review of Metrics Platform Configuration Management UI - https://phabricator.wikimedia.org/T358577#9837914 (10WDoranWMF) 05Open→03Resolved [14:05:41] 06serviceops, 10API Platform, 10MediaWiki-extensions-ReadingList, 06MW-Interfaces-Team, and 2 others: Reading List REST Interface: reroute calls - https://phabricator.wikimedia.org/T348493#9837996 (10BPirkle) Tagging serviceops (hopefully I picked the right tag). This work is getting close to complete now... [15:32:04] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#9838309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host wikikube-worker2002.codfw.wmnet with OS bullseye completed: - wik... [16:28:37] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9838656 (10dancy) >>! In T362518#9834634, @Clement_Goubert wrote: > Just to be completely sure before deleting > >>>! In T362518#9805179, @dancy wrote: >>>>! In T362... [16:58:15] 06serviceops, 06SRE: k8s master capacity issues - https://phabricator.wikimedia.org/T366094 (10hnowlan) 03NEW [18:18:36] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install kafka-main100[6789] and kafka-main1010 - https://phabricator.wikimedia.org/T363212#9839289 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host kafka-main1010.eqiad.wmnet with OS bullseye [18:34:59] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install kafka-main100[6789] and kafka-main1010 - https://phabricator.wikimedia.org/T363212#9839375 (10Jclark-ctr) @akosiaris still failing for same issue for kafka-main1010 [18:49:12] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install kafka-main100[6789] and kafka-main1010 - https://phabricator.wikimedia.org/T363212#9839469 (10Dzahn) @akosiaris re: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1035769/1/modules/profile/data/profile/installserver/preseed.yaml I th... [18:50:58] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install kafka-main100[6789] and kafka-main1010 - https://phabricator.wikimedia.org/T363212#9839473 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host kafka-main1009.eqiad.wmnet with OS bullseye [19:19:40] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q4:rack/setup/install kafka-main100[6789] and kafka-main1010 - https://phabricator.wikimedia.org/T363212#9839652 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host kafka-main1010.eqiad.wmnet... [19:37:02] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install kafka-main100[6789] and kafka-main1010 - https://phabricator.wikimedia.org/T363212#9839744 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host kafka-main1009.eqiad.wmnet with OS bullseye executed... [20:49:37] 06serviceops, 06SRE, 13Patch-For-Review: k8s master capacity issues - https://phabricator.wikimedia.org/T366094#9839951 (10CDanis) Posting a short comment now, before I start drafting a much longer comment (and possibly don't finish before my toddler ends my day): I believe the situation to be stable for no... [21:58:26] 06serviceops, 10MW-on-K8s, 06Traffic: XWD: Allow choosing datacentre in k8s-mwdebug - https://phabricator.wikimedia.org/T365478#9840260 (10jijiki) 05In progress→03Resolved