[06:20:14] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849404 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin2002 for host mc2043.codfw.wmnet with OS bookworm [06:20:17] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849405 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1002 for host mc1043.eqiad.wmnet with OS bookworm [06:53:12] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849411 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1002 for host mc1043.eqiad.wmnet with OS bookworm completed: - mc1043 (**PASS**) - Dow... [06:58:34] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849422 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin2002 for host mc2043.codfw.wmnet with OS bookworm completed: - mc2043 (**PASS**) - Dow... [07:40:46] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849459 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1002 for host mc1042.eqiad.wmnet with OS bookworm [07:40:47] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849460 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin2002 for host mc2042.codfw.wmnet with OS bookworm [08:13:10] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849499 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1002 for host mc1042.eqiad.wmnet with OS bookworm completed: - mc1042 (**PASS**) - Dow... [08:20:12] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849518 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin2002 for host mc2042.codfw.wmnet with OS bookworm completed: - mc2042 (**PASS**) - Dow... [08:47:22] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849598 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1002 for host mc1041.eqiad.wmnet with OS bookworm [08:47:26] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849599 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin2002 for host mc2041.codfw.wmnet with OS bookworm [09:20:05] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849673 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1002 for host mc1041.eqiad.wmnet with OS bookworm completed: - mc1041 (**PASS**) - Dow... [09:25:55] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849677 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin2002 for host mc2041.codfw.wmnet with OS bookworm completed: - mc2041 (**PASS**) - Dow... [09:42:09] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849692 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1002 for host mc1040.eqiad.wmnet with OS bookworm [09:42:12] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849693 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin2002 for host mc2040.codfw.wmnet with OS bookworm [10:14:37] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849737 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1002 for host mc1040.eqiad.wmnet with OS bookworm completed: - mc1040 (**PASS**) - Dow... [10:21:25] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849740 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin2002 for host mc2040.codfw.wmnet with OS bookworm completed: - mc2040 (**PASS**) - Dow... [10:47:45] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849762 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1002 for host mc1039.eqiad.wmnet with OS bookworm [10:48:00] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849765 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin2002 for host mc2039.codfw.wmnet with OS bookworm [11:27:05] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849832 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin2002 for host mc2039.codfw.wmnet with OS bookworm completed: - mc2039 (**PASS**) - Dow... [11:42:14] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849861 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin2002 for host mc2038.codfw.wmnet with OS bookworm [11:47:07] 06serviceops, 13Patch-For-Review: Upgrade memcache and memcached gutter pools to Bookworm - https://phabricator.wikimedia.org/T352891#9849868 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1002 for host mc1039.eqiad.wmnet with OS bookworm executed with errors: - mc1039 (**FAIL... [12:33:43] 06serviceops, 10AQS2.0, 10Data Products (Data Products Sprint 14): Metrics api response sometimes returns cached 301 (from kubernetes ??) - https://phabricator.wikimedia.org/T364253#9849957 (10Sfaci) [12:48:18] 06serviceops, 10AQS2.0, 10Data Products (Data Products Sprint 14): Metrics api response sometimes returns cached 301 (from kubernetes ??) - https://phabricator.wikimedia.org/T364253#9849972 (10akosiaris) >>! In T364253#9849840, @Sfaci wrote: > I have never seen `301` when requesting anything to any AQS servi... [13:22:22] Hello, I have to do a small config change on the registry/jwt_authorizer for the new GitLab major version: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1037043. If there are no concerns I'd like to deploy and test that somewhere next week. Let me know if you have any concerns or thoughts. [13:25:50] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9850026 (10kamila) @VRiley-WMF I am in UTC+2, so US mornings are best for me. Would Tuesday work for you? Thank you! [13:38:06] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install kafka-main100[6789] and kafka-main1010 - https://phabricator.wikimedia.org/T363212#9850039 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host kafka-main1010.eqiad.wmnet with OS bullseye [14:09:56] 06serviceops, 10MW-on-K8s, 10Observability-Metrics, 13Patch-For-Review, 10SRE Observability (FY2023/2024-Q4): Create a per-release deployment of statsd-exporter for mw-on-k8s - https://phabricator.wikimedia.org/T365265#9850125 (10akosiaris) Hmm, this complicates things. We have 2 hypotheses in next yea... [14:17:51] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9850140 (10Clement_Goubert) I plan on removing the following images: - `docker-registry.wikimedia.org/golang`, we've moved on to `docker-registry.wikimedia.org/golang... [14:48:04] 06serviceops, 10AQS2.0, 10Data Products (Data Products Sprint 14): Metrics api response sometimes returns cached 301 (from kubernetes ??) - https://phabricator.wikimedia.org/T364253#9850223 (10VirginiaPoundstone) 05Open→03Resolved Thank you @akosiaris for looking into this. @TheDJ and @Theklan I wi... [14:54:25] 06serviceops, 06Machine-Learning-Team, 10MW-on-K8s, 06SRE: Migrate ml-services to mw-api-int - https://phabricator.wikimedia.org/T362316#9850230 (10elukey) 05Open→03Resolved [15:07:24] elukey: looks like the last move to mw-api-int kicked up some saturation, I'll bump mw-api-int replicas a bit [15:08:00] claime: that is weird, that model server is not used a lot [15:08:09] Ah maybe unrelated then? [15:08:25] looks like it's coming back down actually [15:08:31] I mean there are some constant requests due to changeprop, but the rate is usually related to rev-ids being generated [15:08:40] yeah I don't expect drafttopic to cause damages [15:10:07] (famous last words) [15:10:48] lol [15:11:16] there's definitely *something* happening [15:11:37] https://grafana.wikimedia.org/goto/N7X1KOsSR?orgId=1 [15:20:21] Looks like an increase from Apache-HttpAsyncClient/5.3 (Java/11.0.20) https://logstash.wikimedia.org/goto/d3a3d764b2be9d33638fcddc3bcc1113 [15:20:27] That looks like it's cirrus [15:25:38] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE-OnFire, 10Sustainability (Incident Followup): codfw:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366205#9850325 (10akosiaris) [15:26:17] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, 10Sustainability (Incident Followup): eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9850328 (10akosiaris) [22:48:13] 06serviceops, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Migrate charts to Calico Network Policies - https://phabricator.wikimedia.org/T359423#9851908 (10Scott_French)