[08:36:09] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install mc-misc200[12] - https://phabricator.wikimedia.org/T372800#10858342 (10MoritzMuehlenhoff) 05Resolved→03Open mc-2001 isn't up and there's also no serial console [08:36:49] 06serviceops, 06Growth-Team, 07Wikimedia-production-error: growthexperiments-updatementeedata-s1 errors out on DB timeout - https://phabricator.wikimedia.org/T395299 (10Michael) 03NEW [08:37:20] 06serviceops, 06Growth-Team, 10GrowthExperiments-Mentorship, 07Wikimedia-production-error: growthexperiments-updatementeedata-s1 errors out on DB timeout - https://phabricator.wikimedia.org/T395299#10858356 (10Michael) [09:13:40] 06serviceops, 06Growth-Team, 10GrowthExperiments-Mentorship, 07Wikimedia-production-error: growthexperiments-updatementeedata-s1 errors out on DB timeout - https://phabricator.wikimedia.org/T395299#10858539 (10Urbanecm_WMF) At the first sight, this looks like a transient etcd failure, given the jobs manage... [09:24:54] 06serviceops, 10Ceph, 06Data-Persistence, 10SRE-swift-storage: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#10858578 (10MatthewVernon) >>! In T394476#10856108, @akosiaris wrote: >> If you want to do some testing, I could set you up with a test account on apus. > > That w... [09:32:38] 06serviceops, 10Ceph, 06Data-Persistence, 10SRE-swift-storage: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#10858614 (10elukey) Thanks a lot @MatthewVernon ! >>! In T394476#10858578, @MatthewVernon wrote: >>>! In T394476#10856108, @akosiaris wrote: >>> If you want to do... [09:58:05] 06serviceops, 10ChangeProp: Exclude 404s from changeprop/changeprop-jobqueue logs - https://phabricator.wikimedia.org/T395132#10858702 (10hnowlan) 05Open→03Resolved a:03hnowlan [11:30:53] 06serviceops, 06Growth-Team, 10GrowthExperiments-Mentorship, 07Wikimedia-production-error: growthexperiments-updatementeedata-s1 errors out on DB timeout - https://phabricator.wikimedia.org/T395299#10858944 (10Clement_Goubert) >>! In T395299#10858537, @Urbanecm_WMF wrote: > At the first sight, this looks l... [11:45:10] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 07Wikimedia-production-error: growthexperiments-fixlinkrecommendationdata-dryrun 'link-recommendation' is not a link recommendation task type error - https://phabricator.wikimedia.org/T395249#10858980 (10Clement_Goubert) It runs every day at... [12:02:56] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work, 13Patch-For-Review: Enable memcached in the orchestrator - https://phabricator.wikimedia.org/T391986#10859028 (10cmassaro) >>! In T391986#10856797, @jijiki wrote: > @cmassaro if you would like to do any tests,... [12:11:03] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work: Tidy app chart modules in function-orchestrator - https://phabricator.wikimedia.org/T395254#10859112 (10cmassaro) @jijiki Thank you for making this task. Is this something the AW team should have done--i.e., did... [12:26:59] 06serviceops, 10MediaWiki-extensions-ReadingLists, 06MW-Interfaces-Team, 10RESTBase Sunsetting, and 4 others: Switchover plan from RESTbase to REST Gateway for Reading Lists endpoints - https://phabricator.wikimedia.org/T384891#10859151 (10MSantos) [12:47:20] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Create ValidatingAdmissionPolicy to permit hostPath mounts for mediawiki - https://phabricator.wikimedia.org/T395225#10859217 (10jijiki) 05Open→03In progress p:05Triage→03Medium [14:09:25] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install mc-misc200[12] - https://phabricator.wikimedia.org/T372800#10859557 (10Jhancock.wm) a:05jijiki→03Jhancock.wm [14:32:58] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: wipe-cluster cookbook should check if systemd services have started properly - https://phabricator.wikimedia.org/T389086#10859631 (10JMeybohm) 05Open→03Resolved a:03JMeybohm [16:14:38] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work, 13Patch-For-Review: Enable memcached in the orchestrator - https://phabricator.wikimedia.org/T391986#10860186 (10jijiki) @cmassaro https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1149633 needs t... [17:01:52] 06serviceops, 10MediaWiki-extensions-ReadingLists, 06MW-Interfaces-Team, 10RESTBase Sunsetting, and 4 others: Switchover plan from RESTbase to REST Gateway for Reading Lists endpoints - https://phabricator.wikimedia.org/T384891#10860473 (10BPirkle) Changes look good, +1'd. I'll let @HCoplin-WMF and/or @Sed... [17:41:48] 06serviceops, 10decommission-hardware, 13Patch-For-Review: decommission mw135[8-9], mw136[4-6], mw137[2-3], mw140[0-4], mw1406, mw14[11-13] - https://phabricator.wikimedia.org/T383227#10860762 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jasmine@cumin1002 for hosts: `wikikube-worker[10... [17:55:44] 06serviceops, 06SRE Observability: Stale labels applied when the pod IP of a terminated k8s job is reused - https://phabricator.wikimedia.org/T395052#10860874 (10Scott_French) a:03Scott_French Alright, https://gerrit.wikimedia.org/r/1149505 is merged and was applied in codfw and eqiad around 17:30 and 17:40... [18:36:54] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work: Tidy app chart modules in function-orchestrator - https://phabricator.wikimedia.org/T395254#10861028 (10Jdforrester-WMF) [18:36:56] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work, 13Patch-For-Review: Enable memcached in the orchestrator - https://phabricator.wikimedia.org/T391986#10861029 (10Jdforrester-WMF) [19:54:35] 06serviceops, 10MediaWiki-extensions-ReadingLists, 06MW-Interfaces-Team, 10RESTBase Sunsetting, and 4 others: Switchover plan from RESTbase to REST Gateway for Reading Lists endpoints - https://phabricator.wikimedia.org/T384891#10861236 (10Dbrant) I've done a round of testing of our Reading Lists feature a... [21:18:54] 06serviceops, 10DNS, 06SRE, 06Traffic, 13Patch-For-Review: Create redirect from tj.*.org to tg.*.org - https://phabricator.wikimedia.org/T393803#10861433 (10Scott_French) Ah, I realized when cross-checking https://gerrit.wikimedia.org/r/1148981 against DNS that the `tj` records have not been added yet th...