[00:14:49] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/31 [00:14:54] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/31 [00:15:36] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/24 [00:15:39] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/24 [00:16:20] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/16 [00:16:24] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/16 [00:17:05] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/35 [00:17:05] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/35 [00:17:12] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/35 [00:17:28] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/17 [00:17:33] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/17 [00:17:40] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/23 [00:17:40] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/23 [00:17:47] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/23 [01:16:13] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/26 [01:16:17] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/26 [03:57:04] (03merge) 10countcount: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/17 (owner: 10renovatebot) [04:00:40] (03merge) 10countcount: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/24 (owner: 10renovatebot) [04:15:18] (03update) 10renovatebot: Draft: Update Rust crate tower-http to 0.7.0 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/23 [04:16:26] (03update) 10renovatebot: Draft: Update Rust crate itertools to 0.15.0 [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/16 [07:14:48] (03update) 10samwilson: Use HTTP client from Api class for Wikidata, to set User-Agent [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/7 [07:15:30] (03merge) 10samwilson: Use HTTP client from Api class for Wikidata, to set User-Agent [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/7 [07:32:58] (03open) 10samwilson: Update GitHub references and remove version number from footer [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/10 (https://phabricator.wikimedia.org/T429475) [07:34:49] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1304032 (owner: 10L10n-bot) [07:49:43] 10Cloud-VPS (Debian Bullseye Deprecation): move math-nfs-1 to new instance - https://phabricator.wikimedia.org/T429544#12039332 (10fgiunchedi) >>! In T429544#12033610, @Physikerwelt wrote: > @fgiunchedi thank you so much > > I made it. It worked almost without problems. Only I had to manually edit /etc/fsta... [08:13:29] 10Toolforge, 06tools-platform-team: [jobs-cli] emits a warning to re-create valid jobs - https://phabricator.wikimedia.org/T429231#12039427 (10dcaro) >>! In T429231#12038635, @Wbm1058 wrote: > Just a note that this issue is persisting for jobs which were running before it arose, and are still running. > >... [08:25:54] (03merge) 10dcaro: k8s: Use custom errors for k8s issues [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/313 [08:25:57] (03update) 10dcaro: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 [08:30:05] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.512-20260622082614-e051fd22 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1303 (https://phabricator.wikimedia.org/T359650) [08:30:15] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.512-20260622082614-e051fd22 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1303 (https://phabricator.wikimedia.org/T359650) [08:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:40:55] 06cloud-services-team, 10Toolforge: Cannot join Toolforge: Striker throws IntegrityError while saving user - https://phabricator.wikimedia.org/T429759 (10Kavaljeet_Singh) 03NEW [08:42:35] 06cloud-services-team, 10Cloud-VPS, 07Wikimedia-Incident: Openstack cinder volumes backups are broken - https://phabricator.wikimedia.org/T428867#12039562 (10LSobanski) [08:43:49] 06cloud-services-team, 10Toolforge: Cannot join Toolforge: Striker throws IntegrityError while saving user - https://phabricator.wikimedia.org/T429759#12039567 (10fgiunchedi) From striker logs: `root@cloudweb1004:~# docker logs --tail 10000 striker.service 2>&1 | grep -i kavalje` ` ed_query)\n ^^^^^^... [08:44:03] 06cloud-services-team, 10Striker: Cannot join Toolforge: Striker throws IntegrityError while saving user - https://phabricator.wikimedia.org/T429759#12039568 (10fgiunchedi) [08:50:03] 10Cloud-VPS (Debian Bullseye Deprecation): move math-nfs-1 to new instance - https://phabricator.wikimedia.org/T429544#12039607 (10Physikerwelt) > wmcs-prepare-cinder-volume It also handles the formatting, so I was afraid that it might reformat the existing disk. The puppet config links above says ` profil... [09:01:19] (03update) 10countcount: Update Rust crate itertools to 0.15.0 [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/16 (owner: 10renovatebot) [09:01:29] (03merge) 10countcount: Update Rust crate itertools to 0.15.0 [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/16 (owner: 10renovatebot) [09:03:13] (03update) 10dcaro: Do not require --filelog if --filelog-std(out|err) is used [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/158 (https://phabricator.wikimedia.org/T423828) (owner: 10fnegri) [09:04:20] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: jobs-cli: stop resolving dynamic fields in the cli - https://phabricator.wikimedia.org/T423828#12039707 (10dcaro) 05Resolved→03Open [09:04:37] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: jobs-cli: stop resolving dynamic fields in the cli - https://phabricator.wikimedia.org/T423828#12039710 (10dcaro) a:05Raymond_Ndibe→03fnegri [09:07:54] (03update) 10fnegri: Do not require --filelog if --filelog-std(out|err) is used [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/158 (https://phabricator.wikimedia.org/T423828 https://phabricator.wikimedia.org/T428354) [09:08:03] 10Tool-translatetagger, 06Indic MediaWiki Developers UG, 06Indic-TechCom: Put inline chunks of code in tvars instead of leaving them out of translate tags completely. - https://phabricator.wikimedia.org/T393255#12039722 (10Aafi) >>! In T393255#10802048, @theprotonade wrote: > I was reading some documentation... [09:08:26] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: jobs-cli: stop resolving dynamic fields in the cli - https://phabricator.wikimedia.org/T423828#12039726 (10dcaro) 05Open→03Resolved a:05fnegri→03Raymond_Ndibe [09:09:33] (03update) 10fnegri: Do not require --filelog if --filelog-std(out|err) is used [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/158 (https://phabricator.wikimedia.org/T428354) [09:11:35] 10Tool-translatetagger, 06Indic MediaWiki Developers UG, 06Indic-TechCom: Consistent tvar convention - https://phabricator.wikimedia.org/T428033#12039737 (10Aafi) All localisable links (leading to Wikipedia pages across editions) need to be skipped in the tvar mechanism. Example, [[en:page title|page display... [09:12:05] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12039741 (10dcaro) [09:13:50] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12039742 (10dcaro) [09:20:36] (03approved) 10fnegri: jobs-api: bump to 0.0.512-20260622082614-e051fd22 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1303 (https://phabricator.wikimedia.org/T359650) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:21:54] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12039777 (10dcaro) [09:40:39] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12039875 (10dcaro) [09:41:09] 10Toolforge, 06tools-platform-team: [jobs-cli] emits a warning to re-create valid jobs - https://phabricator.wikimedia.org/T429231#12039876 (10Wbm1058) >>! In T429231#12039427, @dcaro wrote: >>>! In T429231#12038635, @Wbm1058 wrote: >> Just a note that this issue is persisting for jobs which were running b... [09:52:46] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12039906 (10dcaro) [09:56:50] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12039914 (10dcaro) [10:01:37] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12039940 (10dcaro) [10:02:15] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12039941 (10dcaro) [10:04:40] (03PS1) 10Tiziano Fogli: slothslos/report2drive: remove report2drive::instances [labs/private] - 10https://gerrit.wikimedia.org/r/1304759 (https://phabricator.wikimedia.org/T425795) [10:05:06] (03CR) 10Tiziano Fogli: [C:03+2] slothslos/report2drive: remove report2drive::instances [labs/private] - 10https://gerrit.wikimedia.org/r/1304759 (https://phabricator.wikimedia.org/T425795) (owner: 10Tiziano Fogli) [10:05:09] (03CR) 10Tiziano Fogli: [V:03+2 C:03+2] slothslos/report2drive: remove report2drive::instances [labs/private] - 10https://gerrit.wikimedia.org/r/1304759 (https://phabricator.wikimedia.org/T425795) (owner: 10Tiziano Fogli) [10:07:27] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12039960 (10dcaro) [10:08:07] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12039961 (10dcaro) [10:10:04] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [10:14:51] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/31 [10:15:18] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/16 [10:15:40] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/35 [10:15:57] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/23 [10:16:01] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/26 [10:22:12] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [10:23:48] (03PS1) 10Tiziano Fogli: slothslos/report2drive: adjust file extension [labs/private] - 10https://gerrit.wikimedia.org/r/1304763 (https://phabricator.wikimedia.org/T425795) [10:24:32] (03CR) 10Tiziano Fogli: [C:03+2] slothslos/report2drive: adjust file extension [labs/private] - 10https://gerrit.wikimedia.org/r/1304763 (https://phabricator.wikimedia.org/T425795) (owner: 10Tiziano Fogli) [10:24:34] (03CR) 10Tiziano Fogli: [V:03+2 C:03+2] slothslos/report2drive: adjust file extension [labs/private] - 10https://gerrit.wikimedia.org/r/1304763 (https://phabricator.wikimedia.org/T425795) (owner: 10Tiziano Fogli) [10:31:51] 10Toolforge, 06tools-platform-team: [jobs-cli] emits a warning to re-create valid jobs - https://phabricator.wikimedia.org/T429231#12040016 (10dcaro) 05Resolved→03Open @Wbm1058 I think I found the issue, will fix, we are storing the aliases of the image in storage, and those might change also eventually [10:34:13] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [10:35:18] (03open) 10dcaro: core: when comparing storage and runtime, ignore image aliases [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/315 [10:39:29] (03update) 10dcaro: core: when comparing storage and runtime, ignore image aliases [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/315 [10:48:18] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [10:54:30] 10Cloud-VPS (Project-requests): Request creation of wikimedia_commons_uploader_telegram_bot VPS project - https://phabricator.wikimedia.org/T429729#12040073 (10fgiunchedi) Thank you for reaching out and the links; I'm not familiar with the telegram bot api, how would the data flow look like in case of uploads? w... [10:59:29] 06cloud-services-team, 10Cloud-VPS: [ceph] Enable encrypted client traffic for the ceph clusters - https://phabricator.wikimedia.org/T294432#12040074 (10Volans) With the above patches the cross-site traffic (cinder volume backups) is now encrypted and the backups should fail if they are unable, not falling bac... [11:12:01] (03approved) 10dcaro: jobs-api: bump to 0.0.512-20260622082614-e051fd22 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1303 (https://phabricator.wikimedia.org/T359650) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [11:12:12] (03merge) 10dcaro: jobs-api: bump to 0.0.512-20260622082614-e051fd22 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1303 (https://phabricator.wikimedia.org/T359650) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [11:26:42] 06cloud-services-team: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12040193 (10fgiunchedi) >>! In T429387#12030413, @Andrew wrote: >>>! In T429387#12028019, @fgiunchedi wrote: >> Later in the thread there's a few suggested troublesh... [11:27:29] 06cloud-services-team: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12040206 (10fgiunchedi) >>! In T429387#12040193, @fgiunchedi wrote: >>>! In T429387#12030413, @Andrew wrote: >>>>! In T429387#12028019, @fgiunchedi wrote: >>> Later... [11:28:30] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: Establish a blackbox network probe vantage point into cloud realm - https://phabricator.wikimedia.org/T429451#12040218 (10fgiunchedi) Ack, thank you both! I've added a meeting note... [11:30:11] 06cloud-services-team, 10Cloud-VPS: Monitoring/metrics for trove instances - https://phabricator.wikimedia.org/T402738#12040229 (10fgiunchedi) >>! In T402738#12033703, @Andrew wrote: >>>! In T402738#12033646, @fgiunchedi wrote: >> wrt metrics, I'm wondering how easy it would be to bundle node-exporter in the t... [11:32:23] 06cloud-services-team, 10Data-Services: Fix tools dumps availability over NFS check - https://phabricator.wikimedia.org/T188137#12040247 (10fgiunchedi) 05Open→03Invalid toolschecker is no longer [11:33:17] 06cloud-services-team, 10Toolforge: Add a toolschecker to test the tools email relay - https://phabricator.wikimedia.org/T280741#12040250 (10fgiunchedi) 05Open→03Invalid toolschecker is no longer [11:33:31] 06cloud-services-team, 10Toolforge: toolschecker: naming refresh - https://phabricator.wikimedia.org/T277542#12040253 (10fgiunchedi) 05Open→03Invalid Toolschecker is no longer [11:33:40] 06cloud-services-team, 10Toolforge, 07Documentation: Document the toolschecker application in wikitech - https://phabricator.wikimedia.org/T246157#12040256 (10fgiunchedi) 05Open→03Invalid Toolschecker is no longer [11:54:38] 06cloud-services-team, 10Toolforge: [lima-kilo] kind cluster fails to bootstrap - https://phabricator.wikimedia.org/T429783 (10fnegri) 03NEW [12:04:41] 06cloud-services-team, 10Toolforge: [lima-kilo] kind cluster fails to bootstrap - https://phabricator.wikimedia.org/T429783#12040365 (10fnegri) The workaround is not enough, Ansible is unable to connect to the cluster: ` TASK [foxtrot_ldap : Deploy into k8s] ***************************************************... [12:07:46] 06cloud-services-team, 10Toolforge: [lima-kilo] kind cluster fails to bootstrap - https://phabricator.wikimedia.org/T429783#12040379 (10fnegri) [12:07:51] 06cloud-services-team, 10Toolforge: [lima-kilo] kind cluster fails to bootstrap - https://phabricator.wikimedia.org/T429783#12040380 (10fnegri) Updated the workaround in the task description to include an additional command: `lang=shell-session fran@lima-kilo:~$ kind get kubeconfig --name toolforge --internal... [12:08:12] 06cloud-services-team, 10Toolforge: [lima-kilo] kind cluster fails to bootstrap - https://phabricator.wikimedia.org/T429783#12040381 (10fnegri) [12:08:34] 06cloud-services-team, 10Toolforge: [lima-kilo] kind cluster fails to bootstrap - https://phabricator.wikimedia.org/T429783#12040382 (10fnegri) [12:25:00] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/41 [12:38:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:49:24] (03update) 10dcaro: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 [12:49:40] (03update) 10dcaro: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 [12:59:55] 10Toolforge, 06tools-platform-team: [envvars-api, jobs-api] Continuous job failed to start due to missing envvar specified in secrets specification - https://phabricator.wikimedia.org/T422929#12040594 (10dcaro) [13:02:05] (03update) 10dcaro: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 [13:04:57] 06cloud-services-team, 10Toolforge: [lima-kilo] kind cluster fails to bootstrap - https://phabricator.wikimedia.org/T429783#12040622 (10dcaro) It worked for me during the weekend, I'll retry when I have a minute. [13:06:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [13:11:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [13:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:23:10] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 10Data-Services: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12040673 (10fgiunchedi) [13:26:54] 06cloud-services-team: Update all cloud-vps Bullseye NFS servers - https://phabricator.wikimedia.org/T429793 (10Andrew) 03NEW [13:27:59] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-platform-team: Reach out to Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T428196#12040703 (10Andrew) >>! In T428196#12035176, @Don-vip wrote: > Hi @Andrew , I received this e-mail as maintainer of the... [13:28:19] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 10Data-Services, 06Data-Platform-SRE, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12040707 (10fgiunchedi) @brouberol @BTullis please let me know what you think of the load bala... [13:30:46] 06cloud-services-team, 10Toolforge: [lima-kilo] toolforge_run_ansible does not detect ARM arch - https://phabricator.wikimedia.org/T429795 (10fnegri) 03NEW [13:31:07] 10Cloud-VPS (Debian Bullseye Deprecation): move math-nfs-1 to new instance - https://phabricator.wikimedia.org/T429544#12040738 (10Andrew) >>! In T429544#12039331, @fgiunchedi wrote: > I'm not sure if we have an end to end guide for users on how to self-service migrate their nfs servers. Maybe @Andrew knows... [13:33:55] 06cloud-services-team, 10Toolforge: [lima-kilo] kind cluster fails to bootstrap - https://phabricator.wikimedia.org/T429783#12040753 (10dcaro) Seems to be getting stuck, not sure if related, but it seems to be trying to get to harbor.local and failing: ` Jun 22 15:28:00 toolslocal dockerd[10560]: time="2026-06... [13:37:05] 06cloud-services-team, 10Toolforge: [lima-kilo] kind cluster fails to bootstrap - https://phabricator.wikimedia.org/T429783#12040759 (10dcaro) Interestingly enough, for me it did get unstuck, and went through, and failed when installing harbor: ` Error response from daemon: Head "https://registry-1.docker.io/v... [13:48:51] 06cloud-services-team, 10Cloud-VPS: Monitoring/metrics for trove instances - https://phabricator.wikimedia.org/T402738#12040784 (10Andrew) >>! In T402738#12040229, @fgiunchedi wrote: >>>! In T402738#12033703, @Andrew wrote: >>>>! In T402738#12033646, @fgiunchedi wrote: >>> wrt metrics, I'm wondering how easy i... [13:51:48] FIRING: PuppetDisabled: Puppet disabled on cloudcephosd2005-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [13:55:05] 06cloud-services-team, 10Toolforge: [lima-kilo] kind cluster fails to bootstrap - https://phabricator.wikimedia.org/T429783#12040811 (10dcaro) >>! In T429783#12040759, @dcaro wrote: > Interestingly enough, for me it did get unstuck, and went through, and failed when installing harbor: > ` > Error response from... [14:21:48] RESOLVED: PuppetDisabled: Puppet disabled on cloudcephosd2005-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [14:22:32] (03open) 10dcaro: ansible: do harbor before kind [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/327 [14:23:19] 06cloud-services-team, 10Toolforge: [lima-kilo] kind cluster fails to bootstrap - https://phabricator.wikimedia.org/T429783#12040918 (10dcaro) @fnegri This seems to help for me https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/327 (with the kind hanging or delaying if there's no cac... [14:29:55] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: Establish a blackbox network probe vantage point into cloud realm - https://phabricator.wikimedia.org/T429451#12040954 (10LSobanski) p:05Triage→03Medium [14:43:03] (03open) 10fnegri: api_get_jobs: compute correct value for missing fields [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/316 (https://phabricator.wikimedia.org/T428354) [14:51:03] (03open) 10dcaro: blubber: don't chmod -R 777 as it's quite slow [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/163 [14:58:07] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-36 (2026-06-16 to 2026-06-30)), 07OKR-Work: Fix type, media type, and array issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428149#12041111 (10MGoncalves-WMF) a:03MGoncalves-WMF [15:08:10] (03open) 10dcaro: config: update the default builder/runner [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/164 [15:08:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:08:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [15:12:34] (03update) 10dcaro: blubber: don't chmod -R 777 as it's quite slow [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/163 [15:15:34] (03update) 10dcaro: blubber: don't chmod -R 777 as it's quite slow [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/163 [15:17:44] (03update) 10dcaro: ansible: do harbor before kind [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/327 [15:18:14] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-platform-team: Reach out to Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T428196#12041187 (10Don-vip) >>! In T428196#12040703, @Andrew wrote: > You're right! I will take this on -- I just opened T42979... [15:39:43] (03update) 10dcaro: blubber: don't chmod -R 777 as it's quite slow [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/163 [15:43:50] 06cloud-services-team, 10PAWS: PAWS disk space is running out - https://phabricator.wikimedia.org/T429578#12041432 (10Andrew) 05Open→03Resolved a:03Andrew There are some very brief docs about this here: https://wikitech.wikimedia.org/wiki/PAWS/Admin#Deleting_user_data_in_case_of_spam_or_credential_le... [15:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:49:38] (03update) 10fnegri: core: when comparing storage and runtime, ignore image aliases [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/315 (owner: 10dcaro) [15:49:40] (03update) 10fnegri: core: when comparing storage and runtime, ignore image aliases [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/315 (owner: 10dcaro) [15:49:47] (03approved) 10fnegri: core: when comparing storage and runtime, ignore image aliases [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/315 (owner: 10dcaro) [15:49:57] (03update) 10fnegri: core: when comparing storage and runtime, ignore image aliases [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/315 (owner: 10dcaro) [15:53:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [15:54:43] (03approved) 10dcaro: core: when comparing storage and runtime, ignore image aliases [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/315 [15:54:47] (03merge) 10dcaro: core: when comparing storage and runtime, ignore image aliases [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/315 [15:56:17] (03update) 10dcaro: blubber: don't chmod -R 777 as it's quite slow [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/163 [15:58:33] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.513-20260622155500-30972a04 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1304 (https://phabricator.wikimedia.org/T429231) [15:58:34] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.513-20260622155500-30972a04 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1304 (https://phabricator.wikimedia.org/T429231) [16:09:29] 10VPS-project-Codesearch: Index the slothslo gitlab repo in codesearch - https://phabricator.wikimedia.org/T429819 (10elukey) 03NEW [16:15:44] (03update) 10dcaro: blubber: don't chmod -R 777 as it's quite slow [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/163 [16:17:24] (03update) 10dcaro: blubber: don't chmod -R 777 as it's quite slow [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/163 [16:20:43] 10Cloud-VPS (Project-requests): Request creation of wikimedia_commons_uploader_telegram_bot VPS project - https://phabricator.wikimedia.org/T429729#12041675 (10vitaly-zdanevich) Data flow: user sends photo(s) (even zip/rar are supported) - Telegram webhook routes them to this VPS - this VPS uploads photo(s) to W... [16:23:03] (03update) 10dcaro: blubber: don't chmod -R 777 as it's quite slow [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/163 [16:25:47] (03update) 10dcaro: blubber: don't chmod -R 777 as it's quite slow [repos/cloud/toolforge/builds-api] (update_builder_default) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/163 [16:26:08] (03update) 10dcaro: blubber: don't chmod -R 777 as it's quite slow [repos/cloud/toolforge/builds-api] (update_builder_default) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/163 [16:37:40] (03PS1) 10Dzahn: add repos/sre/sloth, repos/sre/slothslos to config [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1304848 (https://phabricator.wikimedia.org/T429819) [16:38:42] (03CR) 10CI reject: [V:04-1] add repos/sre/sloth, repos/sre/slothslos to config [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1304848 (https://phabricator.wikimedia.org/T429819) (owner: 10Dzahn) [16:38:51] 10VPS-project-Codesearch, 13Patch-For-Review: Index the slothslo gitlab repo in codesearch - https://phabricator.wikimedia.org/T429819#12041781 (10Dzahn) a:03Dzahn [17:01:03] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Privacy Engineering, and 2 others: Add global_edit_count to wikireplicas - https://phabricator.wikimedia.org/T344108#12041857 (10sbassett) [17:06:56] FIRING: SystemdUnitDown: The service unit security_group_ssh-from-restricted-bastion_to_project_bd825ee46c214f9e88126a2b290d88f8.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [17:10:22] FIRING: HAProxyBackendUnavailable: HAProxy service wikireplica-db-web-s2 backend clouddb1014.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:13:40] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 06Tech-Docs-Team, 07OKR-Work: Fix description and example issues for page, revision, search, and file paths in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428153#12041864 (10TBurmeister) a:03TBurmeister [17:15:22] RESOLVED: HAProxyBackendUnavailable: HAProxy service wikireplica-db-web-s2 backend clouddb1014.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:44:37] 10Cloud-VPS (Project-requests): Request creation of wikimedia_commons_uploader_telegram_bot VPS project - https://phabricator.wikimedia.org/T429729#12041926 (10Aklapper) As "user bot passwords" were mentioned, what is the authorization workflow between a user using this bot and the uploader account of the file o... [17:48:48] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12041963 (10Jclark-ctr) a:05Jclark-ctr→03None [18:10:56] (03CR) 10Dzahn: "the CI downvote seems to be unrelatedly caused by mirrors.wikimedia.org being shut down https://phabricator.wikimedia.org/T416707" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1304848 (https://phabricator.wikimedia.org/T429819) (owner: 10Dzahn) [18:14:26] RESOLVED: SystemdUnitDown: The service unit security_group_ssh-from-restricted-bastion_to_project_bd825ee46c214f9e88126a2b290d88f8.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:16:29] 06cloud-services-team, 10PAWS: PAWS disk space is running out - https://phabricator.wikimedia.org/T429578#12042051 (10Andrew) 05Resolved→03Open followups: [] Add a runbook to the alert -- if it's generic, make an index page for all nfs full alerts [] Add a daily timer to delete all files older than 2 week... [18:17:26] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/41 (owner: 10l10n-bot) [18:17:30] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/41 (owner: 10l10n-bot) [18:22:03] 10Cloud-VPS (Project-requests): Request creation of wikimedia_commons_uploader_telegram_bot VPS project - https://phabricator.wikimedia.org/T429729#12042061 (10vitaly-zdanevich) Bot asks for username and password. We store encrypted passwords (AES-256-GCM) in sqlite. [18:22:48] 10Cloud-VPS (Project-requests): Request creation of wikimedia_commons_uploader_telegram_bot VPS project - https://phabricator.wikimedia.org/T429729#12042067 (10vitaly-zdanevich) ...and bot deletes the Telegram message with the password. [18:23:26] 10Cloud-VPS (Project-requests): Request creation of wikimedia_commons_uploader_telegram_bot VPS project - https://phabricator.wikimedia.org/T429729#12042068 (10vitaly-zdanevich) ...decryption happens only in memory, per upload. [18:29:21] (03open) 10dcaro: cleanup by image name [repos/cloud/toolforge/builds-api] (dont_chmod_on_build) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/165 [18:29:48] (03update) 10dcaro: cleanup by image name [repos/cloud/toolforge/builds-api] (dont_chmod_on_build) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/165 [18:30:11] (03update) 10dcaro: cleanup by image name [repos/cloud/toolforge/builds-api] (dont_chmod_on_build) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/165 [18:31:23] 10VPS-project-devtools, 10VPS-project-Phabricator: Puppet broken on phabricator-bookworm-3.devtools.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T428069#12042082 (10Arnoldokoth) I terminated this host so should not be an issue now. I'll mark this ticket as resolved. [18:31:36] 10VPS-project-devtools, 10VPS-project-Phabricator: Puppet broken on phabricator-bookworm-3.devtools.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T428069#12042083 (10Arnoldokoth) 05Open→03Resolved [18:33:41] (03update) 10dcaro: config: update the default builder/runner [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/164 [18:43:03] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 06Tech-Docs-Team, 07OKR-Work: Fix description and example issues for page, revision, search, and file paths in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428153#12042121 (10TBurmeister) 05Open→03In progress [18:57:13] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.nfs.migrate_service [18:57:17] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) [19:04:04] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 06Tech-Docs-Team, 07OKR-Work: Fix description and example issues for page, revision, search, and file paths in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428153#12042187 (10TBurmeister) [19:05:17] !log andrew@cloudcumin1001 video START - Cookbook wmcs.nfs.migrate_service [19:05:26] !log andrew@cloudcumin1001 video END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) [19:05:54] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 06Tech-Docs-Team, 07OKR-Work: Fix description and example issues for page, revision, search, and file paths in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428153#12042202 (10TBurmeister) None of the paths listed in the task description ha... [19:09:36] !log andrew@cloudcumin1001 video START - Cookbook wmcs.nfs.add_server [19:23:38] !log andrew@cloudcumin1001 video END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) [19:46:43] !log andrew@cloudcumin1001 video START - Cookbook wmcs.nfs.add_server [19:47:23] !log andrew@cloudcumin1001 video END (ERROR) - Cookbook wmcs.nfs.add_server (exit_code=97) [19:47:28] !log andrew@cloudcumin1001 video START - Cookbook wmcs.nfs.add_server [19:59:24] !log andrew@cloudcumin1001 video END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) [20:10:16] 10Tool-wikimedia-attribution, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07OKR-Work: Include contributor_count in Attribution API responses - https://phabricator.wikimedia.org/T429834 (10HCoplin-WMF) 03NEW [20:20:26] 10Tool-wikimedia-attribution, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07OKR-Work: Include contributor_count in Attribution API responses - https://phabricator.wikimedia.org/T429834#12042495 (10HCoplin-WMF) [20:26:54] 10VPS-Projects, 10VisualEditor, 10Catalyst (PatchDemo): Clean up the patchdemo3.visualeditor.eqiad1.wikimedia.cloud instance - https://phabricator.wikimedia.org/T429836 (10matmarex) 03NEW [21:26:10] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#12042784 (10Dzahn) The bookworm instance has been deleted in T428069#12042082 [23:04:55] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#12043086 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host clouddb1032.eqiad.wmnet with OS trixie [23:05:27] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#12043088 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host clouddb1033.eqiad.wmnet with OS trixie [23:06:06] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#12043089 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host clouddb1031.eqiad.wmnet with OS trixie [23:06:15] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#12043090 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host clouddb1030.eqiad.wmnet with OS trixie [23:40:34] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#12043180 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host clouddb1032.eqiad.wmnet with OS trixie completed: - cl... [23:41:04] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#12043181 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host clouddb1029.eqiad.wmnet with OS trixie [23:47:14] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#12043186 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host clouddb1028.eqiad.wmnet with OS trixie [23:47:37] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#12043187 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host clouddb1030.eqiad.wmnet with OS trixie completed: - cl... [23:48:08] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#12043188 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host clouddb1026.eqiad.wmnet with OS trixie [23:52:07] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#12043205 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host clouddb1031.eqiad.wmnet with OS trixie completed: - cl... [23:52:07] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#12043204 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host clouddb1033.eqiad.wmnet with OS trixie executed with e... [23:53:10] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#12043206 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host clouddb1033.eqiad.wmnet with OS trixie [23:54:40] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#12043207 (10Jclark-ctr) [23:55:27] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#12043208 (10Jclark-ctr) [23:57:44] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#12043209 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host clouddb1027.eqiad.wmnet with OS trixie