[00:13:55] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/6 [00:14:01] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/6 [00:14:11] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/19 [00:14:16] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/19 [00:14:26] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/14 [00:14:30] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/14 [00:15:49] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/46 [00:15:54] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/46 [00:16:38] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/74 [00:16:43] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/74 [00:17:23] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/29 [00:17:27] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/29 [00:18:09] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/38 [00:18:14] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/38 [00:18:35] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/23 [00:18:38] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/23 [00:18:46] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/26 [00:18:48] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/26 [00:18:53] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/26 [00:18:56] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/29 [00:19:02] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/29 [00:19:08] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/29 [01:13:29] (03update) 10raymond-ndibe: compile: use PWD instead of hardcoded /app for locale paths [repos/cloud/toolforge/buildpacks/locale-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/locale-buildpack/-/merge_requests/4 (https://phabricator.wikimedia.org/T428230) [01:29:12] 10Cloud-VPS (Debian Bullseye Deprecation), 10Countervandalism-Network: Upgrade cvn fleet from Debian 11 Bullseye to Debian 12 Bookworm (PHP 7.4 > PHP 8.2) - https://phabricator.wikimedia.org/T395164#12177466 (10Krinkle) Sure, feel free to destroy those and re-create with Trixie. The "new" Bookworm hosts aren'... [02:14:40] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-api] support exposing continuous jobs to the internet - https://phabricator.wikimedia.org/T423408#12177472 (10Raymond_Ndibe) 05In progress→03Resolved [02:14:49] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-cli] support publishing continuous job to the internet - https://phabricator.wikimedia.org/T423410#12177476 (10Raymond_Ndibe) 05In progress→03Resolved [02:14:51] 10Toolforge (Push-to-Deploy), 06tools-platform-team: webservice-cli: detect conflict with HTTPRoutes created by the jobs framework - https://phabricator.wikimedia.org/T431417#12177480 (10Raymond_Ndibe) 05Open→03Resolved [02:57:58] 10Tool-wikinewsie: Algorithmic feed mix for Wikinewsie - https://phabricator.wikimedia.org/T433425#12177485 (10ZhaoFJx) a:03Habst [04:57:52] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [04:58:45] (03open) 10raymond-ndibe: tests: improve test coverage [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/362 [04:59:19] (03open) 10raymond-ndibe: refactor: move observability fields out of CommonJob into ObservabilityJob [repos/cloud/toolforge/jobs-api] (increase_test_coverage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 [04:59:52] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [05:00:17] (03update) 10raymond-ndibe: support webservice job-type [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [05:00:36] (03update) 10raymond-ndibe: models: support webservice job-type [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [05:07:30] (03update) 10raymond-ndibe: models: support webservice job-type [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [05:07:50] (03update) 10raymond-ndibe: models: support webservice job-type [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [05:45:12] (03CR) 10Mahveotm: [C:03+2] updated the copyright tag to reflect current year [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/561853 (owner: 10Mahveotm) [05:45:41] (03Merged) 10jenkins-bot: updated the copyright tag to reflect current year [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/561853 (owner: 10Mahveotm) [07:04:34] (03open) 10countcount: Remove obsolete Vue release-age exclusions [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/20 [07:04:43] (03open) 10countcount: Remove stale happy-dom release-age exception [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/7 [07:06:17] 10Tool-itwiki: Improve the "Template:Conteggio cancellazioni/In corso/Voce" to automatically calculate the "temperature" 0-100 - https://phabricator.wikimedia.org/T433807#12177654 (10valerio.bozzolan) [07:08:07] 10Tool-itwiki: Improve the "Template:Conteggio cancellazioni/In corso/Voce" to automatically calculate the "temperature" 0-100 - https://phabricator.wikimedia.org/T433807#12177660 (10valerio.bozzolan) [07:27:57] (03merge) 10countcount: Remove obsolete Vue release-age exclusions [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/20 [07:28:22] (03merge) 10countcount: Remove stale happy-dom release-age exception [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/7 [07:35:41] 10Tools: Consultations Stats tool is not finding everyone (different, smaller number of participants each time) - https://phabricator.wikimedia.org/T433785#12177704 (10samuelguebo) 05Open→03In progress a:03samuelguebo [07:49:25] 10VPS-project-Codesearch: searching for nothing should display an error to the user - https://phabricator.wikimedia.org/T432987#12177746 (10Jonas2356) p:05Triage→03High a:03-25FJ25 [08:02:05] 10VPS-project-Codesearch: searching for nothing should display an error to the user - https://phabricator.wikimedia.org/T432987#12177759 (10JJMC89) p:05High→03Triage a:05-25FJ25→03None [08:08:53] 10Tools: Consultations Stats tool is not finding everyone (different, smaller number of participants each time) - https://phabricator.wikimedia.org/T433785#12177773 (10samuelguebo) Hey @WhatamIdoing, the inconsistent numbers seem to be caused by some API rate limiting. I've added two changes that should help: -... [08:10:22] (03update) 10dcaro: refactor: move observability fields out of CommonJob into ObservabilityJob [repos/cloud/toolforge/jobs-api] (increase_test_coverage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (owner: 10raymond-ndibe) [08:13:08] (03update) 10dcaro: models: support webservice job-type [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) (owner: 10raymond-ndibe) [08:13:30] (03update) 10dcaro: models: support webservice job-type [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) (owner: 10raymond-ndibe) [08:14:00] (03update) 10renovatebot: Update dependency @napi-rs/wasm-runtime to v1.2.2 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/5 [08:14:21] (03update) 10dcaro: models: support webservice job-type [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) (owner: 10raymond-ndibe) [08:14:26] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/19 [08:14:46] (03update) 10dcaro: refactor: move observability fields out of CommonJob into ObservabilityJob [repos/cloud/toolforge/jobs-api] (increase_test_coverage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (owner: 10raymond-ndibe) [08:25:29] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [08:25:31] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli [08:25:34] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [08:25:45] !log dcaro@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli [08:33:54] (03open) 10dcaro: readme: point to the wiki for deployment [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/170 [08:50:02] (03update) 10dcaro: readme: point to the wiki for deployment [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/170 [08:50:41] (03update) 10dcaro: readme: point to the wiki for deployment [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/170 [08:50:57] (03update) 10dcaro: readme: point to the wiki for deployment [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/170 [08:58:32] 10Tool-lexeme-forms, 06translatewiki.net, 10LPL Projects (Ongoing maintenance), 10LPL sprints (S1 2026 July): l10n-bot cannot create GitLab merge requests due to gitlab-ssh migration - https://phabricator.wikimedia.org/T432838#12177929 (10GGalofre-WMF) [09:01:44] (03approved) 10tlepage: readme: point to the wiki for deployment [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/170 (owner: 10dcaro) [09:10:35] 10Cloud-VPS (Quota-requests), 10Catalyst (Luka Ijo Pimeja Jan): Quota increase request for project catalyst - https://phabricator.wikimedia.org/T432617#12177950 (10jnuche) We have put together a wiki page giving context to this request: https://www.mediawiki.org/wiki/Patch_demo/Code_server [09:13:57] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/6 [09:15:08] 10Cloud-VPS (Debian Bullseye Deprecation), 10wikimaps-warper: Migrate wikimaps warper from Debian Bullseye to Trixie - https://phabricator.wikimedia.org/T431805#12177959 (10Chippyy) 05In progress→03Resolved maps-warper4 is now deleted. Closing this ticket [09:18:16] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-cli] support publishing continuous job to the internet - https://phabricator.wikimedia.org/T423410#12177970 (10dcaro) 05Resolved→03In progress [09:18:27] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-cli] support publishing continuous job to the internet - https://phabricator.wikimedia.org/T423410#12177973 (10dcaro) Not added the docs [09:19:12] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-api] support exposing continuous jobs to the internet - https://phabricator.wikimedia.org/T423408#12177977 (10dcaro) 05Resolved→03In progress [09:19:25] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-api] support exposing continuous jobs to the internet - https://phabricator.wikimedia.org/T423408#12177980 (10dcaro) Forgot to add the changelog [09:21:58] 06tools-platform-team: [jobs-cli,envvars-cli,components-cli,webservice-cli,toolforge-cli] Update deployment docs to point to wiki - https://phabricator.wikimedia.org/T433821 (10dcaro) 03NEW [09:22:13] (03update) 10dcaro: readme: point to the wiki for deployment [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/170 [09:26:58] (03update) 10dcaro: readme: point to the wiki for deployment [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/170 [09:36:00] (03open) 10dcaro: rename module [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/171 [09:36:34] (03update) 10dcaro: rename module [repos/cloud/toolforge/jobs-cli] (update_deployment_docs) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/171 [09:36:47] (03update) 10dcaro: global: rename tjf->jobs_cli [repos/cloud/toolforge/jobs-cli] (update_deployment_docs) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/171 [09:39:38] (03update) 10dcaro: readme: point to the wiki for deployment [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/170 [09:39:42] (03merge) 10dcaro: readme: point to the wiki for deployment [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/170 [09:39:45] (03update) 10dcaro: global: rename tjf->jobs_cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/171 [09:44:49] (03update) 10dcaro: [T432832] replace class mocks with magicmocks [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/361 (https://phabricator.wikimedia.org/T432832) (owner: 10mahveotm) [09:47:05] (03update) 10dcaro: global: rename tjf->jobs_cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/171 [10:02:28] (03update) 10dcaro: global: rename tjf->jobs_cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/171 [10:08:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:12:34] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/169 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:13:51] (03update) 10dcaro: global: rename tjf->jobs_cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/171 [10:22:41] (03update) 10dcaro: tests: improve test coverage [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/362 (owner: 10raymond-ndibe) [10:24:43] 10Cloud-VPS (Project-requests): Request creation of PanoCommons VPS project - https://phabricator.wikimedia.org/T433055#12178194 (10dcaro) a:03dcaro [10:24:53] 10Cloud-VPS (Project-requests): Request creation of PanoCommons VPS project - https://phabricator.wikimedia.org/T433055#12178195 (10dcaro) [10:25:07] 10Cloud-VPS (Project-requests), 06tools-platform-team: Request creation of PanoCommons VPS project - https://phabricator.wikimedia.org/T433055#12178198 (10dcaro) [10:26:49] !log dcaro@cloudcumin1001 panocommons START - Cookbook wmcs.vps.create_project for project panocommons in eqiad1 (T433055) [10:26:51] dcaro@cloudcumin1001: Unknown project "panocommons" [10:26:52] T433055: Request creation of PanoCommons VPS project - https://phabricator.wikimedia.org/T433055 [10:27:32] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project panocommons [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/341 (https://phabricator.wikimedia.org/T433055) [10:30:48] dcaro@cloudcumin1001 create_project (PID 1792007) is awaiting input [10:37:37] (03approved) 10dcaro: projects: added project panocommons [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/341 (https://phabricator.wikimedia.org/T433055) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [10:37:42] (03merge) 10dcaro: projects: added project panocommons [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/341 (https://phabricator.wikimedia.org/T433055) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [10:38:51] !log dcaro@cloudcumin1001 panocommons END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project panocommons in eqiad1 (T433055) [10:38:52] dcaro@cloudcumin1001: Unknown project "panocommons" [10:38:53] T433055: Request creation of PanoCommons VPS project - https://phabricator.wikimedia.org/T433055 [10:39:57] !log dcaro@cloudcumin1001 panocommons START - Cookbook wmcs.openstack.quota_increase by 800 gigabytes [10:39:58] dcaro@cloudcumin1001: Unknown project "panocommons" [10:40:05] !log dcaro@cloudcumin1001 panocommons END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 800 gigabytes [10:40:05] dcaro@cloudcumin1001: Unknown project "panocommons" [10:43:18] 10Cloud-VPS (Project-requests), 06tools-platform-team, 13Patch-For-Review: Request creation of PanoCommons VPS project - https://phabricator.wikimedia.org/T433055#12178232 (10dcaro) 05Open→03Resolved Done! If you have not yet subscribed, please subscribe to [[https://lists.wikimedia.org/postorius/li... [10:51:21] 10Toolforge, 06tools-platform-team: [jobs-cli,envvars-cli,components-cli,webservice-cli,toolforge-cli] Update deployment docs to point to wiki - https://phabricator.wikimedia.org/T433821#12178269 (10taavi) [11:01:18] 10Tool-lexeme-forms, 06translatewiki.net, 10LPL Projects (Ongoing maintenance), 10LPL sprints (S2 2026 August): l10n-bot cannot create GitLab merge requests due to gitlab-ssh migration - https://phabricator.wikimedia.org/T432838#12178306 (10GGalofre-WMF) [11:38:42] 10Quarry, 06tools-platform-team: Allow comments on queries - https://phabricator.wikimedia.org/T71543#12178398 (10CoderAnimeshPathak) [11:40:43] 10Toolforge, 06tools-infrastructure-team: [infra] Upgrade Toolforge K8s etcd nodes to Trixie - https://phabricator.wikimedia.org/T432634#12178400 (10taavi) [11:42:38] 10Toolforge, 06tools-infrastructure-team, 05Goal: Toolforge: migrate to Debian Trixie or later - https://phabricator.wikimedia.org/T433828 (10taavi) 03NEW [11:42:51] 10Toolforge, 06tools-infrastructure-team: [infra] Upgrade Toolforge K8s etcd nodes to Trixie - https://phabricator.wikimedia.org/T432634#12178425 (10taavi) [11:42:53] 10Toolforge, 06tools-infrastructure-team, 05Goal: Toolforge: migrate to Debian Trixie or later - https://phabricator.wikimedia.org/T433828#12178426 (10taavi) [11:42:54] 10Cloud-VPS (Debian Bullseye Deprecation), 10Toolforge, 06tools-infrastructure-team: [infra] Toolforge: migrate to Debian Bookworm or later - https://phabricator.wikimedia.org/T387005#12178427 (10taavi) [11:43:13] 10Cloud-VPS (Debian Bullseye Deprecation), 10Toolforge, 06tools-infrastructure-team: [infra] Toolforge: migrate to Debian Bookworm or later - https://phabricator.wikimedia.org/T387005#12178429 (10taavi) [12:17:42] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/169 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:19:35] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/360 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:19:39] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/360 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:23:22] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.555-20260803121952-d370717b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1354 [12:23:25] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.555-20260803121952-d370717b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1354 [12:33:50] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/169 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:33:59] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] (rename_module) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/169 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:49:08] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: Upgrade IDP server in cloudinfra project off of Bullseye - https://phabricator.wikimedia.org/T402006#12178616 (10SLyngshede-WMF) I've built a new package, it's not yet in apt, but running on idp-test. If test checks out I'll update product... [13:03:24] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: Upgrade IDP server in cloudinfra project off of Bullseye - https://phabricator.wikimedia.org/T402006#12178658 (10taavi) Thanks. If the build is available for me somewhere I can also test it now on the new Trixie WMCS host. [13:11:04] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: Upgrade IDP server in cloudinfra project off of Bullseye - https://phabricator.wikimedia.org/T402006#12178693 (10SLyngshede-WMF) Sure, you can grab the .deb package from idp-test2005.wikimedia.org in /srv/cas-build/ [13:14:06] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: Upgrade IDP server in cloudinfra project off of Bullseye - https://phabricator.wikimedia.org/T402006#12178722 (10SLyngshede-WMF) Just to document on task: Previously the CAS overlay would load a series of module via a "separate" method: h... [13:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:23:09] (03update) 10raymond-ndibe: tests: improve test coverage [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/362 [13:28:03] 10Cloud-VPS, 06tools-infrastructure-team: Rebalance WMCS instance backups between the two hosts - https://phabricator.wikimedia.org/T430018#12178763 (10Andrew) On the last run: cloudbackup1003 started: Aug 02 06:39:01 completed: Aug 03 06:05:10 total: 23:34 cloudbackup1004 started: Aug 02 17:00:07 completed... [13:30:44] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/359 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:31:56] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/359 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:40:18] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:40:22] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team, 13Patch-For-Review: Upgrade IDP server in cloudinfra project off of Bullseye - https://phabricator.wikimedia.org/T402006#12178796 (10taavi) Thanks! That worked well, just needed one more explicit crypto key length setting in the patch ab... [13:42:38] (03update) 10mahveotm: [T432832] replace class mocks with magicmocks [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/361 (https://phabricator.wikimedia.org/T432832) [13:43:49] (03PS1) 10Slyngshede: P:tofurkey: Tofurkey dummy secret [labs/private] - 10https://gerrit.wikimedia.org/r/1320164 (https://phabricator.wikimedia.org/T355446) [13:44:49] (03CR) 10Slyngshede: [V:03+2 C:03+2] P:tofurkey: Tofurkey dummy secret [labs/private] - 10https://gerrit.wikimedia.org/r/1320164 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [13:48:05] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12178808 (10Andrew) Today there are 28 OSDs showing the slow ops warning. Of the 16 OSDs that got the mitigation above (bd... [13:52:49] (03update) 10dcaro: models: support webservice job-type [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) (owner: 10raymond-ndibe) [13:53:12] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [13:55:43] 10Cloud-VPS, 06tools-infrastructure-team: cloudvps-quota divides by zero when requesting quota increase from 0 - https://phabricator.wikimedia.org/T433836 (10taavi) 03NEW [13:56:46] 10Cloud-VPS, 06tools-infrastructure-team: cloudvps-quota should handle /api/request-quota-increase failures gracefully - https://phabricator.wikimedia.org/T433837 (10taavi) 03NEW [13:58:11] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:58:51] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Network unavailable on a few VMs - https://phabricator.wikimedia.org/T432426#12178855 (10Andrew) This morning there are no new Trixie hosts down. Three new bookworm hosts show as down -- two of them have fallen off the network and one of them is f... [14:04:09] 10Cloud-VPS (Quota-requests), 06Wikibase Reuse Team, 10Wikidata: Quota increase request for project pm20database - https://phabricator.wikimedia.org/T433839 (10bking) 03NEW [14:08:03] FIRING: InstanceDown: Project cloudinfra instance cloudinfra-idp-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:11:41] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.2 Toolforge Alerting System - https://phabricator.wikimedia.org/T432860#12178917 (10CCiufo-WMF) >>! In T432860#12174092, @aputhin wrote: > Agree that Toolforge UI will be the go-to place for checking alert history. The most important part of the... [14:13:03] RESOLVED: InstanceDown: Project cloudinfra instance cloudinfra-idp-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:14:11] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [14:20:35] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-harbor-1 [14:21:23] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-harbor-1 [14:24:40] (03approved) 10dcaro: jobs-api: bump to 0.0.555-20260803121952-d370717b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1354 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:24:47] (03merge) 10dcaro: jobs-api: bump to 0.0.555-20260803121952-d370717b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1354 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:27:02] (03approved) 10dcaro: compile: use PWD instead of hardcoded /app for locale paths [repos/cloud/toolforge/buildpacks/locale-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/locale-buildpack/-/merge_requests/4 (https://phabricator.wikimedia.org/T428230) (owner: 10raymond-ndibe) [14:27:37] (03update) 10dcaro: compile: use PWD instead of hardcoded /app for locale paths [repos/cloud/toolforge/buildpacks/locale-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/locale-buildpack/-/merge_requests/4 (https://phabricator.wikimedia.org/T428230) (owner: 10raymond-ndibe) [14:28:41] (03update) 10dcaro: cli: fix BrokenPipeError when tailing logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/166 [14:30:43] (03update) 10raymond-ndibe: cli: fix BrokenPipeError when tailing logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/166 (owner: 10dcaro) [14:30:45] (03approved) 10raymond-ndibe: cli: fix BrokenPipeError when tailing logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/166 (owner: 10dcaro) [14:32:28] (03update) 10raymond-ndibe: global: rename tjf->jobs_cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/171 (owner: 10dcaro) [14:32:31] (03update) 10raymond-ndibe: global: rename tjf->jobs_cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/171 (owner: 10dcaro) [14:32:33] (03approved) 10raymond-ndibe: global: rename tjf->jobs_cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/171 (owner: 10dcaro) [14:36:45] (03PS1) 10Slyngshede: Tofurkey: switch to a secrets file [labs/private] - 10https://gerrit.wikimedia.org/r/1320172 [14:42:20] (03close) 10raymond-ndibe: testing the locale fix [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/92 [14:46:22] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/30 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:50:03] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T431374) [14:50:04] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=97) (T431374) [14:50:09] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T431682) [14:50:10] T431374: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374 [14:50:11] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T431682) [14:50:20] T431682: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682 [14:51:27] (03merge) 10dcaro: global: rename tjf->jobs_cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/171 [14:51:29] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/169 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:51:54] (03update) 10dcaro: cli: fix BrokenPipeError when tailing logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/166 [14:53:17] (03merge) 10dcaro: cli: fix BrokenPipeError when tailing logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/166 [15:00:55] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12179213 (10Andrew) I put 1048 back in service and it seems to be working fine. [15:08:16] 10Toolforge, 06tools-platform-team: [jobs-cli,envvars-cli,components-cli,webservice-cli,toolforge-cli] Update deployment docs to point to wiki - https://phabricator.wikimedia.org/T433821#12179240 (10dcaro) [15:08:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:11:49] 10Toolforge, 06tools-platform-team: [lima-kilo] shows error when copying home dirs - https://phabricator.wikimedia.org/T433849 (10dcaro) 03NEW [15:14:14] !log andrew@cloudcumin1001 cloudinfra-nfs START - Cookbook wmcs.nfs.migrate_service (T433426) [15:14:16] T433426: Replace scratch-1.cloudinfra-nfs with a trixie VM - https://phabricator.wikimedia.org/T433426 [15:15:02] (03update) 10renovatebot: Update pnpm to v11.19.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/8 [15:15:07] (03open) 10renovatebot: Update pnpm to v11.19.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/8 [15:15:21] (03update) 10renovatebot: Update pnpm to v11.19.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/21 [15:15:24] (03open) 10renovatebot: Update pnpm to v11.19.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/21 [15:15:39] (03update) 10renovatebot: Update pnpm to v11.19.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/15 [15:15:46] (03open) 10renovatebot: Update pnpm to v11.19.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/15 [15:16:47] (03update) 10renovatebot: Update pnpm to v11.19.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/47 [15:16:51] (03open) 10renovatebot: Update pnpm to v11.19.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/47 [15:17:52] (03update) 10renovatebot: Update pnpm to v11.19.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/30 [15:17:58] (03open) 10renovatebot: Update pnpm to v11.19.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/30 [15:18:34] !log andrew@cloudcumin1001 cloudinfra-nfs END (PASS) - Cookbook wmcs.nfs.migrate_service (exit_code=0) (T433426) [15:20:02] (03approved) 10tlepage: [T432832] replace class mocks with magicmocks [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/361 (https://phabricator.wikimedia.org/T432832) (owner: 10mahveotm) [15:34:35] (03update) 10tlepage: [T432832] replace class mocks with magicmocks [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/361 (https://phabricator.wikimedia.org/T432832) (owner: 10mahveotm) [15:39:50] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers (T433426) [15:39:55] T433426: Replace scratch-1.cloudinfra-nfs with a trixie VM - https://phabricator.wikimedia.org/T433426 [15:40:03] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [15:41:03] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [15:49:47] (03update) 10mahveotm: [T432832] replace class mocks with magicmocks [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/361 (https://phabricator.wikimedia.org/T432832) [15:57:49] FIRING: PuppetDisabled: Puppet disabled on cloudcephosd1016:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [16:05:03] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-test-k8s-worker-nfs-9 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:05:03] FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance tools-k8s-worker-nfs-35 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:07:41] 10Toolforge, 06tools-platform-team: [jobs-cli,envvars-cli,components-cli,webservice-cli,toolforge-cli] Update deployment docs to point to wiki - https://phabricator.wikimedia.org/T433821#12179637 (10aputhin) p:05Triage→03Low [16:15:03] FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-test-k8s-worker-nfs-10 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:19:23] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12179715 (10Doc_James) When you look at a webm you see the ability to pull up the videocuttool {F97030996} When you look at a still image you can see the Cropt... [16:20:03] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance tools-bastion-15 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:25:03] RESOLVED: [7x] PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:29:33] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [16:30:03] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance tools-bastion-15 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:31:31] 10PAWS, 06tools-platform-team: [paws] automate the current account locking scripts as a first step - https://phabricator.wikimedia.org/T433572#12179822 (10dcaro) 05Open→03Resolved [16:31:33] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [16:41:08] 10Toolforge, 06tools-platform-team: [jobs-api] Allow storing one-off jobs - https://phabricator.wikimedia.org/T433868 (10dcaro) 03NEW [16:41:39] 10Toolforge, 06tools-platform-team: [logs-cli] Add `--direction` flag to choose the direction of the logs shown - https://phabricator.wikimedia.org/T433869 (10dcaro) 03NEW [16:43:43] 10Toolforge, 06tools-platform-team: [logs-cli] add a `--first` flag, or replace `--latest` with `--lines` - https://phabricator.wikimedia.org/T433870 (10dcaro) 03NEW [16:47:10] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/19 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:49:05] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/19 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:49:10] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/19 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:49:26] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/19 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:52:13] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T431374) [16:52:16] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T431374) [16:52:20] T431374: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374 [16:52:21] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/86 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:52:23] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/86 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:52:58] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/178 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:58:27] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12180053 (10Ottomata) a:05Ottomata→03tchin [17:05:57] !log andrew@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers (T433426) [17:06:02] T433426: Replace scratch-1.cloudinfra-nfs with a trixie VM - https://phabricator.wikimedia.org/T433426 [17:07:19] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 (T433426) [17:07:53] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 (T433426) [17:09:47] (03close) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/178 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:11:42] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12180113 (10VRiley-WMF) @Andrew Awesome, sounds good. Would we like to start on 1049? [17:11:56] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12180114 (10VRiley-WMF) [17:13:04] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/101 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:13:09] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/101 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:13:35] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-66 (T433426) [17:13:39] T433426: Replace scratch-1.cloudinfra-nfs with a trixie VM - https://phabricator.wikimedia.org/T433426 [17:13:39] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-66 (T433426) [17:13:46] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-66 (T433426) [17:14:56] 10Toolforge, 06tools-platform-team: [all] first of the month dependency upgrades - https://phabricator.wikimedia.org/T433878 (10dcaro) 03NEW [17:15:05] 10Toolforge, 06tools-platform-team: [all] first of the month dependency upgrades - https://phabricator.wikimedia.org/T433878#12180183 (10dcaro) 05Open→03In progress [17:15:54] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/68 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:20:27] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/334 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:23:19] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/68 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:24:02] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/68 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:24:06] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-66 (T433426) [17:24:14] T433426: Replace scratch-1.cloudinfra-nfs with a trixie VM - https://phabricator.wikimedia.org/T433426 [17:25:25] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-44, tools-k8s-worker-nfs [17:25:25] -43, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs [17:25:25] -21, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-9, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-1 (T433426) [17:26:50] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/68 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:32:43] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/56 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [18:45:29] (03PS14) 10MusikAnimal: Readd third-party suppport [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [18:49:27] (03CR) 10MusikAnimal: [C:03+2] Readd third-party suppport (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [18:50:50] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate WMCS-managed NFS servers off of Bullseye - https://phabricator.wikimedia.org/T401812#12180380 (10Andrew) [18:51:02] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate WMCS-managed NFS servers off of Bullseye - https://phabricator.wikimedia.org/T401812#12180385 (10Andrew) 05Open→03Resolved a:03Andrew Closing until we find the ones we missed [18:52:27] (03Merged) 10jenkins-bot: Readd third-party suppport [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [18:53:37] (03PS6) 10Jsn.sherman: Repository: config-driven dblist assembly [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309801 (https://phabricator.wikimedia.org/T344090) [19:08:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:27:35] 10Tool-wikinewsie: Wikinewsie uses low-resolution thumbnails when higher-resolution exists - https://phabricator.wikimedia.org/T433758#12180465 (10Pharos) Wikinewsie uses two modes of images - those drawn automatically from the lede section of the article, and if that's missing, there is an option of user-genera... [19:29:32] (03update) 10raymond-ndibe: compile: use PWD instead of hardcoded /app for locale paths [repos/cloud/toolforge/buildpacks/locale-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/locale-buildpack/-/merge_requests/4 (https://phabricator.wikimedia.org/T428230) [19:30:11] (03update) 10raymond-ndibe: compile: use PWD instead of hardcoded /app for locale paths [repos/cloud/toolforge/buildpacks/locale-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/locale-buildpack/-/merge_requests/4 (https://phabricator.wikimedia.org/T428230) [19:30:27] (03approved) 10raymond-ndibe: compile: use PWD instead of hardcoded /app for locale paths [repos/cloud/toolforge/buildpacks/locale-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/locale-buildpack/-/merge_requests/4 (https://phabricator.wikimedia.org/T428230) [19:30:36] (03update) 10raymond-ndibe: compile: use PWD instead of hardcoded /app for locale paths [repos/cloud/toolforge/buildpacks/locale-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/locale-buildpack/-/merge_requests/4 (https://phabricator.wikimedia.org/T428230) [19:31:08] (03merge) 10raymond-ndibe: compile: use PWD instead of hardcoded /app for locale paths [repos/cloud/toolforge/buildpacks/locale-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/locale-buildpack/-/merge_requests/4 (https://phabricator.wikimedia.org/T428230) [19:39:56] (03open) 10andrew: cert-manager: update deploy to 1.20.3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1355 (https://phabricator.wikimedia.org/T408786) [19:40:10] (03update) 10andrew: cert-manager: update deploy to 1.20.3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1355 (https://phabricator.wikimedia.org/T408786) [19:40:31] 10VPS-project-Wikistats: Add Coasterpedia to Mediawikis - https://phabricator.wikimedia.org/T433765#12180510 (10Dzahn) ` MariaDB [wikistats]> insert into mediawikis (statsurl, method) values ("https://coasterpedia.net/w/api.php",8); .. dzahn@wikistats-bookworm:~$ /usr/bin/php /usr/lib/wikistats/update.php mw... [19:53:27] 10VPS-project-Wikistats: Add Coasterpedia to Mediawikis - https://phabricator.wikimedia.org/T433765#12180558 (10Dzahn) ` dzahn@wikistats-bookworm:~$ /usr/bin/php /usr/lib/wikistats/update.php mw extinfo new ` [19:54:44] 10VPS-project-Wikistats: Add Coasterpedia to Mediawikis - https://phabricator.wikimedia.org/T433765#12180566 (10Dzahn) 05Open→03Resolved [20:17:03] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-9 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:22:03] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-9 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:36:03] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-44, t [20:36:03] ools-k8s-worker-nfs-43, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-22, t [20:36:03] ools-k8s-worker-nfs-21, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-9, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-1 (T433426) [20:36:06] T433426: Replace scratch-1.cloudinfra-nfs with a trixie VM - https://phabricator.wikimedia.org/T433426 [20:55:15] (03CR) 10MusikAnimal: [C:03+2] Repository: config-driven dblist assembly [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309801 (https://phabricator.wikimedia.org/T344090) (owner: 10Jsn.sherman) [20:56:23] (03Merged) 10jenkins-bot: Repository: config-driven dblist assembly [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309801 (https://phabricator.wikimedia.org/T344090) (owner: 10Jsn.sherman) [20:56:33] (03PS9) 10Jsn.sherman: Repository: cover isWMF branches [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309830 (https://phabricator.wikimedia.org/T344090) [21:01:27] 10Tool-wikinewsie: WikiPortraits mode for Wikinewsie - https://phabricator.wikimedia.org/T432573#12180877 (10SuperHamster) > there should be an option to mark it as such, and link to the relevant Wikimedia Commons category for more WikiPortraits coverage of the event On top of our Commons categories, I'll share... [21:11:32] (03CR) 10MusikAnimal: [C:03+2] Repository: cover isWMF branches (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309830 (https://phabricator.wikimedia.org/T344090) (owner: 10Jsn.sherman) [21:13:56] (03Merged) 10jenkins-bot: Repository: cover isWMF branches [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309830 (https://phabricator.wikimedia.org/T344090) (owner: 10Jsn.sherman) [21:14:06] (03PS5) 10Jsn.sherman: EditCounterRepository: unit test coverage [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312259 (https://phabricator.wikimedia.org/T384711) [21:16:27] (03update) 10renovatebot: Update dependency brace-expansion to v5.0.9 [SECURITY] [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/31 [21:16:28] (03close) 10renovatebot: Update dependency brace-expansion to v5.0.9 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/28 [21:16:29] (03update) 10renovatebot: Update dependency brace-expansion to v5.0.9 - autoclosed [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/28 [21:16:33] (03open) 10renovatebot: Update dependency brace-expansion to v5.0.9 [SECURITY] [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/31 [21:18:05] (03CR) 10MusikAnimal: [C:03+2] EditCounterRepository: unit test coverage [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312259 (https://phabricator.wikimedia.org/T384711) (owner: 10Jsn.sherman) [21:18:58] (03Merged) 10jenkins-bot: EditCounterRepository: unit test coverage [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312259 (https://phabricator.wikimedia.org/T384711) (owner: 10Jsn.sherman) [21:19:35] (03PS5) 10Jsn.sherman: Repository: unbound params in untested branches [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312267 (https://phabricator.wikimedia.org/T384711) [21:27:11] (03CR) 10MusikAnimal: [C:03+2] Repository: unbound params in untested branches [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312267 (https://phabricator.wikimedia.org/T384711) (owner: 10Jsn.sherman) [21:30:05] (03Merged) 10jenkins-bot: Repository: unbound params in untested branches [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312267 (https://phabricator.wikimedia.org/T384711) (owner: 10Jsn.sherman) [21:36:48] (03PS5) 10Jsn.sherman: Repository: unit test coverage [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312268 (https://phabricator.wikimedia.org/T384711) [21:43:52] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 10Ceph, and 3 others: cloudcephosd1044 boot issues - https://phabricator.wikimedia.org/T429267#12181024 (10VRiley-WMF) a:03VRiley-WMF [22:12:23] 10Cloud-VPS (Debian Bullseye Deprecation), 10Internet In A Box: Iiab vps debian migration blocked on exposing rsync - https://phabricator.wikimedia.org/T433714#12181089 (10Bawolff) >>! In T433714#12176121, @Tim-moody wrote: > @Bawolff I added you as a member of iiab. Thanks for your help. Let's try the suggest... [22:18:03] (03CR) 10MusikAnimal: [C:03+2] Repository: unit test coverage [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312268 (https://phabricator.wikimedia.org/T384711) (owner: 10Jsn.sherman) [22:19:17] (03Merged) 10jenkins-bot: Repository: unit test coverage [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312268 (https://phabricator.wikimedia.org/T384711) (owner: 10Jsn.sherman) [22:20:46] (03CR) 10MusikAnimal: [C:03+2] I18nHelper: guard dates against bad ICU locales [labs/xtools] - 10https://gerrit.wikimedia.org/r/1314218 (https://phabricator.wikimedia.org/T384711) (owner: 10Jsn.sherman) [22:22:31] (03Merged) 10jenkins-bot: I18nHelper: guard dates against bad ICU locales [labs/xtools] - 10https://gerrit.wikimedia.org/r/1314218 (https://phabricator.wikimedia.org/T384711) (owner: 10Jsn.sherman) [22:24:08] 10Tool-wikimedia-attribution, 06App Growth, 06MediaWiki-API-Platform-Team, 06WikipediaApp, 07OKR-Work: Make mobile app store badges more visible on Wikipedia.org portal page - https://phabricator.wikimedia.org/T432758#12181173 (10HCoplin-WMF) [22:27:14] (03PS3) 10MusikAnimal: tests: skip tests with app.is_wmf instead of returning early [labs/xtools] - 10https://gerrit.wikimedia.org/r/1310607 [22:27:59] (03CR) 10CI reject: [V:04-1] tests: skip tests with app.is_wmf instead of returning early [labs/xtools] - 10https://gerrit.wikimedia.org/r/1310607 (owner: 10MusikAnimal) [22:34:12] (03PS4) 10MusikAnimal: tests: skip tests with app.is_wmf instead of returning early [labs/xtools] - 10https://gerrit.wikimedia.org/r/1310607 [22:35:29] (03CR) 10CI reject: [V:04-1] tests: skip tests with app.is_wmf instead of returning early [labs/xtools] - 10https://gerrit.wikimedia.org/r/1310607 (owner: 10MusikAnimal) [23:08:56] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:11:29] (03PS5) 10MusikAnimal: tests: skip tests with app.is_wmf instead of returning early [labs/xtools] - 10https://gerrit.wikimedia.org/r/1310607 [23:12:09] (03PS6) 10MusikAnimal: tests: skip tests with app.is_wmf instead of returning early [labs/xtools] - 10https://gerrit.wikimedia.org/r/1310607 [23:12:54] (03CR) 10CI reject: [V:04-1] tests: skip tests with app.is_wmf instead of returning early [labs/xtools] - 10https://gerrit.wikimedia.org/r/1310607 (owner: 10MusikAnimal)