[00:00:08] (03CR) 10MusikAnimal: Add proofreadpage quality support (036 comments) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) (owner: 10Alien4444) [00:10:23] (03PS8) 10Alien4444: Add proofreadpage quality support [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) [00:11:57] (03CR) 10Alien4444: Add proofreadpage quality support (038 comments) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) (owner: 10Alien4444) [00:40:01] (03CR) 10Novem Linguae: AdminStats: +footnote on (re)protect and (re)block (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1275236 (https://phabricator.wikimedia.org/T354024) (owner: 10Alien4444) [01:31:28] 10Toolforge, 06tools-platform-team: jobs-cli: stop resolving dynamic fields in the cli - https://phabricator.wikimedia.org/T423828 (10Raymond_Ndibe) 03NEW [01:34:50] (03open) 10raymond-ndibe: cli.py: stop resolving dynamic fields in the cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/151 (https://phabricator.wikimedia.org/T423828) [02:06:37] (03PS2) 10Alien4444: AdminStats: +footnote on (re)protect and (re)block [labs/xtools] - 10https://gerrit.wikimedia.org/r/1275236 (https://phabricator.wikimedia.org/T354024) [02:26:32] (03update) 10raymond-ndibe: cli.py: stop resolving dynamic fields in the cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/151 (https://phabricator.wikimedia.org/T423828) [02:33:32] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [02:33:43] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [02:38:32] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [02:38:43] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [03:30:00] (03open) 10bd808: Draw the rest of the owl [toolforge-repos/etherpad-backup] - 10https://gitlab.wikimedia.org/toolforge-repos/etherpad-backup/-/merge_requests/1 (https://phabricator.wikimedia.org/T422958 https://phabricator.wikimedia.org/T423014) [03:48:41] (03update) 10bd808: Draw the rest of the owl [toolforge-repos/etherpad-backup] - 10https://gitlab.wikimedia.org/toolforge-repos/etherpad-backup/-/merge_requests/1 (https://phabricator.wikimedia.org/T422958 https://phabricator.wikimedia.org/T423014) [03:53:38] 10Tool-etherpad-backup, 13Patch-For-Review: Setup proof of concept storage and retrieval from WMCS object storage - https://phabricator.wikimedia.org/T422958#11837443 (10bd808) All of the files in the tool's $HOME/public_html/p as of 2026-04-19T22:41Z have been uploaded to the storage bucket. I did this the so... [04:04:16] 10Toolforge, 06tools-platform-team: [Toolforge]: Automate daily count of successful SSH logins as a measure of user activity - https://phabricator.wikimedia.org/T423549#11837446 (10komla) It seems the rotated auth logs are only kept for 7 days so we will limit our activity window to 7 days for now. [06:50:21] (03PS2) 10Lokal Profil: use echo_time when available [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1274022 [07:28:17] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Migrate our use of osbpo away from mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423598#11837669 (10fgiunchedi) I think we should be considering importing osbpo.debian.net apt repo as an upstream into `aptrepo` puppet module (i.e. `mo... [08:46:15] 10Cloud-VPS (Quota-requests): Temporary instance increase for Bullseye servers deprecation - https://phabricator.wikimedia.org/T416803#11837963 (10fgiunchedi) Thank you for letting us know @YochayCO, appreciate it! {{done}}, I have decreased the quota with `wmcs-openstack` since we don't have a quota_decrea... [08:49:55] 10Tools: Store and optionally show the full firehose in csp-report - https://phabricator.wikimedia.org/T423847 (10fgiunchedi) 03NEW [08:50:04] 10Tools: Store and optionally show the full firehose in csp-report - https://phabricator.wikimedia.org/T423847#11837988 (10fgiunchedi) p:05Triage→03Low [08:50:46] 10Tools: CSP violations with known domains in the blocked-uri are not collected by csp-report - https://phabricator.wikimedia.org/T422916#11838001 (10fgiunchedi) >>! In T422916#11830798, @bd808 wrote: >>>! In T422916#11828441, @fgiunchedi wrote: >> That's fair re: user confusion concerns. From my SRE POV I w... [08:50:49] 10Tools: CSP violations with known domains in the blocked-uri are not collected by csp-report - https://phabricator.wikimedia.org/T422916#11838004 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi [08:51:22] 10Tools: Store and optionally show the full firehose in csp-report - https://phabricator.wikimedia.org/T423847#11838008 (10fgiunchedi) [09:18:51] (03update) 10fnegri: refactor to use job_type argument that is already available in jobs-api [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T423412) (owner: 10raymond-ndibe) [09:18:52] (03approved) 10fnegri: refactor to use job_type argument that is already available in jobs-api [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T423412) (owner: 10raymond-ndibe) [09:40:36] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: memcache is a SPOF for designate/tooz coordination - https://phabricator.wikimedia.org/T422646#11838198 (10fgiunchedi) I took a look at the codfw set up and LGTM, with only one thing to change: the tooz backend url should list all zk servers, so we can sa... [09:57:25] (03approved) 10fnegri: helpers:toolforge_deploy: check that .git exists before deployment [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/319 (https://phabricator.wikimedia.org/T423731) (owner: 10raymond-ndibe) [10:23:17] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:28:17] RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:36:40] 06cloud-services-team, 10Cloud-VPS: [tofuinfratest] Magnum cluster creation is failing - https://phabricator.wikimedia.org/T423393#11838508 (10fnegri) 05Open→03Resolved a:03fnegri This worked fine for the past 4 runs, marking as Resolved. `lang=shell-session root@tf-bastion:~/tf-infra-test# journalc... [12:14:43] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Migrate our use of osbpo away from mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423598#11838641 (10MoritzMuehlenhoff) >>! In T423598#11837669, @fgiunchedi wrote: > I think we should be considering importing osbpo.debian.net apt repo... [12:17:29] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07OKR-Work: Add linting rules for operations and paths - https://phabricator.wikimedia.org/T422504#11838663 (10KBach) [12:22:44] 10Tool-etherpad-backup, 13Patch-For-Review: display a list of backed up etherpads somewhere - https://phabricator.wikimedia.org/T423014#11838682 (10Tkarcher) Thanks for improving the handling of the external ressources. Regarding the size of the json list: This was a quick solution without optimization in... [12:29:54] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/69 [12:31:30] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/1275396 (owner: 10L10n-bot) [12:45:32] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [12:50:32] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [12:55:00] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/69 (owner: 10l10n-bot) [12:55:04] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/69 (owner: 10l10n-bot) [13:04:55] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Migrate our use of osbpo away from mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423598#11838813 (10Andrew) > I think we should be considering importing osbpo.debian.net apt repo as an upstream into aptrepo puppet module (i.e. modules... [13:06:05] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Migrate our use of osbpo away from mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423598#11838816 (10fgiunchedi) >>! In T423598#11838641, @MoritzMuehlenhoff wrote: >>>! In T423598#11837669, @fgiunchedi wrote: >> I think we should be co... [13:21:21] (03update) 10raymond-ndibe: cli.py: stop resolving dynamic fields in the cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/151 (https://phabricator.wikimedia.org/T423828) [13:27:11] 06tools-platform-team: jobs-api: set enable_storage to true in values.yaml - https://phabricator.wikimedia.org/T423870 (10Raymond_Ndibe) 03NEW [13:27:13] 06cloud-services-team, 10Toolforge: Buildservice for Rust fails - https://phabricator.wikimedia.org/T423675#11838947 (10fgiunchedi) Is archive.ubuntu.com working now? Was it the only host failing? [13:28:12] 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: [wikireplicas] Update grants for "maintainviews" user - https://phabricator.wikimedia.org/T422806#11838964 (10fnegri) 05In progress→03Resolved I have applied the new grants manually to all clouddb hosts, using the follo... [13:44:41] 10Toolforge, 06tools-platform-team: jobs-api: set enable_storage to true in values.yaml - https://phabricator.wikimedia.org/T423870#11839059 (10taavi) @Raymond_Ndibe please remember to add a project tag like #toolforge in addition to a team tag [14:04:30] (03update) 10taavi: kubernetes: Stop manipulating Ingress objects [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/100 (https://phabricator.wikimedia.org/T392356) [14:05:23] (03update) 10taavi: Undeploy ingress-nginx [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1222 (https://phabricator.wikimedia.org/T392356) [14:20:35] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Migrate our use of osbpo away from mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423598#11839249 (10fgiunchedi) >>! In T423598#11837669, @fgiunchedi wrote: > I think we should be considering importing osbpo.debian.net apt repo as an u... [14:23:26] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Migrate our use of osbpo away from mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423598#11839260 (10Andrew) Filippo says: https://wikitech.wikimedia.org/wiki/Reprepro#Adding_a_new_external_repository should have everything [14:27:54] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#11839293 (10fnegri) [14:34:32] 10Toolforge, 06tools-platform-team: jobs-api: fix dynamic default values resolution - https://phabricator.wikimedia.org/T423880 (10Raymond_Ndibe) 03NEW [14:52:00] (03open) 10raymond-ndibe: values.yaml: set enable_storage to true [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/289 (https://phabricator.wikimedia.org/T423870) [14:54:40] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: jobs-api: set enable_storage to true in values.yaml - https://phabricator.wikimedia.org/T423870#11839502 (10Raymond_Ndibe) >>! In T423870#11839059, @taavi wrote: > @Raymond_Ndibe please remember to add a project tag like #toolforge in addition to a team... [14:56:43] (03update) 10raymond-ndibe: refactor to use job_type argument that is already available in jobs-api [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T423412) [14:59:55] (03update) 10raymond-ndibe: helpers:toolforge_deploy: check that .git exists before deployment [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/319 (https://phabricator.wikimedia.org/T423731) [14:59:57] (03approved) 10raymond-ndibe: helpers:toolforge_deploy: check that .git exists before deployment [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/319 (https://phabricator.wikimedia.org/T423731) [15:00:04] (03merge) 10raymond-ndibe: helpers:toolforge_deploy: check that .git exists before deployment [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/319 (https://phabricator.wikimedia.org/T423731) [15:00:34] 10Tool-wikimedia-attribution: [Proposal] Make the Wikimedia Attribution Framework and API discoverable for AI agents - https://phabricator.wikimedia.org/T423885#11839534 (10Sarai-WMF) Hey @HCoplin-WMF and @RHo! I decided to document this recommendation to improve the discoverability of the Wikimedia Attribution... [15:01:45] (03update) 10raymond-ndibe: refactor to use job_type argument that is already available in jobs-api [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T423412) [15:01:47] (03approved) 10raymond-ndibe: refactor to use job_type argument that is already available in jobs-api [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T423412) [15:02:06] (03merge) 10raymond-ndibe: refactor to use job_type argument that is already available in jobs-api [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T423412) [15:02:13] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T388092) [15:08:36] (03open) 10raymond-ndibe: d/changelog: bump to 16.1.28 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/152 (https://phabricator.wikimedia.org/T423412) [15:11:38] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [15:15:14] (03open) 10raymond-ndibe: tests: fix wrong test data [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/290 (https://phabricator.wikimedia.org/T423880) [15:19:12] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [15:24:34] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [15:27:04] 10Toolforge, 06tools-platform-team: jobs-api: avoid using "runtime"/"storage" in jobs load response message - https://phabricator.wikimedia.org/T423891 (10Raymond_Ndibe) 03NEW [15:27:27] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: lima-kilo: remove git dir check for local deployment - https://phabricator.wikimedia.org/T423731#11839750 (10Raymond_Ndibe) 05Open→03Resolved [15:28:08] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: jobs-api: fix dynamic default values resolution - https://phabricator.wikimedia.org/T423880#11839752 (10Raymond_Ndibe) 05Open→03In progress [15:28:09] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: jobs-cli: stop resolving dynamic fields in the cli - https://phabricator.wikimedia.org/T423828#11839756 (10Raymond_Ndibe) 05Open→03In progress [15:28:13] 10Toolforge, 06tools-platform-team: jobs-api: avoid using "runtime"/"storage" in jobs load response message - https://phabricator.wikimedia.org/T423891#11839760 (10Raymond_Ndibe) 05Open→03In progress [15:32:11] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [15:36:02] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Migrate our use of osbpo away from mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423598#11839823 (10MoritzMuehlenhoff) >>! In T423598#11838816, @fgiunchedi wrote: >>>! In T423598#11838641, @MoritzMuehlenhoff wrote: >>>>! In T423598#11... [15:37:56] (03update) 10raymond-ndibe: core.py: skip one-off jobs when updating storage [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/288 (https://phabricator.wikimedia.org/T423544) [15:38:00] (03update) 10raymond-ndibe: core.py: skip one-off jobs when updating storage [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/288 (https://phabricator.wikimedia.org/T423544) [15:40:34] (03update) 10raymond-ndibe: d/changelog: bump to 16.1.28 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/152 (https://phabricator.wikimedia.org/T423412) [15:42:08] (03approved) 10raymond-ndibe: d/changelog: bump to 16.1.28 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/152 (https://phabricator.wikimedia.org/T423412) [15:43:36] (03merge) 10raymond-ndibe: d/changelog: bump to 16.1.28 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/152 (https://phabricator.wikimedia.org/T423412) [15:47:21] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-cli] refactor to use job_type argument that is already available in jobs-api - https://phabricator.wikimedia.org/T423412#11839905 (10Raymond_Ndibe) 05Open→03Resolved [16:12:58] 10Tool-etherpad-backup: Special characters are not handled consistently in the current backups, should be improved when migrating to S3 - https://phabricator.wikimedia.org/T423905 (10Tkarcher) 03NEW [16:13:33] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: jobs-api: set enable_storage to true in values.yaml - https://phabricator.wikimedia.org/T423870#11840143 (10aputhin) a:03Raymond_Ndibe [16:14:22] (03approved) 10fnegri: values.yaml: set enable_storage to true [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/289 (https://phabricator.wikimedia.org/T423870) (owner: 10raymond-ndibe) [16:25:41] 10Tool-gitlab-account-approval, 06Release-Engineering-Team, 10GitLab (Account Approval): GitLab Account Approval Bot is failing - https://phabricator.wikimedia.org/T423909 (10brennen) 03NEW [16:27:49] 06cloud-services-team, 10Toolforge: Buildservice for Rust fails - https://phabricator.wikimedia.org/T423675#11840231 (10magnusmanske) Only one I saw. build failed earlier today with "empty image" but I think that was something else. Didn't repeat. [16:29:52] 10Tool-gitlab-account-approval, 06Release-Engineering-Team, 10GitLab (Account Approval): GitLab Account Approval Bot is failing - https://phabricator.wikimedia.org/T423909#11840237 (10brennen) From logs: ` 2026-04-20T16:27:13Z glaab.unhandled CRITICAL: Uncaught exception Traceback (most recent call last):... [16:34:24] (03update) 10fnegri: values.yaml: set enable_storage to true [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/289 (https://phabricator.wikimedia.org/T423870) (owner: 10raymond-ndibe) [16:40:24] 10Tool-gitlab-account-approval, 06Release-Engineering-Team, 10GitLab (Account Approval): GitLab Account Approval Bot is failing - https://phabricator.wikimedia.org/T423909#11840355 (10brennen) Looks like this has been happening on (nearly?) every run since 2026-04-18 12:43 UTC. Running a dry-run per the `RE... [16:42:30] (03PS1) 10Akoopal: ignoring some more fields [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1275469 (https://phabricator.wikimedia.org/T55271) [17:06:10] (03Abandoned) 10Akoopal: ignoring some more fields [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1275469 (https://phabricator.wikimedia.org/T55271) (owner: 10Akoopal) [17:09:46] (03PS1) 10Akoopal: Fixing double destination for de-hh_de [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1275474 (https://phabricator.wikimedia.org/T55271) [17:16:11] (03open) 10bd808: Update runtime and patch a small bug [toolforge-repos/gitlab-account-approval] - 10https://gitlab.wikimedia.org/toolforge-repos/gitlab-account-approval/-/merge_requests/22 [17:21:17] (03update) 10bd808: Update runtime and patch a small bug [toolforge-repos/gitlab-account-approval] - 10https://gitlab.wikimedia.org/toolforge-repos/gitlab-account-approval/-/merge_requests/22 (https://phabricator.wikimedia.org/T423909) [17:22:38] (03CR) 10Lokal Profil: [C:03+2] Fixing double destination for de-hh_de [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1275474 (https://phabricator.wikimedia.org/T55271) (owner: 10Akoopal) [17:22:47] 10Tool-gitlab-account-approval, 06Release-Engineering-Team, 10GitLab (Account Approval): GitLab Account Approval Bot is failing - https://phabricator.wikimedia.org/T423909#11840555 (10bd808) p:05Triage→03High a:03bd808 [17:23:03] (03update) 10bd808: Update runtime and patch a small bug [toolforge-repos/gitlab-account-approval] - 10https://gitlab.wikimedia.org/toolforge-repos/gitlab-account-approval/-/merge_requests/22 (https://phabricator.wikimedia.org/T423909) [17:24:44] (03Merged) 10jenkins-bot: Fixing double destination for de-hh_de [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1275474 (https://phabricator.wikimedia.org/T55271) (owner: 10Akoopal) [17:28:30] (03PS1) 10Elukey: Add fake private secrets for discovery2026 PKI intermediate [labs/private] - 10https://gerrit.wikimedia.org/r/1275481 (https://phabricator.wikimedia.org/T420993) [17:29:19] (03merge) 10bd808: Update runtime and patch a small bug [toolforge-repos/gitlab-account-approval] - 10https://gitlab.wikimedia.org/toolforge-repos/gitlab-account-approval/-/merge_requests/22 (https://phabricator.wikimedia.org/T423909) [17:38:09] 10Tool-gitlab-account-approval, 06Release-Engineering-Team, 10GitLab (Account Approval): GitLab Account Approval Bot is failing - https://phabricator.wikimedia.org/T423909#11840604 (10bd808) 05Open→03Resolved I forgot to add the task id when logging [[https://sal.toolforge.org/log/qu3yq50BffdvpiTrPwa... [17:45:07] 10Tool-centralnotice-banner-editor: Title: Add support for localisable messages in banner text - https://phabricator.wikimedia.org/T423925 (10Oyelola_Victoria) 03NEW [17:45:48] 10Tool-centralnotice-banner-editor: Title: Add support for localisable messages in banner text - https://phabricator.wikimedia.org/T423925#11840628 (10Oyelola_Victoria) 05Open→03Invalid [17:46:45] 10Tool-centralnotice-banner-editor: Add support for localisable messages in banner text - https://phabricator.wikimedia.org/T423927 (10Oyelola_Victoria) 03NEW [17:46:56] 10Tool-centralnotice-banner-editor: Add support for localisable messages in banner text - https://phabricator.wikimedia.org/T423927#11840664 (10Oyelola_Victoria) p:05Triage→03High [18:01:32] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:01:43] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [18:06:32] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:06:43] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [18:10:07] 10Tool-centralnotice-banner-editor: Text element grows in size when padding is increased and can overflow - https://phabricator.wikimedia.org/T423934 (10Oyelola_Victoria) 03NEW [18:10:11] 10Tool-centralnotice-banner-editor: Text element grows in size when padding is increased and can overflow - https://phabricator.wikimedia.org/T423934#11840835 (10Oyelola_Victoria) p:05Triage→03High [18:15:32] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:15:43] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [18:18:51] 10Tool-centralnotice-banner-editor: Add auto width and height option for banner elements - https://phabricator.wikimedia.org/T423936 (10Oyelola_Victoria) 03NEW [18:19:41] (03CR) 10JHathaway: [C:03+1] Add fake private secrets for discovery2026 PKI intermediate [labs/private] - 10https://gerrit.wikimedia.org/r/1275481 (https://phabricator.wikimedia.org/T420993) (owner: 10Elukey) [18:19:58] 10Tool-centralnotice-banner-editor: Add auto width and height option for banner elements - https://phabricator.wikimedia.org/T423936#11840912 (10Oyelola_Victoria) p:05Triage→03Medium [18:20:32] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:20:43] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [18:22:32] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:27:32] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:29:32] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:44:32] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:45:32] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:45:43] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [18:50:32] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:50:43] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [18:52:02] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:54:47] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:57:02] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [19:42:28] (03CR) 10Akoopal: [C:03+2] use echo_time when available [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1274022 (owner: 10Lokal Profil) [19:45:56] (03Merged) 10jenkins-bot: use echo_time when available [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1274022 (owner: 10Lokal Profil) [19:57:41] 10Tool-etherpad-backup: Special characters are not handled consistently in the current backups, should be improved when migrating to S3 - https://phabricator.wikimedia.org/T423905#11841375 (10bd808) I //think// the object storage backend will get rid of issues with needing to encode pad names on disk. I can see... [20:24:20] (03PS1) 10Lokal Profil: Catch duplicate field dest values in monuments_config [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1275515 [20:26:37] (03PS2) 10Lokal Profil: Fix removed (in 9+) pywikibot APIs in bot scripts [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1270569 (https://phabricator.wikimedia.org/T409003) [20:29:20] 06cloud-services-team, 10superset.wmcloud.org: Sunset superset.wmcloud.org - https://phabricator.wikimedia.org/T416373#11841489 (10Andrew) @Dev I will add you to the project; can you tell me your toolforge/cloud-vps username or shell name? [20:29:32] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [20:29:43] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [20:30:25] 06cloud-services-team, 10superset.wmcloud.org: Sunset superset.wmcloud.org - https://phabricator.wikimedia.org/T416373#11841492 (10Andrew) @KCVelaga_WMF I have added you as a project member. [20:31:15] 06cloud-services-team, 10superset.wmcloud.org: Sunset superset.wmcloud.org - https://phabricator.wikimedia.org/T416373#11841498 (10Andrew) @Dev and @KCVelaga_WMF I would like to announce the new stewardship of this project as a followup to this email: https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lis... [20:34:32] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [20:34:43] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [20:38:59] (03PS3) 10Lokal Profil: Fix removed (in 9+) pywikibot APIs in bot scripts [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1270569 (https://phabricator.wikimedia.org/T409003) [20:40:57] (03CR) 10CI reject: [V:04-1] Fix removed (in 9+) pywikibot APIs in bot scripts [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1270569 (https://phabricator.wikimedia.org/T409003) (owner: 10Lokal Profil) [20:55:31] (03PS1) 10Lokal Profil: Escape field defaults when generating CREATE TABLE SQL [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1275522 [21:13:32] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [21:13:43] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [21:18:32] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [21:18:43] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [21:20:12] (03PS1) 10Lokal Profil: Tighten monuments_config schema and validation tests [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1275532 [21:25:39] (03PS4) 10Lokal Profil: Fix removed (in 9+) pywikibot APIs in bot scripts [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1270569 (https://phabricator.wikimedia.org/T409003) [21:33:50] 10Tool-wikimedia-attribution: [Proposal] Make the Wikimedia Attribution Framework and API discoverable for AI agents - https://phabricator.wikimedia.org/T423885#11841647 (10RHo) Thanks for raising this Sarai! This fits into the long term vision to help any Reuser including AI agents to follow best attribution pr... [21:45:46] (03CR) 10Akoopal: Fix removed (in 9+) pywikibot APIs in bot scripts (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1270569 (https://phabricator.wikimedia.org/T409003) (owner: 10Lokal Profil) [21:51:32] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [21:56:32] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [22:01:32] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [22:01:43] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [22:04:09] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24692640192 (https://github.com/cluebotng/component-configs/commits/6ccc54ff83a9536189370f93b0f3b7f9225e4259) [22:04:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [22:04:37] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24692640197 (https://github.com/cluebotng/component-configs/commits/6ccc54ff83a9536189370f93b0f3b7f9225e4259) [22:04:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [22:05:52] 10Tool-etherpad-backup, 13Patch-For-Review: Setup proof of concept storage and retrieval from WMCS object storage - https://phabricator.wikimedia.org/T422958#11841722 (10bd808) [22:06:10] 06cloud-services-team, 10Cloud-VPS: Support newer database engines on Trove - https://phabricator.wikimedia.org/T420737#11841725 (10Andrew) 05Open→03Resolved Available db engines are now: mariadb12.2.2, postgresql18.3. [22:06:32] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [22:06:43] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [22:11:32] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [22:15:32] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [22:16:43] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [22:20:32] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [22:21:43] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [22:59:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [23:22:44] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24695320410 (https://github.com/cluebotng/component-configs/commits/279edf060f43353ea66e6d057773bfdb883b16a1) [23:22:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [23:39:08] 06cloud-services-team, 10PAWS, 10Lingua-Libre: Allows request to all endpoints in wikidata allowlist - https://phabricator.wikimedia.org/T423969 (10Yug) 03NEW [23:43:40] 06cloud-services-team, 10Cloud-VPS: [tofu-cloudvps] Add support for importing legacy cloudvps_puppet_prefix objects - https://phabricator.wikimedia.org/T423970 (10bd808) 03NEW