[00:12:27] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24540678933 (https://github.com/cluebotng/component-configs/commits/26849735bbefbe218cbe0ce41db5a35941798c7b) [00:12:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [00:14:30] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24540678937 (https://github.com/cluebotng/component-configs/commits/26849735bbefbe218cbe0ce41db5a35941798c7b) [00:14:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [00:20:25] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24540900579 (https://github.com/cluebotng/component-configs/commits/bfd1f6b6b742e082dbbc587464ca135bfe827765) [00:20:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [01:33:34] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24543036687 (https://github.com/cluebotng/component-configs/commits/b97517784b752f7d0055837db122456017b21866) [01:33:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [02:51:39] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24545107848 (https://github.com/cluebotng/component-configs/commits/93640977d7dd4117faf49bd179aac37720eee5b2) [02:51:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [03:38:41] FIRING: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:44:20] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#11832473 (10Marostegui) I think it may take around 5-6 hours. But to be on the safe side in case of issues, I'd suggest we announce a possible 2... [07:01:51] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Engineering-Radar, 06DBA: Re-run maintainviews on all clouddb* and an-redacteddb1001.eqiad.wmnet - https://phabricator.wikimedia.org/T422459#11832571 (10FCeratto-WMF) [07:03:01] 06cloud-services-team, 10Data-Services, 06DBA: Table/View imagelinks in commonswiki_p throws an error - https://phabricator.wikimedia.org/T423550#11832577 (10FCeratto-WMF) I've run the `maintain-views --all-databases --replace --table imagelinks` cleanup on the s4 an-redacted and clouddb hosts except clouddb... [07:21:00] 06cloud-services-team, 10Toolforge: Buildservice for Rust fails - https://phabricator.wikimedia.org/T423675 (10magnusmanske) 03NEW [07:56:16] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Engineering-Radar, 06DBA: Re-run maintainviews on all clouddb* and an-redacteddb1001.eqiad.wmnet - https://phabricator.wikimedia.org/T422459#11832656 (10FCeratto-WMF) The run on s5 had an error that needs further investigation: https://ph... [07:58:34] 06cloud-services-team, 10Toolforge: Buildservice for Rust fails - https://phabricator.wikimedia.org/T423675#11832657 (10magnusmanske) Update: Seems to work now, waiting for completion (it's been slow the last few days) [08:01:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [08:02:39] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-31 (2026-04-07 to 2026-04-21)), 13Patch-For-Review: Fix linter issues discovered during implementation of the OAD example - https://phabricator.wikimedia.org/T414974#11832662 (10KBach) [08:08:13] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-31 (2026-04-07 to 2026-04-21)), 13Patch-For-Review: Fix linter issues discovered during implementation of the OAD example - https://phabricator.wikimedia.org/T414974#11832673 (10KBach) `wikimedia-paths-parameter-description-enums` is no longer a fal... [08:13:48] 06cloud-services-team, 10Data-Services, 06DBA: Table/View imagelinks in commonswiki_p throws an error - https://phabricator.wikimedia.org/T423550#11832692 (10Wurgl) Thanks. Works for me. [08:18:49] 06cloud-services-team, 10Data-Services, 06DBA: Table/View imagelinks in commonswiki_p throws an error - https://phabricator.wikimedia.org/T423550#11832699 (10FCeratto-WMF) 05Open→03Resolved Ok, thanks [08:31:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [10:02:19] 06cloud-services-team, 10Toolforge, 06Infrastructure-Foundations, 06SRE: Adjust WMCS Gitlab CI/CD repo to stop using mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423596#11832943 (10A_smart_kitten) (per T423596#11829501 & as the files seem potentially toolforge-related FWICS; feel free to reta... [10:59:50] (03update) 10sarai-wmf: Copyedits [toolforge-repos/wikimedia-attribution] - 10https://gitlab.wikimedia.org/toolforge-repos/wikimedia-attribution/-/merge_requests/2 (owner: 10apaskulin) [11:03:55] (03approved) 10sarai-wmf: Copyedits [toolforge-repos/wikimedia-attribution] - 10https://gitlab.wikimedia.org/toolforge-repos/wikimedia-attribution/-/merge_requests/2 (owner: 10apaskulin) [11:04:07] (03merge) 10sarai-wmf: Copyedits [toolforge-repos/wikimedia-attribution] - 10https://gitlab.wikimedia.org/toolforge-repos/wikimedia-attribution/-/merge_requests/2 (owner: 10apaskulin) [11:32:48] 06cloud-services-team, 10Data-Services, 06DBA, 06DC-Ops, and 3 others: decommission clouddb1019.eqiad.wmnet - https://phabricator.wikimedia.org/T423151#11833255 (10Jclark-ctr) [11:32:50] 06cloud-services-team, 10Data-Services, 06DBA, 06DC-Ops, and 3 others: decommission clouddb1019.eqiad.wmnet - https://phabricator.wikimedia.org/T423151#11833256 (10Jclark-ctr) 05Open→03Resolved [12:33:58] (03update) 10aghirelli: fix(linter): fix five false positives on OAD example [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/15 [12:40:33] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#11833394 (10fnegri) @Marostegui for s6, let's proceed with announcing a 24h maintenance window. Let me know when you have time to do it and I'll... [12:43:00] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#11833397 (10Marostegui) >>! In T409557#11833394, @fnegri wrote: > @Marostegui for s6, let's proceed with announcing a 24h maintenance window. Le... [14:15:02] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24569591336 (https://github.com/cluebotng/component-configs/commits/f80158cf5eb8b262e699d1bd27b4793819987bf6) [14:15:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [14:15:34] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24569591292 (https://github.com/cluebotng/component-configs/commits/f80158cf5eb8b262e699d1bd27b4793819987bf6) [14:15:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [14:51:22] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Engineering-Radar, 06DBA: Re-run maintainviews on all clouddb* and an-redacteddb1001.eqiad.wmnet - https://phabricator.wikimedia.org/T422459#11833785 (10FCeratto-WMF) [15:02:50] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06tools-platform-team, 13Patch-For-Review: [wikireplicas] add proper dry-run/diff mode to maintain-views - https://phabricator.wikimedia.org/T351637#11833832 (10fnegri) Here's an example of showing a diff with `--debug`: `lang=shell-session fnegr... [15:23:25] (03open) 10taavi: kubernetes: Stop manipulating Ingress objects [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/100 (https://phabricator.wikimedia.org/T392356) [15:23:27] (03update) 10taavi: kubernetes: Stop manipulating Ingress objects [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/100 (https://phabricator.wikimedia.org/T392356) [15:24:53] (03open) 10taavi: Remove tool ability to interact with Ingress objects [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/86 (https://phabricator.wikimedia.org/T418276) [15:24:56] (03update) 10taavi: Remove tool ability to interact with Ingress objects [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/86 (https://phabricator.wikimedia.org/T418276) [15:25:31] (03update) 10taavi: Remove tool ability to interact with Ingress objects [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/86 (https://phabricator.wikimedia.org/T392356 https://phabricator.wikimedia.org/T418276) [15:25:36] (03update) 10taavi: Remove tool ability to interact with Ingress objects [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/86 (https://phabricator.wikimedia.org/T392356 https://phabricator.wikimedia.org/T418276) [15:26:27] (03open) 10taavi: Undeploy ingress-nginx [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1222 (https://phabricator.wikimedia.org/T392356) [15:26:29] (03update) 10taavi: Undeploy ingress-nginx [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1222 (https://phabricator.wikimedia.org/T392356) [15:38:25] 06cloud-services-team, 10superset.wmcloud.org: Sunset superset.wmcloud.org - https://phabricator.wikimedia.org/T416373#11833942 (10Dev) I would like to formally volunteer to take on **maintainership ** of the Superset instance at https://superset.wmcloud.org I am a member of the Indic MediaWiki Developers Use... [15:42:50] 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: [wikireplicas] Update grants for "maintainviews" user - https://phabricator.wikimedia.org/T422806#11833968 (10fnegri) I merged the patches but they need to be applied manually. So far I've only applied the new grants to clouddb... [15:45:03] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#11833972 (10fnegri) > I can work on it on Tuesday. Announcement sent: https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedi... [15:51:52] (03update) 10aghirelli: fix(linter): fix five false positives on OAD example [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/15 [15:59:37] 06cloud-services-team, 10Toolforge, 07Documentation: uploadmap reverse proxy broken because of platform changes - https://phabricator.wikimedia.org/T423652#11834016 (10GPSLeo) I used the config you created and it works. Thanks! I only had to change one thing: The space for the names of the proxy services see... [16:07:07] 06cloud-services-team, 10superset.wmcloud.org: Sunset superset.wmcloud.org - https://phabricator.wikimedia.org/T416373#11834076 (10Dev) Adding to the above - there is also recent community-driven usage of this instance. Members of the Indic MediaWiki Developers User Group have created a dashboard for the Indi... [16:59:02] (03update) 10raymond-ndibe: refactor to use job_type argument that is already available in jobs-api [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T423412) [17:11:06] (03update) 10raymond-ndibe: refactor to use job_type argument that is already available in jobs-api [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T423412) [17:12:25] (03update) 10danyya: Draft: Refactor charts [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/9 [17:15:11] (03update) 10danyya: Refactor charts [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/9 [17:15:20] (03merge) 10danyya: Refactor charts [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/9 [17:17:49] 10Tool-humaniki-2: Make chart filters collapsible - https://phabricator.wikimedia.org/T423053#11834389 (10Danya) 05Open→03In progress [17:18:05] 06tools-platform-team: lima-kilo: remove git dir check for local deployment - https://phabricator.wikimedia.org/T423731 (10Raymond_Ndibe) 03NEW [17:18:27] (03open) 10danyya: Draft: Collapsible chart filters [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/10 [17:22:16] 10Toolforge, 06tools-platform-team: lima-kilo: remove git dir check for local deployment - https://phabricator.wikimedia.org/T423731#11834407 (10taavi) [17:22:29] 10Toolforge, 06tools-platform-team: [jobs-cli] refactor to use job_type argument that is already available in jobs-api - https://phabricator.wikimedia.org/T423412#11834408 (10taavi) [17:22:51] 10Cloud-VPS, 06tools-infrastructure-team: Consider allowing cumin access to all Cloud VPS VMs - https://phabricator.wikimedia.org/T422801#11834409 (10taavi) [17:23:02] 10Cloud-VPS, 06tools-infrastructure-team: Openstack uwsgi logging to '.log' - https://phabricator.wikimedia.org/T422830#11834411 (10taavi) [17:23:12] 10Toolforge, 06tools-platform-team: [builds-builder] use yq instead of tomljson/jq/jsontoml - https://phabricator.wikimedia.org/T422691#11834412 (10taavi) [17:23:29] (03open) 10raymond-ndibe: helpers:toolforge_deploy: check that .git exists before deployment [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/319 (https://phabricator.wikimedia.org/T423731) [17:28:44] 06cloud-services-team, 10Toolforge, 07Documentation: uploadmap reverse proxy broken because of platform changes - https://phabricator.wikimedia.org/T423652#11834427 (10bd808) >>! In T423652#11834016, @GPSLeo wrote: > I used the config you created and it works. Thanks! I only had to change one thing: The spac... [17:31:17] 06cloud-services-team, 10Toolforge, 07Documentation: uploadmap reverse proxy broken because of platform changes - https://phabricator.wikimedia.org/T423652#11834437 (10bd808) >>! In T423652#11834016, @GPSLeo wrote: > One page that still needs an update to reflect this change is: https://wikitech.wikimedia.or... [17:33:06] (03update) 10danyya: Draft: Collapsible chart filters [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/10 [17:35:11] (03update) 10danyya: Collapsible chart filters [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/10 [17:35:24] (03merge) 10danyya: Collapsible chart filters [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/10 [17:37:34] 10Tool-humaniki-2: Make chart filters collapsible - https://phabricator.wikimedia.org/T423053#11834466 (10Danya) 05In progress→03Resolved [18:16:54] 10Tool-humaniki-2: [Birth-years view] Stats for persons with at least one Wikipedia page - https://phabricator.wikimedia.org/T422340#11834586 (10Danya) 05Open→03In progress [18:17:10] 10Tool-humaniki-2: [Birth-years view] Stats for persons with at least one wiki page - https://phabricator.wikimedia.org/T422340#11834587 (10Danya) [18:18:16] (03open) 10danyya: Draft: Add date-of-birth stats for Wikidata elements with at least one wiki page [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/11 [18:23:13] (03approved) 10aaron: Add 'path' column to spec result table [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/14 (owner: 10wikigit) [18:24:13] (03merge) 10aaron: Add 'path' column to spec result table [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/14 (owner: 10wikigit) [20:50:45] (03PS1) 10Lokal Profil: always use PYWIKIBOT_BIN env variable [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1274009 [21:00:44] (03PS1) 10Lokal Profil: use echo_time when available [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1274022 [21:13:42] (03CR) 10Akoopal: "I just checked on toolforge, echo_time doesn't seem to be available, so this will break things." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1274022 (owner: 10Lokal Profil) [21:47:52] (03PS1) 10Lokal Profil: Pin venv interpreter via TOOLFORGE_VENV_PYTHON [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1274060 [22:30:18] (03CR) 10Lokal Profil: [C:03+2] ignoring some more fields [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271840 (https://phabricator.wikimedia.org/T55271) (owner: 10Akoopal) [22:31:57] (03PS1) 10Lokal Profil: Sync scheduled Toolforge jobs from jobs.yml.template [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1274108 [22:32:27] (03Merged) 10jenkins-bot: ignoring some more fields [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271840 (https://phabricator.wikimedia.org/T55271) (owner: 10Akoopal) [22:47:26] (03CR) 10Lokal Profil: "echo_time is locally defined in defaults.sh, which is already loaded in each of these cases" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1274022 (owner: 10Lokal Profil) [23:47:36] 10Tool-refill, 07Regression: regression: diff feature is broken - https://phabricator.wikimedia.org/T423566#11835314 (10Novem_Linguae) I did this hacky little patch a couple days ago to get the wdiff (aka WikEdDiff) library working a little bit in the master branch. This may be a good clue as to why diffs in t... [23:52:02] 10Tool-wikimonitor: 30-Minute SseEmitter Timeout with Event Resumption - https://phabricator.wikimedia.org/T422194#11835317 (10Gerges) Hi @Praffq, Are you still planning to submit a new pull request instead of reopening the closed one?