[02:18:15] 10Tool-wd-act: Add basic CI/CD scripts for deploying to Toolforge - https://phabricator.wikimedia.org/T434466 (10Tekneos) 03NEW [02:18:31] 10Tool-wd-act: Add basic CI/CD scripts for deploying to Toolforge - https://phabricator.wikimedia.org/T434466#12200397 (10Tekneos) 05Open→03In progress [02:25:23] 10Tool-wd-act: Add basic CI/CD scripts for deploying to Toolforge - https://phabricator.wikimedia.org/T434466#12200399 (10Tekneos) After the scripts are written and checked in, I have to figure out how to copy them onto the toolforge build server so I can actually use them. [02:38:42] (03open) 10raymond-ndibe: Draft: feat: support jobs exec [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/368 [02:38:59] (03close) 10raymond-ndibe: draft: container exec experiment [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/349 [02:39:35] (03open) 10raymond-ndibe: Draft: feat: support jobs exec [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/174 [02:51:10] (03update) 10raymond-ndibe: Draft: feat: support jobs exec [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/368 [03:08:51] 10Tool-wikinewsie: Adjustable image alignment on Wikinewsie - https://phabricator.wikimedia.org/T434467 (10Pharos) 03NEW [03:08:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:15:44] 10Tool-wikinewsie: 'See also' mode for Wikinewsie - https://phabricator.wikimedia.org/T434468 (10Pharos) 03NEW [03:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:27:04] 10Tool-wikinewsie: Recent sources for Wikinewsie - https://phabricator.wikimedia.org/T433528#12200435 (10Pharos) >>! In T433528#12198879, @ZhaoFJx wrote: > @Pharos A machine-readable rated list of the references... Perhaps a list like this, but in a more structured way? > > {F97871513} Yes indeed, something li... [06:32:13] (03update) 10waldir: Add labels to the Phabricator and GitLab badges [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/4 [06:51:51] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 10BetaFeatures, and 6 others: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#12200579 (10Gehel) [06:57:07] 06Toolforge-standards-committee, 10Tools, 06Abstract Wikipedia team, 07Privacy: abstract-data tool should use Toolforge FontCDN mirror - https://phabricator.wikimedia.org/T434452#12200686 (10DSantamaria) Thanks for the report! It should be fixed now; let me know if you find it isn't. On the tags, I haven'... [07:00:23] 06Toolforge-standards-committee, 10Tools, 06Abstract Wikipedia team, 07Privacy: abstract-data tool should use Toolforge FontCDN mirror - https://phabricator.wikimedia.org/T434452#12200764 (10DSantamaria) 05Open→03Resolved a:03DSantamaria [07:00:42] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Engineering, and 4 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#12200775 (10Gehel) [07:01:14] 10Cloud Services Proposals, 10Data-Services, 06tools-platform-team, 06Data-Persistence, and 2 others: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#12200790 (10Gehel) [07:01:22] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12200792 (10Gehel) [07:01:55] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Engineering, and 4 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#12200801 (10Gehel) [07:02:47] 10Tool-abstract-data, 06Toolforge-standards-committee, 07Privacy: abstract-data tool should use Toolforge FontCDN mirror - https://phabricator.wikimedia.org/T434452#12200823 (10DSantamaria) [07:02:55] 10Tool-abstract-data, 06Toolforge-standards-committee, 06Abstract Wikipedia team, 07Privacy: abstract-data tool should use Toolforge FontCDN mirror - https://phabricator.wikimedia.org/T434452#12200824 (10DSantamaria) [07:12:04] 10Toolforge, 06tools-platform-team: TOOL_NAME environment variable - https://phabricator.wikimedia.org/T434431#12200854 (10DamianZaremba) As a workaround you can read `/var/run/secrets/kubernetes.io/serviceaccount/namespace` and strip off the `tool-` prefix from inside the pod. [07:25:00] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1323957 (owner: 10L10n-bot) [07:54:27] (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/116 [07:54:32] (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/116 [07:54:49] (03approved) 10dcaro: start-devenv: no need for fancy copy function [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/336 [07:54:53] (03merge) 10dcaro: start-devenv: no need for fancy copy function [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/336 [07:55:56] (03update) 10dcaro: ansible: stop using injected facts [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/337 (https://phabricator.wikimedia.org/T413782) (owner: 10mahveotm) [07:59:38] (03update) 10dcaro: tests: add behavior tests for previously untested code branches [repos/cloud/toolforge/jobs-api] (refactor_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/362 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [09:06:48] (03update) 10dcaro: tests: refactor test helpers, fixtures and existing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/366 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [09:08:42] (03update) 10dcaro: refactor: rename CommonJob to CommonOptions [repos/cloud/toolforge/jobs-api] (move_cmd_out_of_common_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/367 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [09:14:51] 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance: [lima-kilo,cert-manager] failing to create from scratch - https://phabricator.wikimedia.org/T434492 (10dcaro) 03NEW p:05Triage→03High [09:14:54] 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance: [lima-kilo,cert-manager] failing to create from scratch - https://phabricator.wikimedia.org/T434492#12201397 (10dcaro) 05Open→03In progress [09:14:57] 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance: [lima-kilo,cert-manager] failing to create from scratch - https://phabricator.wikimedia.org/T434492#12201398 (10dcaro) a:03dcaro [09:19:11] (03update) 10dcaro: remove "wait: true" from certmanager ansible script [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/335 (https://phabricator.wikimedia.org/T408786) (owner: 10andrew) [09:20:19] 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance: [lima-kilo,cert-manager] failing to create from scratch - https://phabricator.wikimedia.org/T434492#12201406 (10dcaro) I think that it might just be that we are missing https://gitlab.wikimedia.org/repo... [09:25:59] 10Toolforge, 06tools-platform-team: TOOL_NAME environment variable - https://phabricator.wikimedia.org/T434431#12201438 (10Gouvernathor) Nope, doesn't work, not at build time. It gets me "image-build". Same issue as T434369. [09:30:16] 10Toolforge, 06tools-platform-team: TOOL_NAME environment variable - https://phabricator.wikimedia.org/T434431#12201480 (10dcaro) >>! In T434431#12201438, @Gouvernathor wrote: > Nope, doesn't work, not at build time. It gets me "image-build". Same issue as T434369. Yep, that's because the builds don't happen... [09:33:00] (03merge) 10dcaro: remove "wait: true" from certmanager ansible script [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/335 (https://phabricator.wikimedia.org/T408786) (owner: 10andrew) [09:33:24] 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance: [lima-kilo,cert-manager] failing to create from scratch - https://phabricator.wikimedia.org/T434492#12201484 (10dcaro) yep, that was it, merged, I'll close. [09:33:27] 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance: [lima-kilo,cert-manager] failing to create from scratch - https://phabricator.wikimedia.org/T434492#12201485 (10dcaro) 05In progress→03Resolved [09:37:27] 10Toolforge, 06tools-platform-team: TOOL_NAME environment variable - https://phabricator.wikimedia.org/T434431#12201497 (10taavi) An image is not inherently tried to a tool, as it is possible to build an image with one tool and deploy it to an another. https://wikitech.wikimedia.org/wiki/Tool:Containers is a g... [09:45:48] 10Toolforge, 06tools-platform-team: TOOL_NAME environment variable - https://phabricator.wikimedia.org/T434431#12201541 (10dcaro) >>! In T434431#12201497, @taavi wrote: > Do we expose that as an option in the component manifest yet? Created {T434496} to not forget xd [09:46:06] 10Toolforge, 06tools-platform-team: [builds-api,envvars-admission] add TOOL_NAME environment variable - https://phabricator.wikimedia.org/T434431#12201543 (10dcaro) [09:46:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [09:46:21] 10Toolforge, 06tools-platform-team: [builds-api] TOOL_DATA_DIR not available during `toolforge build start` execution - https://phabricator.wikimedia.org/T434369#12201546 (10dcaro) [09:47:55] (03update) 10mahveotm: ansible: stop using injected facts [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/337 (https://phabricator.wikimedia.org/T413782) [09:59:27] (03update) 10mahveotm: ansible: stop using injected facts [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/337 (https://phabricator.wikimedia.org/T413782) [10:09:20] (03CR) 10FNegri: [C:03+1] vps: create_instance: Add flag to specify number of first server [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1308683 (owner: 10Majavah) [10:11:00] (03CR) 10Majavah: [C:03+2] vps: create_instance: Add flag to specify number of first server [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1308683 (owner: 10Majavah) [10:14:24] (03Merged) 10jenkins-bot: vps: create_instance: Add flag to specify number of first server [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1308683 (owner: 10Majavah) [10:16:34] FIRING: ToolsNfsAlmostFull: Toolforge NFS is almost full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsNfsAlmostFull [10:18:04] (03CR) 10Filippo Giunchedi: [C:03+1] vps: create_instance: Add codfw1dev support [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1320917 (owner: 10Majavah) [10:19:27] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12201688 (10fnegri) 05In progress→03Resolved History length remained low for the past 7 days, I'm marking this task as Resolved. {F98052048} [10:22:20] 10Toolforge, 06tools-platform-team: [components-api] split cancel endpoint to avoid duplicated operation id - https://phabricator.wikimedia.org/T434499 (10dcaro) 03NEW [10:26:04] (03update) 10dcaro: [T401993] Add a "description" field to the deployment [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/181 (https://phabricator.wikimedia.org/T401993) (owner: 10mahveotm) [11:45:34] 10Tool-wmf-openapi-linter, 06Content-Platform-Team (Work In Progress), 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 05MW-1.47-notes (1.47.0-wmf.15; 2026-08-11), 07OKR-Work: Add parameter examples to TransformHandler.php - https://phabricator.wikimedia.org/T432228#12202067 (10AGhirelli... [11:59:52] (03update) 10dcaro: [T401993] Add a "description" field to the deployment [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/181 (https://phabricator.wikimedia.org/T401993) (owner: 10mahveotm) [12:16:14] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12202165 (10fgiunchedi) >>! In T431682#12174859, @taavi wrote: > Good question. Looking at the Netbox changelog that cha... [12:21:32] 06cloud-services-team, 10Quarry, 06tools-platform-team, 06RoadToWiki, and 3 others: "1 rows" should be "1 row" - https://phabricator.wikimedia.org/T419564#12202187 (10Nihar_Chakravarti) Sorry for the late reply! Picked this back up after a break - double checked everything locally and it's all working corr... [12:23:54] (03update) 10dcaro: ansible: stop using injected facts [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/337 (https://phabricator.wikimedia.org/T413782) (owner: 10mahveotm) [12:26:36] (03approved) 10dcaro: ansible: stop using injected facts [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/337 (https://phabricator.wikimedia.org/T413782) (owner: 10mahveotm) [12:27:17] (03merge) 10dcaro: ansible: stop using injected facts [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/337 (https://phabricator.wikimedia.org/T413782) (owner: 10mahveotm) [12:29:04] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: lima-kilo: INJECT_FACTS_AS_VARS default to `True` is deprecated - https://phabricator.wikimedia.org/T413782#12202231 (10dcaro) 05In progress→03Resolved [12:29:21] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [lima-kilo] ansible deprecation errors - https://phabricator.wikimedia.org/T429343#12202233 (10dcaro) Only the apt repository warning is left :) [12:42:09] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12202330 (10fgiunchedi) >>! In T431682#12180113, @VRiley-WMF wrote: > @Andrew Awesome, sounds good. Would we like to sta... [13:19:39] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12202616 (10Ottomata) > eventstreams beta (at https:... [13:37:57] 10Cloud-VPS, 06tools-infrastructure-team, 10Ceph: "osd.224 observed stalled read indications in DB device" - https://phabricator.wikimedia.org/T434402#12202820 (10dcaro) Currently it's another host complaining: ` [WRN] DB_DEVICE_STALLED_READ_ALERT: 1 OSD(s) experiencing stalled read in db device of BlueFS... [13:42:17] (03update) 10devjadiya: Complete German, Chinese, Portuguese, Spanish, and French interface translations [toolforge-repos/clip2commons] - 10https://gitlab.wikimedia.org/toolforge-repos/clip2commons/-/merge_requests/10 (https://phabricator.wikimedia.org/T434092) (owner: 10anushka10patel) [13:45:07] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12202903 (10Ottomata) @btullis ah! we have to wrap this up eh? Find me anyt... [13:46:24] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [13:47:10] 10Tool-clip2commons, 13Patch-For-Review: Complete remaining translations - https://phabricator.wikimedia.org/T434092#12202927 (10Dev) Thankyou for taking on this task! I have reviewed the current [[ https://gitlab.wikimedia.org/toolforge-repos/clip2commons/-/merge_requests/10 | merge request ]]. The five remai... [13:53:17] 06tools-platform-team, 10Toolhub: Add banner about Toolhub Evolved - https://phabricator.wikimedia.org/T434556 (10aputhin) 03NEW [13:53:59] 06tools-platform-team, 10Toolhub: Add banner about Toolhub Evolved - https://phabricator.wikimedia.org/T434556#12203029 (10aputhin) 05Open→03In progress p:05Triage→03Medium [14:10:54] (03update) 10raymond-ndibe: global: add log write endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/32 (owner: 10dcaro) [14:12:40] (03update) 10fnegri: Add new alerts for tool maintainers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/66 (https://phabricator.wikimedia.org/T432863) [14:19:21] 10Tool-wikimedia-attribution, 06MediaWiki-API-Platform-Team, 06MediaWiki-Core-Platform-Team, 10MediaWiki-REST-API, 07OKR-Work: Download CTA should be present only on Wikipedia projects - https://phabricator.wikimedia.org/T434214#12203216 (10matmarex) [14:19:27] 10Tool-wikimedia-attribution, 06MediaWiki-API-Platform-Team, 06MediaWiki-Core-Platform-Team, 10MediaWiki-REST-API, 07OKR-Work: Download CTA should be present only on Wikipedia projects - https://phabricator.wikimedia.org/T434214#12203218 (10matmarex) [14:19:57] 10Tool-wikimedia-attribution, 06MediaWiki-API-Platform-Team, 06MediaWiki-Core-Platform-Team, 10MediaWiki-REST-API, 07OKR-Work: Download CTA should be present only on Wikipedia projects - https://phabricator.wikimedia.org/T434214#12203227 (10pmiazga) a:05pmiazga→03None [14:24:17] (03update) 10dcaro: global: add log write endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/32 [14:26:03] (03update) 10dcaro: global: add log write endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/32 [14:42:29] (03update) 10raymond-ndibe: global: add log write endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/32 (owner: 10dcaro) [14:51:01] (03update) 10fnegri: Add new alerts for tool maintainers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/66 (https://phabricator.wikimedia.org/T432863) [14:57:09] 10Cloud-VPS, 06tools-infrastructure-team, 10Ceph: "osd.224 observed stalled read indications in DB device" - https://phabricator.wikimedia.org/T434402#12203457 (10Andrew) "ceph osd out 168" [14:58:00] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.roll_restart_osd_daemons (1324319) [15:01:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [15:05:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [15:20:51] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.roll_restart_osd_daemons (exit_code=0) (1324319) [15:25:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [15:25:23] 06cloud-services-team, 10Toolforge, 06tools-platform-team: Toolforge Build Service unavailable and `become` passwordless sudo failing on tools-bastion-15 - https://phabricator.wikimedia.org/T434573 (10Taiwania_Justo) 03NEW [15:47:48] 06cloud-services-team, 10Toolforge, 06tools-platform-team: Toolforge Build Service unavailable and `become` passwordless sudo failing on tools-bastion-15 - https://phabricator.wikimedia.org/T434573#12203796 (10dcaro) Hi @Taiwania_Justo, you were running vscode in the bastion, that filled up your memory quota... [15:48:34] 10Cloud-VPS, 06tools-infrastructure-team: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#12203801 (10Odinaldo) Hi @Andrew, Apologies for the slow response (there has been some staff unavailability slowing processes). Since the initial choice for Cloud VPS, we have rea... [15:51:09] 06cloud-services-team, 10Toolforge, 06tools-platform-team: VS-code using too many resources in bastion makes user session unstable - https://phabricator.wikimedia.org/T434573#12203814 (10dcaro) [16:04:10] 10Tool-wd-act: Update app to use default Python conventions for Toolforge, to simplify CI/CD - https://phabricator.wikimedia.org/T434577 (10Tekneos) 03NEW [16:04:32] 10Tool-wd-act: Update app to use default Python conventions for Toolforge, to simplify CI/CD - https://phabricator.wikimedia.org/T434577#12203903 (10Tekneos) 05Open→03In progress [16:05:24] 10Toolforge, 06tools-platform-team: [functional-tests] Do a pass to see what can be done to speed them up and how much effort will it take - https://phabricator.wikimedia.org/T434578 (10dcaro) 03NEW [16:05:53] (03update) 10dcaro: global: add log write endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/32 [16:14:34] 10Tool-itwiki: Improve the "Template:Conteggio cancellazioni/In corso/Voce" to automatically calculate the "temperature" 0-100 - https://phabricator.wikimedia.org/T433807#12203956 (10valerio.bozzolan) 05Open→03Resolved a:03Sakretsu Thanks @Sakretsu 😻 😻 😻 😻 😻 I've updated the bot to use your Lua featur... [16:17:41] (03update) 10anushka10patel: Remember last-selected license across uploads [toolforge-repos/clip2commons] - 10https://gitlab.wikimedia.org/toolforge-repos/clip2commons/-/merge_requests/11 (https://phabricator.wikimedia.org/T434084) [16:19:46] (03update) 10anushka10patel: Complete German, Chinese, Portuguese, Spanish, and French interface translations [toolforge-repos/clip2commons] - 10https://gitlab.wikimedia.org/toolforge-repos/clip2commons/-/merge_requests/10 (https://phabricator.wikimedia.org/T434092) [16:23:10] 10Tool-itwiki: Improve the "Template:Conteggio cancellazioni/In corso/Voce" to automatically calculate the "temperature" 0-100 - https://phabricator.wikimedia.org/T433807#12204003 (10valerio.bozzolan) For historical records, currently the BotCancellazioni has **2148363** edits. As I said, I expect the next... [16:29:16] (03update) 10dcaro: global: add log write endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/32 [16:38:57] (03update) 10fnegri: Add new alerts for tool maintainers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/66 (https://phabricator.wikimedia.org/T432863) [17:06:34] RESOLVED: ToolsNfsAlmostFull: Toolforge NFS is almost full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsNfsAlmostFull [17:11:33] 10Cloud-VPS, 06tools-infrastructure-team: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#12204310 (10Andrew) Thanks for the followup. 1) Probably! The only limitation here is that services hosted in toolforge will communicate via proxies so if you are doing something we... [18:22:05] 10Tool-itwiki: BotCancellazioni: rollback "cllimit=500" workaround after T433922 has been resolved - https://phabricator.wikimedia.org/T434598 (10valerio.bozzolan) 03NEW p:05Triage→03Low [18:22:23] 10Tool-itwiki: BotCancellazioni: rollback "cllimit=500" workaround after T433922 has been resolved - https://phabricator.wikimedia.org/T434598#12204715 (10valerio.bozzolan) [18:22:27] 10Tool-itwiki, 10MediaWiki-Action-API, 06MediaWiki-API-Platform-Team: Missing categories from generator=categorymembers (just 1 instead of 2) in itwiki - https://phabricator.wikimedia.org/T433922#12204716 (10valerio.bozzolan) [18:25:14] 10Tool-itwiki: Improve OrphanizerBot to handle disambiguation pages differently - https://phabricator.wikimedia.org/T320693#12204722 (10valerio.bozzolan) [18:26:44] (03CR) 10MusikAnimal: [C:03+2] "Thank you!!" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276371 (https://phabricator.wikimedia.org/T198872) (owner: 10Alien4444) [18:27:34] (03Merged) 10jenkins-bot: SimpleEditCounter: add latest edit and log [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276371 (https://phabricator.wikimedia.org/T198872) (owner: 10Alien4444) [18:31:00] (03CR) 10MusikAnimal: [C:03+2] "> putting three linting comments in a single function feels awkward." [labs/xtools] - 10https://gerrit.wikimedia.org/r/1320920 (https://phabricator.wikimedia.org/T433394) (owner: 10Alien4444) [18:31:57] (03Merged) 10jenkins-bot: EditCounter: fix local-time breaking timecard [labs/xtools] - 10https://gerrit.wikimedia.org/r/1320920 (https://phabricator.wikimedia.org/T433394) (owner: 10Alien4444) [18:37:23] 10Tool-itwiki, 07Upstream: Aggiungere un'etichetta alle modifiche di JWB - https://phabricator.wikimedia.org/T363923#12204785 (10valerio.bozzolan) [18:43:40] 10Tool-itwiki, 07Upstream: Aggiungere un'etichetta alle modifiche di JWB - https://phabricator.wikimedia.org/T363923#12204811 (10valerio.bozzolan) Credo ci sia ancora da creare una "issue" sul tema dei tag: https://github.com/wikimedia-gadgets/JWB/issues Tutti i tag di itwiki: https://it.wikipedia.org/wiki/... [18:57:45] 06cloud-services-team, 10Tool-paulina, 10Toolforge, 06tools-platform-team: Automated traffic affects tool performance - https://phabricator.wikimedia.org/T432878#12204997 (10Pepe_piton) @BLiviero-WMF @bd808 Thanks for you answers! >>! In T432878#12147778, @BLiviero-WMF wrote: > If it's visible to you, thi... [19:14:14] 10Tool-wikinewsie: Wikinewsie 'Mix' interface takes up too much space on mobile - https://phabricator.wikimedia.org/T434606 (10Pharos) 03NEW [19:39:38] 10Tool-wikinewsie: Dropdown for continents/regions on Wikinewsie - https://phabricator.wikimedia.org/T434608 (10Pharos) 03NEW [19:43:43] 10Tool-wikinewsie: Dropdown for continents/regions on Wikinewsie - https://phabricator.wikimedia.org/T434608#12205190 (10Pharos) [19:47:01] 10Tool-wikinewsie: Dropdown for continents/regions on Wikinewsie - https://phabricator.wikimedia.org/T434608#12205202 (10Pharos) [19:54:18] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Shutdown timeline extension requests for maintained projects - https://phabricator.wikimedia.org/T434103#12205236 (10BLiviero-WMF) [20:25:42] 10Tool-wikinewsie, 06WikiNYC Tech Committee: Set up Wikinews Pulse tool as alpha release on Toolforge - https://phabricator.wikimedia.org/T427575#12205402 (10Pharos) [20:30:03] 10Tool-wikinewsie: Dropdown for continents/regions on Wikinewsie - https://phabricator.wikimedia.org/T434608#12205418 (10ZhaoFJx) p:05Triage→03Medium [20:33:27] 10Tool-wikinewsie: Wikinewsie 'Mix' interface takes up too much space on mobile - https://phabricator.wikimedia.org/T434606#12205423 (10ZhaoFJx) p:05Triage→03High [20:36:49] 10Tool-wikinewsie: Adjustable image alignment on Wikinewsie - https://phabricator.wikimedia.org/T434467#12205434 (10ZhaoFJx) p:05Triage→03Medium [20:37:48] 10Tool-wikinewsie: 'See also' mode for Wikinewsie - https://phabricator.wikimedia.org/T434468#12205437 (10ZhaoFJx) p:05Triage→03Low [20:39:07] 10Tool-wikinewsie: Dropdown for continents/regions on Wikinewsie - https://phabricator.wikimedia.org/T434608#12205444 (10Pharos) [20:48:16] (03approved) 10zhaofjx: Add labels to the Phabricator and GitLab badges [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/4 (owner: 10waldir) [20:48:24] (03update) 10zhaofjx: Add labels to the Phabricator and GitLab badges [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/4 (owner: 10waldir) [20:56:06] (03merge) 10zhaofjx: Add labels to the Phabricator and GitLab badges [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/4 (owner: 10waldir) [20:58:09] (03update) 10zhaofjx: Update contributor credits and normalize git authorship data [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/6 (owner: 10waldir) [20:58:10] (03approved) 10zhaofjx: Update contributor credits and normalize git authorship data [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/6 (owner: 10waldir) [20:58:15] (03update) 10zhaofjx: Update contributor credits and normalize git authorship data [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/6 (owner: 10waldir) [20:58:23] (03merge) 10zhaofjx: Update contributor credits and normalize git authorship data [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/6 (owner: 10waldir) [21:11:04] 10Tool-wikinewsie, 07Spike: Explore Possible Wikinewsie Deploy Pipeline - https://phabricator.wikimedia.org/T434616 (10ZhaoFJx) 03NEW [21:11:38] 10Tool-wikinewsie, 07Spike: Explore Possible Wikinewsie Deploy Pipeline - https://phabricator.wikimedia.org/T434616#12205654 (10ZhaoFJx) p:05Triage→03Medium [21:11:57] 10Tool-wikinewsie, 07Spike: Explore Possible Wikinewsie Deploy Pipeline - https://phabricator.wikimedia.org/T434616#12205671 (10ZhaoFJx) 05Open→03In progress [21:12:01] 10Tool-wikinewsie, 07Spike: Explore Possible Wikinewsie Deploy Pipeline - https://phabricator.wikimedia.org/T434616#12205672 (10ZhaoFJx) a:03ZhaoFJx [21:12:15] 10Tool-wikinewsie, 07Spike: Explore Possible Wikinewsie Deploy Pipeline - https://phabricator.wikimedia.org/T434616#12205676 (10ZhaoFJx) [21:21:36] 10Tool-wikinewsie: Dropdown for continents/regions on Wikinewsie - https://phabricator.wikimedia.org/T434608#12205706 (10Pharos) I guess one implementation could be a single (very long) dropdown, with all of the continents and subregions. Another might be might be five separate continent-based dropdown menus, an... [21:24:27] 10Tool-wikinewsie: Adjustable image alignment on Wikinewsie - https://phabricator.wikimedia.org/T434467#12205709 (10Pharos) A rather more ambitious version of this idea would more resemble [[ https://commons.wikimedia.org/wiki/Commons:CropTool | Commons:CropTool ]] [21:28:27] 10Tool-wikinewsie: Strip all Wikinewsie text to i18n files - https://phabricator.wikimedia.org/T434617 (10ZhaoFJx) 03NEW [21:29:39] 10Tool-wikinewsie: Strip all Wikinewsie text to i18n files - https://phabricator.wikimedia.org/T434617#12205733 (10ZhaoFJx) p:05Triage→03Medium [21:29:55] 10Tool-wikinewsie: Strip all Wikinewsie text to i18n files - https://phabricator.wikimedia.org/T434617#12205736 (10ZhaoFJx) a:03ZhaoFJx [21:30:02] 10Tool-wikinewsie: Strip all Wikinewsie text to i18n files - https://phabricator.wikimedia.org/T434617#12205738 (10ZhaoFJx) 05Open→03In progress [21:34:36] 10Tool-wikinewsie, 06WikiNYC Tech Committee: Set up Wikinews Pulse tool as alpha release on Toolforge - https://phabricator.wikimedia.org/T427575#12205746 (10ZhaoFJx) p:05Triage→03High [21:34:57] 10Tool-wikinewsie, 06WikiNYC Tech Committee: Set up Wikinews Pulse tool as alpha release on Toolforge - https://phabricator.wikimedia.org/T427575#12205748 (10ZhaoFJx) [21:35:53] 10Tool-wikinewsie: Wikinewsie 'Mix' interface takes up too much space on mobile - https://phabricator.wikimedia.org/T434606#12205749 (10ZhaoFJx) a:03ZhaoFJx [21:35:56] 10Tool-wikinewsie: Wikinewsie 'Mix' interface takes up too much space on mobile - https://phabricator.wikimedia.org/T434606#12205751 (10ZhaoFJx) 05Open→03In progress [23:20:12] (03open) 10tekneos: Replace shell scripts with Ansible. #T434466 [toolforge-repos/wd-act] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-act/-/merge_requests/1 [23:23:04] (03approved) 10tekneos: Replace shell scripts with Ansible. #T434466 [toolforge-repos/wd-act] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-act/-/merge_requests/1 [23:23:22] (03merge) 10tekneos: Replace shell scripts with Ansible. #T434466 [toolforge-repos/wd-act] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-act/-/merge_requests/1 [23:31:05] 10Tool-wd-act: Add basic CI/CD scripts for deploying to Toolforge - https://phabricator.wikimedia.org/T434466#12206014 (10Tekneos) 05In progress→03Resolved Knocked this out in this merge request: https://gitlab.wikimedia.org/toolforge-repos/wd-act/-/merge_requests/1 After looking at this some, I decided... [23:31:49] 10Tool-wd-act: Add basic CI/CD scripts for deploying to Toolforge - https://phabricator.wikimedia.org/T434466#12206017 (10Tekneos) I forgot to post the link - here's the page about GitLab CI in wikimedia: https://www.mediawiki.org/wiki/GitLab/Workflows/CI [23:34:20] 10Tool-wd-act: Replace development web server with production-ready server - https://phabricator.wikimedia.org/T431040#12206027 (10Tekneos) 05Open→03Resolved a:03Tekneos This is done. The framework used by Toolforge to deliver this application as a web service - Cloud Native Buildpacks - includes gunic... [23:36:11] 10Tool-wd-act: Likely Dockerize the application - https://phabricator.wikimedia.org/T431038#12206042 (10Tekneos) 05Open→03Resolved a:03Tekneos This is done. Toolforge supports using Cloud Native Buildpacks, which automatically generate a Docker image for the project. So in other words, we don't have to...