[00:17:50] (03open) 10andrew: Rearrange many many helmfiles [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1370 [00:19:13] (03update) 10andrew: Bring component helmfiles into the modern age. [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1370 [00:19:31] (03update) 10andrew: Bring component helmfiles into the modern age. [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1370 [00:21:46] (03open) 10andrew: k8s v1.33.13, helm v4.1.4, helmfile v1.7.3, kindest/node v1.32.11 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/338 (https://phabricator.wikimedia.org/T433132) [01:55:37] 06cloud-services-team, 10Toolforge, 06tools-platform-team: VS-code using too many resources in bastion makes user session unstable - https://phabricator.wikimedia.org/T434573#12206283 (10Taiwania_Justo) Okay, after the problem occurred and read the pervious cases, I stopped the VS Code's remote ssh function.... [04:44:33] (03update) 10raymond-ndibe: auth: Allow specifying allowed urls for superusers [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/103 (owner: 10dcaro) [04:50:22] (03approved) 10raymond-ndibe: api-gateway: allow jobs-api to send logs to logs-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1365 (owner: 10dcaro) [04:50:23] (03update) 10raymond-ndibe: api-gateway: allow jobs-api to send logs to logs-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1365 (owner: 10dcaro) [04:50:25] (03update) 10raymond-ndibe: api-gateway: allow jobs-api to send logs to logs-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1365 (owner: 10dcaro) [05:00:46] (03update) 10raymond-ndibe: deployment: added certificate with client rights [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/365 (owner: 10dcaro) [05:01:29] (03update) 10raymond-ndibe: deployment: added certificate with client rights [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/365 (owner: 10dcaro) [05:04:04] (03update) 10raymond-ndibe: bump_version: move to trixie [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/116 (owner: 10dcaro) [05:04:05] (03approved) 10raymond-ndibe: bump_version: move to trixie [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/116 (owner: 10dcaro) [05:04:20] (03update) 10raymond-ndibe: bump_version: move to trixie [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/20 (owner: 10dcaro) [05:04:21] (03approved) 10raymond-ndibe: bump_version: move to trixie [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/20 (owner: 10dcaro) [05:04:34] (03update) 10raymond-ndibe: bump_version: move to trixie [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/71 (owner: 10dcaro) [05:04:43] (03approved) 10raymond-ndibe: bump_version: move to trixie [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/71 (owner: 10dcaro) [05:05:02] (03update) 10raymond-ndibe: bump_version: move to trixie [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/118 (owner: 10dcaro) [05:05:05] (03approved) 10raymond-ndibe: bump_version: move to trixie [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/118 (owner: 10dcaro) [05:05:29] (03approved) 10raymond-ndibe: bump_version: move to trixie + update script [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/173 (owner: 10dcaro) [05:05:30] (03update) 10raymond-ndibe: bump_version: move to trixie + update script [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/173 (owner: 10dcaro) [05:05:42] (03update) 10raymond-ndibe: bump_version: move to trixie [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/145 (owner: 10dcaro) [05:05:45] (03approved) 10raymond-ndibe: bump_version: move to trixie [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/145 (owner: 10dcaro) [05:16:25] (03update) 10raymond-ndibe: models: support publish option [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/180 (https://phabricator.wikimedia.org/T433355) [05:17:52] (03update) 10raymond-ndibe: models: support publish option [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/180 (https://phabricator.wikimedia.org/T433355) [05:33:16] 10Tool-findlinkfast: findlinkfast: Use wikidata labels instead of bare article titles - https://phabricator.wikimedia.org/T434628 (10Audiodude) 03NEW [05:42:55] (03update) 10raymond-ndibe: tests: refactor test helpers, fixtures and existing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/366 (https://phabricator.wikimedia.org/T434166) [06:08:15] 10Cloud-VPS (Debian Bullseye Deprecation), 10Wikimedia-Developer-Portal: Update Dev Portal staging to Debian Bookworm - https://phabricator.wikimedia.org/T429490#12206444 (10KBach) 05In progress→03Resolved I have now deleted the old instance - devportal-demo01. Closing as resolved. [06:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:33:41] 10Cloud-VPS, 06tools-infrastructure-team: cloudcephosd1043 drives are very very busy - https://phabricator.wikimedia.org/T434334#12206550 (10hashar) After `bdev_enable_discard` got rolled back ( https://gerrit.wikimedia.org/r/c/operations/puppet/+/1322878 ), the disk went well again on cloudcephosd1043 and the... [07:44:01] 10Tool-wmf-openapi-linter, 06MediaWiki-API-Platform-Team, 06Content-Platform-Team (Work In Progress), 05MW-1.47-notes (1.47.0-wmf.15; 2026-08-11), 07OKR-Work: Add parameter examples to TransformHandler.php - https://phabricator.wikimedia.org/T432228#12206589 (10OWresch-WMF) [07:44:17] 10Tool-wmf-openapi-linter, 06MediaWiki-API-Platform-Team, 05MW-1.47-notes (1.47.0-wmf.15; 2026-08-11), 07OKR-Work, 13Patch-For-Review: Fix top-level issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428147#12206595 (10OWresch-WMF) [07:45:44] 10Cloud-VPS, 06tools-infrastructure-team, 10Research-collaborations: Change mount propagation for /mnt/nfs in wikinav-bookworm.research-collaborations-api - https://phabricator.wikimedia.org/T433668#12206605 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi >>! In T433668#12184320, @fkaelin wrote: >... [08:13:59] 06cloud-services-team, 10Toolforge, 06tools-platform-team: VS-code using too many resources in bastion makes user session unstable - https://phabricator.wikimedia.org/T434573#12206709 (10dcaro) 05Open→03Resolved a:03dcaro [08:17:31] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06collaboration-services, 10Continuous-Integration-Infrastructure (Zuul upgrade): Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12206720 (10LSobanski) [08:18:12] 10VPS-project-Codesearch, 06collaboration-services, 13Patch-For-Review: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199#12206725 (10LSobanski) p:05High→03Medium [08:39:59] (03update) 10dcaro: auth: Allow specifying allowed urls for superusers [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/103 [08:43:50] (03update) 10dcaro: auth: Allow specifying allowed urls for superusers [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/103 [08:43:56] (03update) 10dcaro: auth: Allow specifying allowed urls for superusers [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/103 [08:44:15] (03update) 10dcaro: auth: Allow specifying allowed urls for superusers [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/103 [08:45:46] (03approved) 10dcaro: Add new alerts for tool maintainers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/66 (https://phabricator.wikimedia.org/T432863) (owner: 10fnegri) [08:45:47] (03update) 10dcaro: Add new alerts for tool maintainers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/66 (https://phabricator.wikimedia.org/T432863) (owner: 10fnegri) [08:55:21] 10Cloud-VPS, 06tools-infrastructure-team: Clean up decomissioned cloudvirt from openstack resource providers - https://phabricator.wikimedia.org/T434639 (10fgiunchedi) 03NEW [09:05:15] (03update) 10dcaro: tests: refactor test helpers, fixtures and existing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/366 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [09:08:25] (03update) 10dcaro: components-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1367 (https://phabricator.wikimedia.org/T433355) (owner: 10raymond-ndibe) [09:13:04] (03approved) 10dcaro: models: support publish option [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/180 (https://phabricator.wikimedia.org/T433355) (owner: 10raymond-ndibe) [09:13:12] (03update) 10dcaro: models: support publish option [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/180 (https://phabricator.wikimedia.org/T433355) (owner: 10raymond-ndibe) [09:56:54] (03merge) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/116 [09:59:24] (03open) 10dcaro: d/changelog: bump to 0.0.16 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/117 (https://phabricator.wikimedia.org/T405024 https://phabricator.wikimedia.org/T408783 https://phabricator.wikimedia.org/T422184 https://phabricator.wikimedia.org/T424368 https://phabricator.wikimedia.org/T434050) [10:01:54] 10Tool-lexeme-forms, 06translatewiki.net, 10LPL Projects (Ongoing maintenance), 10LPL sprints (S2 2026 August): l10n-bot cannot create GitLab merge requests due to gitlab-ssh migration - https://phabricator.wikimedia.org/T432838#12207040 (10abi_) >>! In T432838#12149796, @Nikerabbit wrote: > One idea is to... [10:02:46] (03update) 10fnegri: Add new alerts for tool maintainers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/66 (https://phabricator.wikimedia.org/T432863) [10:04:05] (03merge) 10fnegri: Add new alerts for tool maintainers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/66 (https://phabricator.wikimedia.org/T432863) [10:06:12] FIRING: [70x] ToolFrequentRestarts: Containers in tool-ameisenbot are restarting repeatedly - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolFrequentRestarts - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolFrequentRestarts [10:11:12] FIRING: [71x] ToolFrequentRestarts: Containers in tool-alchimista are restarting repeatedly - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolFrequentRestarts - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolFrequentRestarts [10:14:56] (03open) 10fnegri: maintaner-alerts: remove team=wmcs [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/67 (https://phabricator.wikimedia.org/T432863) [10:16:12] FIRING: [72x] ToolFrequentRestarts: Containers in tool-alchimista are restarting repeatedly - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolFrequentRestarts - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolFrequentRestarts [10:16:31] (03approved) 10taavi: maintaner-alerts: remove team=wmcs [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/67 (https://phabricator.wikimedia.org/T432863) (owner: 10fnegri) [10:18:54] (03open) 10mahveotm: [T401993] Add deployment descriptions to the CLI [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/90 (https://phabricator.wikimedia.org/T401993) [10:20:31] (03merge) 10fnegri: maintaner-alerts: remove team=wmcs [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/67 (https://phabricator.wikimedia.org/T432863) [10:23:42] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: Define alert conditions in Prometheus (if possible) - https://phabricator.wikimedia.org/T432863#12207111 (10fnegri) The alerts are working as expected: {F98198423} However, notifications are being sent to IRC (`#wikimedia-cloud-feed`) an... [10:26:12] FIRING: [72x] ToolFrequentRestarts: Containers in tool-alchimista are restarting repeatedly - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolFrequentRestarts - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolFrequentRestarts [10:26:18] FIRING: [5x] ToolWebserviceErrors: Webservice in tool-aw-gerrit-gitlab-bridge is returning errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolWebserviceErrors - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolWebserviceErrors [10:26:18] FIRING: [3x] ToolWebserviceDown: Webservice in tool-autodesc is not responding - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolWebserviceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolWebserviceDown [10:31:18] FIRING: [5x] ToolWebserviceErrors: Webservice in tool-aw-gerrit-gitlab-bridge is returning errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolWebserviceErrors - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolWebserviceErrors [10:31:18] FIRING: [4x] ToolWebserviceDown: Webservice in tool-autodesc is not responding - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolWebserviceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolWebserviceDown [10:31:40] (03open) 10fnegri: maintainer-alerts: disable alerts temporarily [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/68 (https://phabricator.wikimedia.org/T432863) [10:34:38] (03update) 10fnegri: maintainer-alerts: disable alerts temporarily [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/68 (https://phabricator.wikimedia.org/T432863) [10:36:18] FIRING: [5x] ToolWebserviceErrors: Webservice in tool-aw-gerrit-gitlab-bridge is returning errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolWebserviceErrors - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolWebserviceErrors [10:36:18] FIRING: [4x] ToolWebserviceDown: Webservice in tool-autodesc is not responding - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolWebserviceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolWebserviceDown [10:38:37] (03update) 10fnegri: maintainer-alerts: disable alerts temporarily [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/68 (https://phabricator.wikimedia.org/T432863) [10:38:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:42:32] (03update) 10fnegri: maintainer-alerts: disable alerts temporarily [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/68 (https://phabricator.wikimedia.org/T432863) [10:45:40] (03update) 10fnegri: maintainer-alerts: disable alerts temporarily [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/68 (https://phabricator.wikimedia.org/T432863) [10:48:31] (03approved) 10taavi: maintainer-alerts: disable alerts temporarily [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/68 (https://phabricator.wikimedia.org/T432863) (owner: 10fnegri) [10:48:33] (03approved) 10dcaro: maintainer-alerts: disable alerts temporarily [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/68 (https://phabricator.wikimedia.org/T432863) (owner: 10fnegri) [10:51:12] FIRING: [71x] ToolFrequentRestarts: Containers in tool-ameisenbot are restarting repeatedly - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolFrequentRestarts - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolFrequentRestarts [10:51:18] FIRING: [2x] ToolWebserviceDown: Webservice in tool-ftl is not responding - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolWebserviceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolWebserviceDown [10:51:33] (03update) 10dcaro: global: add log write endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/32 [10:51:52] 10Toolforge, 06tools-platform-team, 07OKR-Work: Set up a Toolforge “alerting service” that emails tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865#12207200 (10fnegri) [10:52:38] (03merge) 10fnegri: maintainer-alerts: disable alerts temporarily [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/68 (https://phabricator.wikimedia.org/T432863) [10:52:38] (03update) 10fnegri: maintainer-alerts: disable alerts temporarily [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/68 (https://phabricator.wikimedia.org/T432863) [10:56:18] FIRING: [4x] ToolWebserviceErrors: Webservice in tool-aw-gerrit-gitlab-bridge is returning errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolWebserviceErrors - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolWebserviceErrors [11:01:12] FIRING: [73x] ToolFrequentRestarts: Containers in tool-ameisenbot are restarting repeatedly - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolFrequentRestarts - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolFrequentRestarts [11:01:16] (03update) 10dcaro: global: add log write endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/32 [11:01:18] FIRING: [4x] ToolWebserviceErrors: Webservice in tool-aw-gerrit-gitlab-bridge is returning errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolWebserviceErrors - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolWebserviceErrors [11:01:18] FIRING: [2x] ToolWebserviceDown: Webservice in tool-ftl is not responding - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolWebserviceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolWebserviceDown [11:06:12] FIRING: [73x] ToolFrequentRestarts: Containers in tool-ameisenbot are restarting repeatedly - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolFrequentRestarts - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolFrequentRestarts [11:06:18] FIRING: [3x] ToolWebserviceDown: Webservice in tool-favicons is not responding - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolWebserviceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolWebserviceDown [11:11:12] RESOLVED: [72x] ToolFrequentRestarts: Containers in tool-ameisenbot are restarting repeatedly - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolFrequentRestarts - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolFrequentRestarts [11:11:18] RESOLVED: [3x] ToolWebserviceErrors: Webservice in tool-aw-gerrit-gitlab-bridge is returning errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolWebserviceErrors - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolWebserviceErrors [11:11:18] RESOLVED: [2x] ToolWebserviceDown: Webservice in tool-favicons is not responding - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolWebserviceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolWebserviceDown [11:50:39] 10Cloud-VPS, 06tools-infrastructure-team: Clean up decomissioned cloudvirt from openstack resource providers - https://phabricator.wikimedia.org/T434639#12207337 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi I did the following to clean up both stale resource providers and allocations: ` # nova-ma... [11:53:44] (03open) 10taavi: monitoring: Copy PrometheusReloadFailed to explicitely set team: wmcs [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/69 [11:53:58] 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance: [sample-php-buildpack-app] failing to run - https://phabricator.wikimedia.org/T434655 (10dcaro) 03NEW p:05Triage→03Medium [11:55:03] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli [11:55:09] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli [11:56:10] (03update) 10taavi: monitoring: Copy PrometheusReloadFailed to explicitely set team: wmcs [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/69 [11:58:36] (03update) 10fnegri: monitoring: Copy PrometheusReloadFailed to explicitely set team: wmcs [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/69 (owner: 10taavi) [11:58:46] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [clis] use trixie to bump versions - https://phabricator.wikimedia.org/T434050#12207353 (10dcaro) Related to {T404733} [11:58:50] (03update) 10taavi: monitoring: Copy PrometheusReloadFailed to explicitely set team: wmcs [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/69 [11:59:32] (03approved) 10fnegri: monitoring: Copy PrometheusReloadFailed to explicitely set team: wmcs [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/69 (owner: 10taavi) [12:01:03] (03update) 10taavi: monitoring: Copy PrometheusReloadFailed to explicitely set team: wmcs [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/69 [12:01:17] (03update) 10taavi: monitoring: Copy PrometheusReloadFailed to explicitely set team: wmcs [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/69 [12:02:45] (03merge) 10taavi: monitoring: Copy PrometheusReloadFailed to explicitely set team: wmcs [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/69 [12:03:09] (03PS3) 10David Caro: aptly: Stop updating pre-Trixie repositories [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1188832 (https://phabricator.wikimedia.org/T404733) (owner: 10Majavah) [12:04:33] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Update Toolforge client packages to build on Trixie only - https://phabricator.wikimedia.org/T404733#12207366 (10dcaro) [12:04:47] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Update Toolforge client packages to build on Trixie only - https://phabricator.wikimedia.org/T404733#12207369 (10dcaro) a:03dcaro [12:05:35] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Update Toolforge client packages to build on Trixie only - https://phabricator.wikimedia.org/T404733#12207374 (10dcaro) 05Open→03In progress [12:06:21] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Update Toolforge client packages to build on Trixie only - https://phabricator.wikimedia.org/T404733#12207375 (10dcaro) [12:06:26] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [clis] use trixie to bump versions - https://phabricator.wikimedia.org/T434050#12207376 (10dcaro) [12:08:07] (03CR) 10David Caro: [C:03+2] aptly: Stop updating pre-Trixie repositories [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1188832 (https://phabricator.wikimedia.org/T404733) (owner: 10Majavah) [12:11:22] (03Merged) 10jenkins-bot: aptly: Stop updating pre-Trixie repositories [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1188832 (https://phabricator.wikimedia.org/T404733) (owner: 10Majavah) [12:14:52] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Update Toolforge client packages to build on Trixie only - https://phabricator.wikimedia.org/T404733#12207402 (10dcaro) 05In progress→03Resolved [12:15:01] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli [12:15:07] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli [12:20:23] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli [12:20:29] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli [12:23:05] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli [12:23:11] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli [12:23:36] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli [12:24:19] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli [12:26:26] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli [12:27:16] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli [12:27:29] (03approved) 10dcaro: d/changelog: bump to 0.0.16 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/117 (https://phabricator.wikimedia.org/T405024 https://phabricator.wikimedia.org/T408783 https://phabricator.wikimedia.org/T422184 https://phabricator.wikimedia.org/T424368 https://phabricator.wikimedia.org/T434050) [12:27:41] (03merge) 10dcaro: d/changelog: bump to 0.0.16 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/117 (https://phabricator.wikimedia.org/T405024 https://phabricator.wikimedia.org/T408783 https://phabricator.wikimedia.org/T422184 https://phabricator.wikimedia.org/T424368 https://phabricator.wikimedia.org/T434050) [12:29:03] (03merge) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/20 [12:31:29] (03open) 10dcaro: d/changelog: bump to 1.49.5 [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/21 (https://phabricator.wikimedia.org/T408783 https://phabricator.wikimedia.org/T422184 https://phabricator.wikimedia.org/T434050) [12:32:31] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli [12:32:36] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [clis] use trixie to bump versions - https://phabricator.wikimedia.org/T434050#12207483 (10dcaro) [12:32:49] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli [12:40:05] (03PS1) 10David Caro: inventory: remove toolsbeta-bastion-6 [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1324703 [12:43:04] 10Toolforge, 06tools-platform-team: [lima-kilo] shows error when copying home dirs - https://phabricator.wikimedia.org/T433849#12207539 (10dcaro) 05Open→03Resolved [12:43:40] (03CR) 10Majavah: [C:03+1] inventory: remove toolsbeta-bastion-6 [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1324703 (owner: 10David Caro) [12:44:51] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-bastion-6 (T401804) [12:44:55] T401804: Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804 [12:45:11] (03CR) 10David Caro: [C:03+2] inventory: remove toolsbeta-bastion-6 [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1324703 (owner: 10David Caro) [12:45:45] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-bastion-6 (T401804) [12:46:03] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli [12:46:18] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli [12:48:56] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli [12:48:56] (03Merged) 10jenkins-bot: inventory: remove toolsbeta-bastion-6 [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1324703 (owner: 10David Caro) [12:49:04] (03approved) 10devjadiya: Complete German, Chinese, Portuguese, Spanish, and French interface translations [toolforge-repos/clip2commons] - 10https://gitlab.wikimedia.org/toolforge-repos/clip2commons/-/merge_requests/10 (https://phabricator.wikimedia.org/T434092) (owner: 10anushka10patel) [12:49:08] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli [12:49:33] (03merge) 10devjadiya: Complete German, Chinese, Portuguese, Spanish, and French interface translations [toolforge-repos/clip2commons] - 10https://gitlab.wikimedia.org/toolforge-repos/clip2commons/-/merge_requests/10 (https://phabricator.wikimedia.org/T434092) (owner: 10anushka10patel) [12:54:52] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli [13:05:11] 10Tool-clip2commons, 13Patch-For-Review: Complete remaining translations - https://phabricator.wikimedia.org/T434092#12207703 (10Dev) Completed and merged. The five remaining locale files (German, Chinese, Portuguese, Spanish, and French) were reviewed and merged in [[ https://gitlab.wikimedia.org/toolforge-r... [13:05:54] 10Tool-clip2commons, 13Patch-For-Review: Complete remaining translations - https://phabricator.wikimedia.org/T434092#12207707 (10Dev) 05Open→03Resolved [13:07:48] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli [13:13:31] 10Toolforge, 06tools-platform-team: [components-api] split cancel endpoint to avoid duplicated operation id - https://phabricator.wikimedia.org/T434499#12207739 (10Mahveotm) a:03Mahveotm [13:15:17] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli [13:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:28:57] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli [13:34:23] 06cloud-services-team, 10Toolforge, 06tools-platform-team: `toolforge jobs logs` is limited to 5000 lines - https://phabricator.wikimedia.org/T401553#12207867 (10dcaro) 05Open→03Resolved a:03dcaro This is fixed! 🎉 [13:35:52] 10Tool-clip2commons, 13Patch-For-Review: Remember previously selected license - https://phabricator.wikimedia.org/T434084#12207871 (10Dev) Thanks for the implementation. Before I proceed with the review, please update `remember-last-license` against the current `main`. The target branch has advanced since thi... [13:39:32] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.remove_instance for instance cloudinfra-idp-1 [13:40:19] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloudinfra-idp-1 [13:41:47] (03open) 10mahveotm: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) [13:45:44] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [components-api] split cancel endpoint to avoid duplicated operation id - https://phabricator.wikimedia.org/T434499#12207892 (10Mahveotm) 05Open→03In progress [13:49:00] 06cloud-services-team, 10Toolforge, 06tools-platform-team: Duplicated utils scripts - https://phabricator.wikimedia.org/T402949#12207905 (10fnegri) [13:49:14] 06cloud-services-team, 10Toolforge, 06tools-platform-team: Duplicated utils scripts - https://phabricator.wikimedia.org/T402949#12207910 (10fnegri) p:05Medium→03Triage [13:51:43] (03open) 10tlepage: Alloy relabeling [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1371 [13:52:02] 10Toolforge, 06tools-platform-team: Duplicated utils scripts - https://phabricator.wikimedia.org/T402949#12207916 (10aputhin) [13:53:36] (03update) 10raymond-ndibe: global: add log write endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/32 (owner: 10dcaro) [13:53:37] (03approved) 10raymond-ndibe: global: add log write endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/32 (owner: 10dcaro) [13:54:10] (03approved) 10dcaro: d/changelog: bump to 1.49.5 [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/21 (https://phabricator.wikimedia.org/T408783 https://phabricator.wikimedia.org/T422184 https://phabricator.wikimedia.org/T434050) [13:54:21] (03merge) 10dcaro: d/changelog: bump to 1.49.5 [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/21 (https://phabricator.wikimedia.org/T408783 https://phabricator.wikimedia.org/T422184 https://phabricator.wikimedia.org/T434050) [13:55:05] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [clis] use trixie to bump versions - https://phabricator.wikimedia.org/T434050#12207941 (10dcaro) [13:58:45] (03update) 10tlepage: Alloy relabeling [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1371 [13:59:45] (03update) 10tlepage: Alloy relabeling [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1371 [14:02:04] 10Toolforge, 06tools-platform-team: debian packaging: create common guidelines and workflow - https://phabricator.wikimedia.org/T212291#12207968 (10fnegri) We talked about this in the latest #tools-platform-team retro, @dcaro recently did some work in {T434050} but while the pipeline is now shared at https://g... [14:03:44] 10Toolforge, 06tools-platform-team: Duplicated utils scripts - https://phabricator.wikimedia.org/T402949#12207989 (10fnegri) [14:03:45] 10Toolforge, 06tools-platform-team: debian packaging: create common guidelines and workflow - https://phabricator.wikimedia.org/T212291#12207990 (10fnegri) [14:06:45] 10Toolforge, 06tools-platform-team: De-duplicate utils scripts across repositories and remove unused ones - https://phabricator.wikimedia.org/T402949#12207995 (10aputhin) [14:08:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:08:42] 10Toolforge, 06tools-platform-team: De-duplicate utils scripts across repositories and remove unused ones - https://phabricator.wikimedia.org/T402949#12208014 (10aputhin) p:05Triage→03Low [14:15:14] 10Toolforge, 06tools-platform-team: debian packaging: create common guidelines and workflow - https://phabricator.wikimedia.org/T212291#12208043 (10aputhin) 05Open→03Resolved Closing for now, as we see that enough progress has been made and, as mentioned above, the processes between Toolforge and SRE a... [14:16:41] 10Toolforge, 06tools-platform-team: Automate deployment of Toolforge CLIs - https://phabricator.wikimedia.org/T434676 (10aputhin) 03NEW [14:17:13] 10Toolforge, 06tools-platform-team: Automate deployment of Toolforge CLIs - https://phabricator.wikimedia.org/T434676#12208065 (10aputhin) Related to: T402949 [14:27:01] (03update) 10raymond-ndibe: tests: refactor test helpers, fixtures and existing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/366 (https://phabricator.wikimedia.org/T434166) [14:27:29] (03update) 10raymond-ndibe: tests: refactor test helpers, fixtures and existing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/366 (https://phabricator.wikimedia.org/T434166) [14:41:16] (03CR) 10Majavah: [C:03+2] Remove unneeded IO setting for default readwrite Diffusion repos [labs/striker] - 10https://gerrit.wikimedia.org/r/1315945 (https://phabricator.wikimedia.org/T409341) (owner: 10Aklapper) [14:42:32] 10Toolforge, 06tools-platform-team: [functional-tests] Review the current tests to see what can be done to speed them up and how much effort will it take - https://phabricator.wikimedia.org/T434578#12208165 (10dcaro) [14:43:45] 10Toolforge, 06tools-platform-team: [functional-tests] Review the current tests to see what can be done to speed them up and how much effort will it take - https://phabricator.wikimedia.org/T434578#12208167 (10dcaro) p:05Triage→03Medium [14:44:13] (03Merged) 10jenkins-bot: Remove unneeded IO setting for default readwrite Diffusion repos [labs/striker] - 10https://gerrit.wikimedia.org/r/1315945 (https://phabricator.wikimedia.org/T409341) (owner: 10Aklapper) [14:45:12] 10Toolforge, 06tools-platform-team: [functional-tests] Review the current tests to see what can be done to speed them up and how much effort will it take - https://phabricator.wikimedia.org/T434578#12208178 (10dcaro) [14:46:24] 06cloud-services-team, 10Data-Services, 06tools-infrastructure-team, 06Collaboration-Services, 10Continuous-Integration-Infrastructure (Zuul upgrade): Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12208182 (10dcaro) [14:51:16] 10Toolforge, 06tools-platform-team: [components-api] add the option to the tool configuration bulid section to allow passing built time envvars - https://phabricator.wikimedia.org/T434496#12208199 (10dcaro) p:05Triage→03Medium [14:52:57] 10Toolforge, 06tools-platform-team: [toolsdb] Upgrade to MariaDB 10.11 - https://phabricator.wikimedia.org/T434679 (10fnegri) 03NEW [14:55:46] (03merge) 10dcaro: bump_version: move to trixie + update script [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/173 [14:56:14] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [14:56:18] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli [14:57:56] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [14:58:01] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli [14:59:14] (03open) 10dcaro: d/changelog: bump to 16.2.1 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/175 (https://phabricator.wikimedia.org/T399080 https://phabricator.wikimedia.org/T433405 https://phabricator.wikimedia.org/T433821 https://phabricator.wikimedia.org/T434050) [15:01:07] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [15:02:25] 10Striker, 06tools-platform-team, 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎): Striker: Always set all IO to none and hidden for internal URIs when creating Diffusion repositories - https://phabricator.wikimedia.org/T409341#12208281 (10Aklapper) [15:02:54] 10Striker, 06tools-platform-team, 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎): Striker: Always set all IO to none and hidden for internal URIs when creating Diffusion repositories - https://phabricator.wikimedia.org/T409341#12208284 (10Aklapper) [Hopefully] last patch merged; to be v... [15:05:24] (03update) 10raymond-ndibe: tests: refactor test helpers, fixtures and existing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/366 (https://phabricator.wikimedia.org/T434166) [15:09:13] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [15:17:02] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [15:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:21:41] (03update) 10raymond-ndibe: tests: add behavior tests for previously untested code branches [repos/cloud/toolforge/jobs-api] (refactor_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/362 (https://phabricator.wikimedia.org/T434166) [15:21:55] (03update) 10raymond-ndibe: models: move cmd field out of CommonJob into job-type models [repos/cloud/toolforge/jobs-api] (increase_test_coverage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/364 (https://phabricator.wikimedia.org/T434166) [15:22:12] (03update) 10raymond-ndibe: refactor: rename CommonJob to CommonOptions [repos/cloud/toolforge/jobs-api] (move_cmd_out_of_common_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/367 (https://phabricator.wikimedia.org/T434166) [15:22:44] (03update) 10raymond-ndibe: refactor: move observability fields out of CommonOptions into ObservabilityOptions [repos/cloud/toolforge/jobs-api] (rename_common_job_to_common_options) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) [15:22:46] (03update) 10raymond-ndibe: models: support webservice job-type [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [15:24:26] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.33.13 (T433131) [15:24:31] T433131: Upgrade toolsbeta to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433131 [15:26:16] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [15:28:21] (03approved) 10dcaro: d/changelog: bump to 16.2.1 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/175 (https://phabricator.wikimedia.org/T399080 https://phabricator.wikimedia.org/T433405 https://phabricator.wikimedia.org/T433821 https://phabricator.wikimedia.org/T434050) [15:28:30] (03merge) 10dcaro: d/changelog: bump to 16.2.1 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/175 (https://phabricator.wikimedia.org/T399080 https://phabricator.wikimedia.org/T433405 https://phabricator.wikimedia.org/T433821 https://phabricator.wikimedia.org/T434050) [15:32:10] 10Toolforge, 06tools-platform-team: [clis] use trixie to bump versions - https://phabricator.wikimedia.org/T434050#12208400 (10dcaro) [15:35:11] 10Toolforge, 06tools-platform-team: [jobs-cli] broken pipe error when using `toolforge jobs logs | head` - https://phabricator.wikimedia.org/T433405#12208430 (10dcaro) 05Open→03Resolved a:03dcaro [15:37:01] (03merge) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/145 [15:38:17] (03open) 10dcaro: d/changelog: bump to 0.0.27 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/146 (https://phabricator.wikimedia.org/T434050) [15:39:46] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-cli [15:40:06] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli [15:40:47] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade to 1.33.13 (T433131) [15:40:50] T433131: Upgrade toolsbeta to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433131 [15:49:46] (03PS1) 10David Caro: wmcs.toolforge.component.deploy: add some logs [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1324758 [15:51:06] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-cli [15:52:35] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli [15:52:50] 10Striker, 06tools-platform-team: 500 internal error when accessing toolsadmin ssh-keys - https://phabricator.wikimedia.org/T434684#12208501 (10dcaro) [15:53:31] (03CR) 10CI reject: [V:04-1] wmcs.toolforge.component.deploy: add some logs [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1324758 (owner: 10David Caro) [15:54:43] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-cli [15:56:46] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli [15:58:05] (03approved) 10dcaro: d/changelog: bump to 0.0.27 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/146 (https://phabricator.wikimedia.org/T434050) [15:58:13] (03merge) 10dcaro: d/changelog: bump to 0.0.27 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/146 (https://phabricator.wikimedia.org/T434050) [16:00:09] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [clis] use trixie to bump versions - https://phabricator.wikimedia.org/T434050#12208555 (10dcaro) [16:01:50] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [clis] use trixie to bump versions - https://phabricator.wikimedia.org/T434050#12208567 (10dcaro) 05Open→03In progress [16:02:04] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) [16:02:16] (03merge) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/71 [16:07:56] (03open) 10dcaro: d/changelog: bump to 0.3.8 [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/72 (https://phabricator.wikimedia.org/T408783 https://phabricator.wikimedia.org/T434050) [16:08:38] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli [16:09:25] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-cli [16:10:44] 10Striker, 06tools-platform-team: 500 internal error when accessing toolsadmin ssh-keys - https://phabricator.wikimedia.org/T434684#12208615 (10PacmanD) FWIW I'm seeing 500 on this page about half of the time; I can constantly repro. Request ID: 9a2f272610164a4b9d76615250ddd9a8 [16:20:10] (03update) 10dcaro: d/changelog: bump to 0.3.8 [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/72 (https://phabricator.wikimedia.org/T408783 https://phabricator.wikimedia.org/T434050) [16:21:10] (03update) 10dcaro: d/changelog: bump to 0.3.9 [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/72 (https://phabricator.wikimedia.org/T408783 https://phabricator.wikimedia.org/T434050) [16:21:53] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli [16:22:25] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-cli [16:23:56] (03PS2) 10David Caro: wmcs.toolforge.component.deploy: show tests already running [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1324758 [16:26:44] (03CR) 10CI reject: [V:04-1] wmcs.toolforge.component.deploy: show tests already running [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1324758 (owner: 10David Caro) [16:27:36] 10Toolforge, 06tools-platform-team, 07OKR-Work: Define alert conditions in Prometheus (if possible) - https://phabricator.wikimedia.org/T432863#12208706 (10fnegri) From a discussion in IRC with @taavi: `lang=irc 12:51 but yeah, you either need completely separate prometheus+AM instances, or you need... [16:27:55] (03approved) 10dcaro: tests: refactor test helpers, fixtures and existing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/366 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [16:28:01] (03update) 10dcaro: tests: refactor test helpers, fixtures and existing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/366 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [16:29:59] !log andrew@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) [16:32:48] (03PS3) 10David Caro: wmcs.toolforge.component.deploy: show tests already running [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1324758 [16:35:42] (03CR) 10CI reject: [V:04-1] wmcs.toolforge.component.deploy: show tests already running [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1324758 (owner: 10David Caro) [16:36:57] 10Striker, 06tools-platform-team: 500 internal error when accessing toolsadmin ssh-keys - https://phabricator.wikimedia.org/T434684#12208804 (10bd808) >>! In T434684#12208615, @PacmanD wrote: > FWIW I'm seeing 500 on this page about half of the time; I can constantly repro. > > Request ID: 9a2f272610164a4b9d7... [16:37:36] (03PS4) 10David Caro: wmcs.toolforge.component.deploy: show tests already running [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1324758 [16:38:05] 10Tool-wsindex, 10Wikisource Reader App (Android), 07Arabic-Sites: Add Arabic language to Wikisource reader app - https://phabricator.wikimedia.org/T409407#12208817 (10Bodhisattwa) [16:39:25] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) [16:39:25] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) [16:39:41] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) [16:41:51] 10Striker, 06tools-platform-team: 500 internal error when accessing toolsadmin ssh-keys - https://phabricator.wikimedia.org/T434684#12208876 (10bd808) > "This is probably temporary and should be fixed soon." Those are static words I put on [[https://gerrit.wikimedia.org/r/plugins/gitiles/labs/striker/+/8d38b7... [16:46:20] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) [16:48:53] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) [16:55:08] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) [16:56:10] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) [16:56:10] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) [16:59:00] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.33.13 (TT433131) [16:59:00] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.33.13 (TT433131) [17:01:24] 10Striker, 06tools-platform-team: 500 internal error when accessing toolsadmin ssh-keys - https://phabricator.wikimedia.org/T434684#12209019 (10PacmanD) If I'm able to load this page (and others, without an active developer account), it could mean that some servers behind load balancer running with a different... [17:17:30] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.33.13 to 1.33.13 (TT433131) [17:17:30] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.33.13 to 1.33.13 (TT433131) [17:36:43] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) [17:37:46] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) [17:39:02] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) [17:40:05] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) [17:46:04] 10Striker, 06tools-platform-team: 500 internal error when accessing toolsadmin ssh-keys - https://phabricator.wikimedia.org/T434684#12209145 (10bd808) Poking around using a live `poetry run python3 manage.py shell` session inside a production container: `lang=python >>> from striker.labsauth.models import Labs... [17:55:26] (03update) 10raymond-ndibe: tests: refactor test helpers, fixtures and existing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/366 (https://phabricator.wikimedia.org/T434166) [17:55:44] (03update) 10anushka10patel: Remember last-selected license across uploads [toolforge-repos/clip2commons] - 10https://gitlab.wikimedia.org/toolforge-repos/clip2commons/-/merge_requests/11 (https://phabricator.wikimedia.org/T434084) [17:57:24] (03update) 10raymond-ndibe: auth: Allow specifying allowed urls for superusers [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/103 (owner: 10dcaro) [17:57:44] (03approved) 10raymond-ndibe: auth: Allow specifying allowed urls for superusers [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/103 (owner: 10dcaro) [18:07:20] (03approved) 10raymond-ndibe: tests: refactor test helpers, fixtures and existing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/366 (https://phabricator.wikimedia.org/T434166) [18:07:30] (03merge) 10raymond-ndibe: tests: refactor test helpers, fixtures and existing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/366 (https://phabricator.wikimedia.org/T434166) [18:07:30] (03update) 10raymond-ndibe: tests: add behavior tests for previously untested code branches [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/362 (https://phabricator.wikimedia.org/T434166) [18:10:55] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.558-20260812180740-04ee66e8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1372 (https://phabricator.wikimedia.org/T434166) [18:11:03] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.558-20260812180740-04ee66e8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1372 (https://phabricator.wikimedia.org/T434166) [18:15:33] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [18:15:41] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [18:41:04] (03update) 10raymond-ndibe: components-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1367 (https://phabricator.wikimedia.org/T433355) [18:43:14] FIRING: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [18:43:24] FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node toolsbeta-test-k8s-control-10 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [18:52:43] (03update) 10raymond-ndibe: components-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1367 (https://phabricator.wikimedia.org/T433355) [18:53:45] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster [18:54:02] (03update) 10raymond-ndibe: components-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1367 (https://phabricator.wikimedia.org/T433355) [18:54:10] (03update) 10raymond-ndibe: components-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1367 (https://phabricator.wikimedia.org/T433355) [18:54:11] (03approved) 10raymond-ndibe: components-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1367 (https://phabricator.wikimedia.org/T433355) [19:02:01] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12209331 (10VRiley-WMF) Hey @fgiunchedi Yes, that would work for me! Would monday next week work for you? [19:04:13] (03open) 10mahveotm: [T398425] Treat invalid YAML as a user error [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/91 (https://phabricator.wikimedia.org/T398425) [19:05:34] !log andrew@cloudcumin1001 toolsbeta Added a new k8s control toolsbeta-test-k8s-control-13.toolsbeta.eqiad1.wikimedia.cloud to the cluster [19:05:34] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster [19:07:32] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) [19:08:35] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) [19:10:51] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) [19:11:52] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [components-cli] Invalid YAML file error should not encourage reporting the issue to admins - https://phabricator.wikimedia.org/T398425#12209365 (10Mahveotm) a:03Mahveotm [19:12:03] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [components-cli] Invalid YAML file error should not encourage reporting the issue to admins - https://phabricator.wikimedia.org/T398425#12209366 (10Mahveotm) 05Open→03In progress [19:17:44] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) [19:23:15] (03update) 10countcount: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/29 (owner: 10renovatebot) [19:27:04] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [components-api] Deployment token should not be a GET param - https://phabricator.wikimedia.org/T397712#12209439 (10Mahveotm) [19:34:44] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) [19:41:09] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) [19:42:20] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) [19:43:14] FIRING: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [19:50:14] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) [19:50:23] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) [19:50:23] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) [19:50:41] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) [19:50:41] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) [19:53:14] FIRING: [3x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [19:54:17] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) [19:54:17] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) [19:54:31] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-13 from 1.32.13 to 1.33.13 (TT433131) [19:54:31] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-13 from 1.32.13 to 1.33.13 (TT433131) [19:56:26] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) [19:56:26] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) [19:56:35] 10Tool-linkdata: Make the HTTP client a first-class citizen - https://phabricator.wikimedia.org/T434717 (10Audiodude) 03NEW [20:06:20] (03open) 10mahveotm: [T397863] Show image name in build details [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/147 (https://phabricator.wikimedia.org/T397863) [20:09:05] 10Tool-wsindex: Weekly and Monthly stat in Analytics - https://phabricator.wikimedia.org/T431184#12209648 (10Bodhisattwa) [20:10:15] 10Data-Services, 06tools-platform-team: Expose the gb_block_email column of the centralauth_p.globalblocks table in the wiki replicas view - https://phabricator.wikimedia.org/T434044#12209652 (10bd808) [20:10:18] 10Data-Services, 06tools-platform-team: Expose the gb_block_email column of the centralauth_p.globalblocks table in the wiki replicas view - https://phabricator.wikimedia.org/T434044#12209656 (10bd808) Added in {696a3fccf8baafae75b27ef9dd5ad084fbc9b464}. @Dreamy_Jazz, can you help find out what the consensus... [20:11:05] 10Tool-wsindex: Add yearly requests in Analytics - https://phabricator.wikimedia.org/T434721 (10Bodhisattwa) 03NEW [20:11:49] 10Data-Services, 06tools-platform-team: Expose the gb_block_email column of the centralauth_p.globalblocks table in the wiki replicas view - https://phabricator.wikimedia.org/T434044#12209683 (10Dreamy_Jazz) It will be fine to add this field to the view [20:13:14] FIRING: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [20:16:29] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/29 [20:16:31] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/29 [20:22:27] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) [20:22:31] T433131: Upgrade toolsbeta to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433131 [20:22:36] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) [20:22:42] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) [20:22:44] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) [20:22:51] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) [20:23:57] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) [20:26:28] 10Data-Services, 06tools-platform-team, 10VPS-Projects: NFS shared storage for Cloud VPS project `globaleducation` - https://phabricator.wikimedia.org/T434723 (10Gabinaluz) 03NEW [20:31:15] (03PS4) 10BryanDavis: Upgrade dev wiki to 1.43 LTS and drop OpenStackManager [labs/striker] - 10https://gerrit.wikimedia.org/r/1295943 (https://phabricator.wikimedia.org/T427824) (owner: 10Arendpieter) [20:31:15] (03PS1) 10BryanDavis: labsauth: Log a LabsUser's name and dn when LdapUser fetch fails [labs/striker] - 10https://gerrit.wikimedia.org/r/1324800 (https://phabricator.wikimedia.org/T434684) [20:31:16] 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: Upgrade toolsbeta to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433131#12209742 (10Andrew) this is not going great. 1) hang during upgrade of -control-10, ctrl-c but of course the cookbook won't resume. Everything looks up to... [20:37:55] 10Tool-wsindex: Add yearly requests in Analytics - https://phabricator.wikimedia.org/T434721#12209783 (10Bodhisattwa) 05Open→03Resolved a:03Bodhisattwa It;s now live on https://wsindex.toolforge.org/analytics/ [20:46:45] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) [20:46:50] T433131: Upgrade toolsbeta to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433131 [20:47:51] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) [20:48:00] 10Tool-wsindex, 10Wikisource Reader App (Android), 07Esperanto-Sites: Add Esperanto to Wsindex API and Wikisource reader app - https://phabricator.wikimedia.org/T434447#12209826 (10Bodhisattwa) [20:48:54] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) [20:52:22] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) [20:52:26] T433131: Upgrade toolsbeta to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433131 [20:52:48] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) [20:54:14] (03PS5) 10BryanDavis: dev: Upgrade MediaWiki to 1.43 LTS and drop OpenStackManager [labs/striker] - 10https://gerrit.wikimedia.org/r/1295943 (https://phabricator.wikimedia.org/T427824) (owner: 10Arendpieter) [20:54:14] (03PS2) 10BryanDavis: labsauth: Log a LabsUser's name and dn when LdapUser fetch fails [labs/striker] - 10https://gerrit.wikimedia.org/r/1324800 (https://phabricator.wikimedia.org/T434684) [20:54:55] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T433131) [20:56:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:00:57] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T433131) [21:01:03] T433131: Upgrade toolsbeta to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433131 [21:01:03] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T433131) [21:01:06] 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: Upgrade toolsbeta to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433131#12209885 (10Andrew) [] documents say to run wmcs.toolforge.k8s.worker.upgrade_ingresses which doesn't exist anymore, I assume because we don't have special... [21:01:14] (03CR) 10BryanDavis: [C:03+2] dev: Upgrade MediaWiki to 1.43 LTS and drop OpenStackManager (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/1295943 (https://phabricator.wikimedia.org/T427824) (owner: 10Arendpieter) [21:01:38] FIRING: [4x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:04:06] (03Merged) 10jenkins-bot: dev: Upgrade MediaWiki to 1.43 LTS and drop OpenStackManager [labs/striker] - 10https://gerrit.wikimedia.org/r/1295943 (https://phabricator.wikimedia.org/T427824) (owner: 10Arendpieter) [21:09:31] 10Tool-itwiki: BotCancellazioni: rollback "cllimit=500" workaround after T433922 has been resolved - https://phabricator.wikimedia.org/T434598#12209896 (10Umherirrender) The workround is also a performance optimization, as less requests are needed to get the `batchcomplete` key for the continue, you can also use... [21:15:32] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/29 [21:29:16] 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: Upgrade toolsbeta to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433131#12209998 (10Andrew) a:03Andrew Despite all that, everything is now upgraded and tests are passing. control-10 remains a puzzle. [21:46:12] 10Toolforge, 06tools-platform-team: [builds-api] TOOL_DATA_DIR not available during `toolforge build start` execution - https://phabricator.wikimedia.org/T434369#12210031 (10bd808) >>! In T434369#12200143, @Gouvernathor wrote: > Oh. Hadn't thought of that nuance. Well yes, in the case of static websites you ne... [21:51:32] 06cloud-services-team, 10Striker, 06tools-platform-team: Striker dev MediaWiki image fails to build: 1.41 is EOL and Composer now blocks advisory-affected dependencies - https://phabricator.wikimedia.org/T427824#12210037 (10bd808) 05Open→03Resolved Thanks for the patch @Arendpieter [21:54:56] 10Toolforge, 06tools-platform-team: [builds-api] TOOL_DATA_DIR not available during `toolforge build start` execution - https://phabricator.wikimedia.org/T434369#12210050 (10bd808) >>! In T434369#12200214, @Gouvernathor wrote: > Though, before introducing a revolution in how image building works, the only thin... [21:56:48] 10Toolforge, 06tools-platform-team: [builds-api,envvars-admission] add TOOL_NAME environment variable - https://phabricator.wikimedia.org/T434431#12210059 (10bd808) Dup of {T419601} [21:58:20] 10Toolforge, 06tools-platform-team: [builds-api,envvars-admission] add TOOL_NAME environment variable - https://phabricator.wikimedia.org/T434431#12210062 (10JJMC89) →14Duplicate dup:03T419601 [21:58:21] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [envars] Provide a $TOOL or $TOOL_NAME default envvar - https://phabricator.wikimedia.org/T419601#12210064 (10JJMC89) [22:52:17] 06cloud-services-team, 10Tool-paulina, 10Toolforge, 06tools-platform-team: Automated traffic affects tool performance - https://phabricator.wikimedia.org/T432878#12210183 (10bd808) >>! In T432878#12204997, @Pepe_piton wrote: > I sometimes detect very strange behavior, like a big delay in loading the /stati... [22:56:46] (03update) 10raymond-ndibe: support --webservice option [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T428898) [23:03:48] 10Tool-itwiki, 07Upstream: Aggiungere un'etichetta alle modifiche di JWB - https://phabricator.wikimedia.org/T363923#12210198 (10Titore) 05Open→03Resolved a:03Titore Supporto aggiunto l'anno scorso a seguito di pull request! * [[https://en.wikipedia.org/wiki/Special:Diff/1288466846]] * [[https://gith... [23:04:35] 06cloud-services-team, 10Tool-paulina, 10Toolforge, 06tools-platform-team: Automated traffic affects tool performance - https://phabricator.wikimedia.org/T432878#12210202 (10bd808) >>! In T432878#12204997, @Pepe_piton wrote: >>>! In T432878#12147778, @BLiviero-WMF wrote: >> If it's visible to you, this pha... [23:16:32] 06cloud-services-team, 10Tool-paulina, 10Toolforge, 06tools-platform-team: Automated traffic affects tool performance - https://phabricator.wikimedia.org/T432878#12210209 (10bd808) >>! In T432878#12204997, @Pepe_piton wrote: >>>! In T432878#12147898, @bd808 wrote: >> The quickest protection that a web tool... [23:26:37] 06cloud-services-team, 10Tool-paulina, 10Toolforge, 06tools-platform-team: Automated traffic affects tool performance - https://phabricator.wikimedia.org/T432878#12210217 (10bd808) >>! In T432878#12210183, @bd808 wrote: >>>! In T432878#12204997, @Pepe_piton wrote: >> I sometimes detect very strange behavio... [23:35:14] 10Tool-itwiki: Verifica automatica dei requisiti per la Commissione arbitrale di Itwiki - https://phabricator.wikimedia.org/T363826#12210223 (10Titore) 05Open→03Resolved a:03Titore Nel frattempo @Sakretsu ha aggiornato l'accessorio Requisiti di voto aggiungendo il supporto ai requisiti ArbCom via itwik... [23:46:11] 10Cloud-VPS (Debian Bullseye Deprecation), 10Wikimedia-Developer-Portal: Update Dev Portal staging to Debian Bookworm - https://phabricator.wikimedia.org/T429490#12210248 (10bd808) >>! In T429490#12187102, @KBach wrote: >>>! In T429490#12187083, @taavi wrote: >>> I realized that the instance should use Boo...