[00:02:21] (03approved) 10raymond-ndibe: values.yaml: set enable_storage to true [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/289 (https://phabricator.wikimedia.org/T423870) [00:02:34] (03merge) 10raymond-ndibe: values.yaml: set enable_storage to true [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/289 (https://phabricator.wikimedia.org/T423870) [00:04:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [00:56:55] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.484-20260421000258-fe5dffd0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1223 (https://phabricator.wikimedia.org/T423870) [00:57:04] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.484-20260421000258-fe5dffd0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1223 (https://phabricator.wikimedia.org/T423870) [00:57:37] (03update) 10raymond-ndibe: cli.py: stop resolving dynamic fields in the cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/151 (https://phabricator.wikimedia.org/T423828) [01:01:00] (03update) 10raymond-ndibe: tests: fix wrong test data [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/290 (https://phabricator.wikimedia.org/T423880) [01:04:16] (03update) 10raymond-ndibe: Draft: fix jobs load bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/287 (https://phabricator.wikimedia.org/T423544) [01:04:54] (03update) 10raymond-ndibe: Draft: fix jobs load bug [repos/cloud/toolforge/jobs-api] (fix_data_handling_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/287 (https://phabricator.wikimedia.org/T423544) [01:13:04] (03update) 10raymond-ndibe: Draft: fix jobs load bug [repos/cloud/toolforge/jobs-api] (fix_data_handling_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/287 (https://phabricator.wikimedia.org/T423544) [01:13:52] (03update) 10raymond-ndibe: Draft: fix jobs load bug [repos/cloud/toolforge/jobs-api] (fix_data_handling_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/287 (https://phabricator.wikimedia.org/T423544) [01:17:27] (03update) 10raymond-ndibe: core.py: skip one-off jobs when updating storage [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/288 (https://phabricator.wikimedia.org/T423544) [01:17:52] (03update) 10raymond-ndibe: core.py: skip one-off jobs when updating storage [repos/cloud/toolforge/jobs-api] (fix_data_handling_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/288 (https://phabricator.wikimedia.org/T423544) [01:17:59] (03update) 10raymond-ndibe: core.py: skip one-off jobs when updating storage [repos/cloud/toolforge/jobs-api] (fix_data_handling_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/288 (https://phabricator.wikimedia.org/T423544) [01:19:07] (03update) 10raymond-ndibe: Draft: fix jobs load bug [repos/cloud/toolforge/jobs-api] (fix_oneoff_storage_error) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/287 (https://phabricator.wikimedia.org/T423544) [01:19:20] (03update) 10raymond-ndibe: Draft: fix jobs load bug [repos/cloud/toolforge/jobs-api] (fix_oneoff_storage_error) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/287 (https://phabricator.wikimedia.org/T423544) [01:19:28] (03update) 10raymond-ndibe: Draft: fix jobs load bug [repos/cloud/toolforge/jobs-api] (fix_oneoff_storage_error) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/287 (https://phabricator.wikimedia.org/T423544) [02:06:54] (03open) 10andrew: trove: update tests to use the new db engines [repos/cloud/cloud-vps/tf-infra-test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tf-infra-test/-/merge_requests/7 [02:55:20] (03update) 10raymond-ndibe: cli.py: stop resolving dynamic fields in the cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/151 (https://phabricator.wikimedia.org/T423828) [02:56:22] (03update) 10raymond-ndibe: Draft: fix jobs load bug [repos/cloud/toolforge/jobs-api] (fix_oneoff_storage_error) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/287 (https://phabricator.wikimedia.org/T423544) [03:04:20] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 13Patch-For-Review: OpenAPI linting: Add missing OpenAPI spec elements to Response Components - https://phabricator.wikimedia.org/T422739#11842352 (10HCoplin-WMF) 05Open→03Invalid It seems like this was covered in other tasks. M... [03:15:40] 10Tool-wikimedia-attribution: Document referrer recommendations for Participation CTA - https://phabricator.wikimedia.org/T423567#11842383 (10Khantstop) Thank you @Sarai-WMF, I responded in the doc with suggested language but also looped in my colleagues who might have a better idea of how to frame the rec. [04:41:30] (03update) 10raymond-ndibe: Draft: fix jobs load bug [repos/cloud/toolforge/jobs-api] (fix_oneoff_storage_error) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/287 (https://phabricator.wikimedia.org/T423544) [04:41:59] (03update) 10raymond-ndibe: Draft: fix jobs load bug [repos/cloud/toolforge/jobs-api] (fix_oneoff_storage_error) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/287 (https://phabricator.wikimedia.org/T423544) [04:47:14] (03update) 10raymond-ndibe: fix jobs load bug [repos/cloud/toolforge/jobs-api] (fix_oneoff_storage_error) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/287 (https://phabricator.wikimedia.org/T423544) [04:48:55] (03update) 10raymond-ndibe: Undeploy ingress-nginx [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1222 (https://phabricator.wikimedia.org/T392356) (owner: 10taavi) [04:48:57] (03approved) 10raymond-ndibe: Undeploy ingress-nginx [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1222 (https://phabricator.wikimedia.org/T392356) (owner: 10taavi) [05:32:22] FIRING: HAProxyBackendUnavailable: HAProxy service wikireplica-db-web-s6 backend clouddb1015.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [05:33:22] FIRING: [2x] HAProxyWikiReplicaSectionUnavailable: Wiki replica section s6 has no available servers on cloudlb1001:9900 - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyWikiReplicaSectionUnavailable [05:36:55] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#11842562 (10Marostegui) >>! In T409557#11833972, @fnegri wrote: >> I can work on it on Tuesday. > > Announcement sent: https://lists.wikimedia.... [06:02:22] RESOLVED: HAProxyBackendUnavailable: HAProxy service wikireplica-db-web-s6 backend clouddb1015.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [06:03:22] RESOLVED: [2x] HAProxyWikiReplicaSectionUnavailable: Wiki replica section s6 has no available servers on cloudlb1001:9900 - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyWikiReplicaSectionUnavailable [07:15:29] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#11842672 (10Marostegui) [08:00:34] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#11842785 (10fnegri) Downtime for s6 was limited to less than 30 mins, from 05:30 UTC to 05:58 UTC: `lang=shell-session root@cloudlb1001:~# jour... [08:06:19] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#11842807 (10fnegri) [08:14:09] (03CR) 10Muehlenhoff: [C:03+1] Add fake private secrets for discovery2026 PKI intermediate [labs/private] - 10https://gerrit.wikimedia.org/r/1275481 (https://phabricator.wikimedia.org/T420993) (owner: 10Elukey) [08:15:05] (03approved) 10fnegri: trove: update tests to use the new db engines [repos/cloud/cloud-vps/tf-infra-test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tf-infra-test/-/merge_requests/7 (owner: 10andrew) [08:19:46] (03CR) 10Elukey: [V:03+2 C:03+2] Add fake private secrets for discovery2026 PKI intermediate [labs/private] - 10https://gerrit.wikimedia.org/r/1275481 (https://phabricator.wikimedia.org/T420993) (owner: 10Elukey) [08:27:31] 06cloud-services-team, 10Cloud-VPS: [tofu-cloudvps] Add support for importing legacy cloudvps_puppet_prefix objects - https://phabricator.wikimedia.org/T423970#11842873 (10taavi) As far as I can tell, the provider already has code for this. Did you try that and find it not working, or was this task filed on th... [08:32:28] 06tools-infrastructure-team, 10Datasets-General-or-Unknown, 06SRE: Move internal dumps NFS clients to clouddumps1001 - https://phabricator.wikimedia.org/T416677#11842883 (10taavi) 05Open→03Declined This does not seem relevant after moving dumps HTTPS behind LVS. [08:50:17] 06cloud-services-team, 10Quarry: The imagelinks replica in hewiki is not available - https://phabricator.wikimedia.org/T423998 (10IKhitron) 03NEW [08:52:25] 06cloud-services-team, 10Quarry, 07Regression: The imagelinks replica in hewiki is not available - https://phabricator.wikimedia.org/T423998#11843036 (10IKhitron) [08:59:16] 06cloud-services-team, 10Quarry, 07Regression: The imagelinks replica in hewiki is not available - https://phabricator.wikimedia.org/T423998#11843046 (10jeremyb) [08:59:23] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Engineering-Radar, 06DBA: Re-run maintainviews on all clouddb* and an-redacteddb1001.eqiad.wmnet - https://phabricator.wikimedia.org/T422459#11843047 (10jeremyb) [09:20:57] 06cloud-services-team, 10Data-Services, 07Regression: The imagelinks replica in hewiki is not available - https://phabricator.wikimedia.org/T423998#11843115 (10taavi) [09:21:12] 06cloud-services-team, 10Data-Services, 07Regression: The imagelinks replica in hewiki is not available - https://phabricator.wikimedia.org/T423998#11843117 (10taavi) [09:21:21] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Engineering-Radar, 06DBA: Re-run maintainviews on all clouddb* and an-redacteddb1001.eqiad.wmnet - https://phabricator.wikimedia.org/T422459#11843118 (10taavi) [09:21:30] 06cloud-services-team, 10Data-Services, 07Regression: The imagelinks replica in hewiki is not available - https://phabricator.wikimedia.org/T423998#11843119 (10taavi) [09:21:50] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Engineering-Radar, 06DBA: Re-run maintainviews on all clouddb* and an-redacteddb1001.eqiad.wmnet - https://phabricator.wikimedia.org/T422459#11843120 (10taavi) [09:22:18] 06cloud-services-team, 10Data-Services, 07Regression: The imagelinks replica in hewiki is not available - https://phabricator.wikimedia.org/T423998#11843125 (10Marostegui) p:05Triage→03Unbreak! a:03FCeratto-WMF [09:26:15] 06cloud-services-team, 10Data-Services, 07Regression: The imagelinks replica in hewiki is not available - https://phabricator.wikimedia.org/T423998#11843139 (10IKhitron) [09:30:22] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linting - detect examples in nested schema.properties - https://phabricator.wikimedia.org/T424002 (10KBach) 03NEW [09:33:12] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linting - unify example validation - https://phabricator.wikimedia.org/T423411#11843184 (10KBach) Validation of `(...).[every individual property].example` isn't fully covered by cases described in this task. I created T424002 to discuss th... [09:38:14] (03CR) 10Alien4444: AdminStats: +footnote on (re)protect and (re)block (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1275236 (https://phabricator.wikimedia.org/T354024) (owner: 10Alien4444) [09:38:48] (03CR) 10Alien4444: [C:03+2] Localisation updates from https://translatewiki.net. [labs/xtools] - 10https://gerrit.wikimedia.org/r/1275394 (owner: 10L10n-bot) [09:48:35] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Engineering-Radar, 06DBA: Re-run maintainviews on all clouddb* and an-redacteddb1001.eqiad.wmnet - https://phabricator.wikimedia.org/T422459#11843223 (10FCeratto-WMF) [09:48:42] 06cloud-services-team, 10Data-Services, 07Regression: The imagelinks replica in hewiki is not available - https://phabricator.wikimedia.org/T423998#11843226 (10FCeratto-WMF) I rerun maintains-view again and it seems ok now. [09:50:50] 06cloud-services-team, 10Data-Services, 07Regression: The imagelinks replica in hewiki is not available - https://phabricator.wikimedia.org/T423998#11843228 (10FCeratto-WMF) 05Open→03Resolved [10:01:21] 06cloud-services-team, 10Data-Services, 07Regression: The imagelinks replica in hewiki is not available - https://phabricator.wikimedia.org/T423998#11843261 (10IKhitron) Thank you. But the querry 104505 that returned 450 answers yesterday, gives only 3 now. I'll try understand what happens and which resu... [11:25:14] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/1275396 (owner: 10L10n-bot) [12:18:18] (03CR) 10Novem Linguae: AdminStats: +footnote on (re)protect and (re)block (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1275236 (https://phabricator.wikimedia.org/T354024) (owner: 10Alien4444) [12:25:21] 06cloud-services-team: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#11843859 (10Odinaldo) Dear @bd808, Many thanks for picking this up Bryan. As part of the separation of the functionality outlined above, we are revisiting the database provision on both sides. Are the... [12:43:56] FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:48:56] RESOLVED: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:51:15] (03update) 10taavi: kubernetes: Stop manipulating Ingress objects [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/100 (https://phabricator.wikimedia.org/T392356) [12:51:22] (03merge) 10taavi: kubernetes: Stop manipulating Ingress objects [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/100 (https://phabricator.wikimedia.org/T392356) [12:53:50] (03open) 10taavi: d/changelog: bump to 0.103.20 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/101 (https://phabricator.wikimedia.org/T392356) [12:55:17] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [12:55:48] 10Tool-etherpad-backup: Special characters are not handled consistently in the current backups, should be improved when migrating to S3 - https://phabricator.wikimedia.org/T423905#11843979 (10Tkarcher) 05Open→03In progress The immediate issue is resolved: I updated the renderer so all pad titles from the cur... [12:56:35] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli [12:56:40] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [12:57:33] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 07OKR-Work: Add the remaining linting rules - https://phabricator.wikimedia.org/T422600#11843988 (10HCoplin-WMF) [12:57:34] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 07OKR-Work: Add linting rules for operations and paths - https://phabricator.wikimedia.org/T422504#11843990 (10HCoplin-WMF) [12:57:50] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 07Essential-Work, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)): Linter: Split rules into more logical groupings. - https://phabricator.wikimedia.org/T415914#11843992 (10HCoplin-WMF) [12:57:56] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 07OKR-Work: Improve linting - unify example validation - https://phabricator.wikimedia.org/T423411#11843994 (10HCoplin-WMF) [12:57:58] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli [12:58:12] (03merge) 10taavi: d/changelog: bump to 0.103.20 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/101 (https://phabricator.wikimedia.org/T392356) [13:06:04] (03update) 10fnegri: Replace only views that need updating [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/9 (https://phabricator.wikimedia.org/T351637) [13:26:11] (03update) 10taavi: Remove tool ability to interact with Ingress objects [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/86 (https://phabricator.wikimedia.org/T392356 https://phabricator.wikimedia.org/T418276) [13:30:13] (03approved) 10filippo: Remove tool ability to interact with Ingress objects [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/86 (https://phabricator.wikimedia.org/T392356 https://phabricator.wikimedia.org/T418276) (owner: 10taavi) [13:32:02] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [13:32:07] !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers [13:35:40] (03merge) 10taavi: Remove tool ability to interact with Ingress objects [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/86 (https://phabricator.wikimedia.org/T392356 https://phabricator.wikimedia.org/T418276) [13:38:03] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-kubeusers: bump to 0.0.194-20260421133553-217c468a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1224 (https://phabricator.wikimedia.org/T392356 https://phabricator.wikimedia.org/T418276) [13:38:15] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-kubeusers: bump to 0.0.194-20260421133553-217c468a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1224 (https://phabricator.wikimedia.org/T392356 https://phabricator.wikimedia.org/T418276) [13:40:07] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [13:41:11] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [13:41:44] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [13:42:58] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [13:45:16] (03merge) 10taavi: maintain-kubeusers: bump to 0.0.194-20260421133553-217c468a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1224 (https://phabricator.wikimedia.org/T392356 https://phabricator.wikimedia.org/T418276) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:11:18] (03merge) 10andrew: trove: update tests to use the new db engines [repos/cloud/cloud-vps/tf-infra-test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tf-infra-test/-/merge_requests/7 [14:16:00] 06cloud-services-team, 10Toolforge, 07Documentation, 07good first task: Find and fix inaccuracies in Toolforge Django tutorial - https://phabricator.wikimedia.org/T245683#11844615 (10Anushka10patel) This is my fast good task, and I have completed a security-focused audit of the Help:Toolforge/My first Djan... [14:33:57] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 07OKR-Work: Improve linting - unify example validation - https://phabricator.wikimedia.org/T423411#11844790 (10HCoplin-WMF) p:05Triage→03Medium [14:35:46] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 13Patch-For-Review: OpenAPI linting: Add missing OpenAPI spec elements to Response Components - https://phabricator.wikimedia.org/T422739#11844813 (10Mooeypoo) 05Invalid→03Open (reopening just to get the patch merged -- unless we wa... [14:39:19] 06cloud-services-team, 10Data-Services: The imagelinks replica in hewiki is not available - https://phabricator.wikimedia.org/T423998#11844820 (10JJMC89) [15:00:41] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 13Patch-For-Review: OpenAPI linting: Add missing OpenAPI spec elements to Response Components - https://phabricator.wikimedia.org/T422739#11845135 (10Mooeypoo) [15:05:14] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 13Patch-For-Review: Fix linter issues discovered during implementation of the OAD example - https://phabricator.wikimedia.org/T414974#11845184 (10HCoplin-WMF) [15:09:50] (03update) 10raymond-ndibe: core.py: skip one-off jobs when updating storage [repos/cloud/toolforge/jobs-api] (fix_data_handling_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/288 (https://phabricator.wikimedia.org/T423544) [15:10:03] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 13Patch-For-Review: Fix linter issues discovered during implementation of the OAD example - https://phabricator.wikimedia.org/T414974#11845230 (10pmiazga) [15:14:26] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 07OKR-Work: Improve linting - unify example validation - https://phabricator.wikimedia.org/T423411#11845259 (10AGhirelli-WMF) a:03AGhirelli-WMF [15:14:42] (03update) 10raymond-ndibe: cli.py: stop resolving dynamic fields in the cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/151 (https://phabricator.wikimedia.org/T423828) [15:27:07] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [15:30:37] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment codfw1dev for all services [15:33:35] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [15:36:27] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [15:48:34] 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, 06Release-Engineering-Team, 07Essential-Work: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11845433 (10A_smart_kitten) [16:11:15] 06cloud-services-team, 10Cloud-VPS: [tofu-cloudvps] Add support for importing legacy cloudvps_puppet_prefix objects - https://phabricator.wikimedia.org/T423970#11845517 (10bd808) It may just be tracking down the ID that I didn't know exactly how to do. I think I tried to figure out how to do an import several... [16:43:43] 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, 06Release-Engineering-Team, 07Essential-Work: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11845787 (10Dzahn) [16:54:17] 10Cloud-VPS (Project-requests): Request creation of wise VPS project - https://phabricator.wikimedia.org/T424068 (10Carandraug) 03NEW [16:54:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.991% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [16:54:43] (03PS1) 10Alien4444: PagesCreated: add mimetype column for ns6 [labs/xtools] - 10https://gerrit.wikimedia.org/r/1275974 (https://phabricator.wikimedia.org/T352737) [16:59:34] RESOLVED: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.978% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [17:01:26] 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, 06Release-Engineering-Team, 07Essential-Work: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846019 (10Dzahn) Here is a doc with "the plan" (tm) how to move... [17:02:03] 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, 06Release-Engineering-Team, 07Essential-Work: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846024 (10Dzahn) @Arnoldokoth Could you comment on the quota si... [17:08:34] 06cloud-services-team: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#11846051 (10bd808) >>! In T416590#11843859, @Odinaldo wrote: > Are there any particular OSI-compliant database servers recommended for use //within// Cloud VPS? For example, are there examples of best pr... [17:09:36] 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, 06Release-Engineering-Team, 07Essential-Work: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846054 (10Arnoldokoth) @Dzahn The quota looks healthy. I think... [17:11:26] 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, 06Release-Engineering-Team, 07Essential-Work: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846060 (10Dzahn) [17:11:54] 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, 06Release-Engineering-Team, 07Essential-Work: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846062 (10Dzahn) Thank you @Arnoldokoth ! Sounds good. I don't... [17:12:33] 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, 06Release-Engineering-Team, 07Essential-Work: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846071 (10Dzahn) [17:15:56] FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:20:56] RESOLVED: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:09:03] (03PS1) 10Dzahn: jenkins: add fake password for docker registry for PipelineLib [labs/private] - 10https://gerrit.wikimedia.org/r/1275993 (https://phabricator.wikimedia.org/T423968) [18:09:10] 10Tool-centralnotice-banner-editor, 03Wikimedia-Hackathon-2026: Wikimedia Hackathon 2026: CentralNotice Banner Editor – What's new and What's next - https://phabricator.wikimedia.org/T424074 (10Oyelola_Victoria) 03NEW [18:09:28] (03CR) 10Dzahn: [V:03+2 C:03+2] jenkins: add fake password for docker registry for PipelineLib [labs/private] - 10https://gerrit.wikimedia.org/r/1275993 (https://phabricator.wikimedia.org/T423968) (owner: 10Dzahn) [18:16:04] 10VPS-project-Codesearch, 06collaboration-services: confd fails with "no such host" in SRV lookup from _etcd-client-ssl._tcp.codesearch.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T417458#11846436 (10Dzahn) I found I still had an open tab with a comment that I never submitted and was from a whil... [18:20:32] 10VPS-project-Phabricator, 10Bitu, 06collaboration-services, 06Infrastructure-Foundations, 10Phabricator: Allow Bitu to link Phabricator account - https://phabricator.wikimedia.org/T406495#11846463 (10Dzahn) Is this ticket resolved? [18:33:41] 10Tool-centralnotice-banner-editor, 03Wikimedia-Hackathon-2026: Wikimedia Hackathon 2026: Wikimedia Commons image lookup in the CentralNotice Banner Editor - https://phabricator.wikimedia.org/T424078 (10Oyelola_Victoria) 03NEW [18:35:08] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846567 (10brennen) [18:38:19] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846580 (10Dzahn) instance `phabricator-bullseye` already exists. We creat... [18:39:18] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846582 (10Dzahn) @brennen Wanna try to deploy latest to `phabricator-bull... [18:40:38] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846585 (10Dzahn) Ignore the latest comments, mixed up with the even older... [18:42:21] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846591 (10Dzahn) **phabricator-bookworm:** https://horizon.wikimedia.org... [18:43:36] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846594 (10Dzahn) **phabricator-trixie:** https://horizon.wikimedia.org/p... [19:34:13] 06cloud-services-team, 10superset.wmcloud.org: Sunset superset.wmcloud.org - https://phabricator.wikimedia.org/T416373#11846873 (10KCVelaga_WMF) >I would like to announce the new stewardship of this project as a followup to this email: > And what names would you like me to use for you? Sounds good. As I inte... [19:48:04] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846990 (10Dzahn) created a proxy to see what's on the existing VM: https... [19:49:18] 06cloud-services-team, 10superset.wmcloud.org: Sunset superset.wmcloud.org - https://phabricator.wikimedia.org/T416373#11847009 (10Dev) >>! In T416373#11841489, @Andrew wrote: > @Dev I will add you to the project; can you tell me your toolforge/cloud-vps username or shell name? >>! In T416373#11841498, @And... [21:12:10] 10Tool-centralnotice-banner-editor, 03Wikimedia-Hackathon-2026: Wikimedia Hackathon 2026: Wikimedia Commons image lookup in the CentralNotice Banner Editor - https://phabricator.wikimedia.org/T424078#11847362 (10Oyelola_Victoria) a:03Oyelola_Victoria [21:12:40] 10Tool-centralnotice-banner-editor, 03Wikimedia-Hackathon-2026: Wikimedia Hackathon 2026: CentralNotice Banner Editor – What's new and What's next - https://phabricator.wikimedia.org/T424074#11847363 (10Oyelola_Victoria) a:03Oyelola_Victoria [22:30:14] (03PS1) 10Alien4444: PagesCreated: add an incoming link count column [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276066 (https://phabricator.wikimedia.org/T276520) [22:31:02] (03CR) 10CI reject: [V:04-1] PagesCreated: add an incoming link count column [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276066 (https://phabricator.wikimedia.org/T276520) (owner: 10Alien4444) [22:45:44] (03PS2) 10Alien4444: PagesCreated: add an incoming link count column [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276066 (https://phabricator.wikimedia.org/T276520) [22:46:49] (03CR) 10CI reject: [V:04-1] PagesCreated: add an incoming link count column [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276066 (https://phabricator.wikimedia.org/T276520) (owner: 10Alien4444) [22:52:00] (03PS3) 10Alien4444: PagesCreated: add an incoming link count column [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276066 (https://phabricator.wikimedia.org/T276520) [22:59:15] (03PS4) 10Alien4444: PagesCreated: add an incoming link count column [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276066 (https://phabricator.wikimedia.org/T276520)