[00:30:51] 10Cloud-VPS, 06tools-infrastructure-team: Reactivate wlm-it-visual - https://phabricator.wikimedia.org/T442473 (10Ferdi2005) 03NEW [00:33:50] 10Cloud-VPS, 06tools-infrastructure-team: Reactivate wlm-it-visual - https://phabricator.wikimedia.org/T442473#12475571 (10JJMC89) [00:33:54] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: 2026-10-21 Toolforge and Cloud VPS down - https://phabricator.wikimedia.org/T441908#12475572 (10JJMC89) [00:48:01] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:02:58] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: 2026-10-21 Toolforge and Cloud VPS down - https://phabricator.wikimedia.org/T441908#12475661 (10Soda) [01:14:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [01:19:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [01:53:01] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:58:23] 10Cloud-VPS (Debian Bullseye Deprecation): Temporarily add 12 TB to the Cloud VPS project quota - https://phabricator.wikimedia.org/T442480 (10saper) 03NEW [02:00:22] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12475764 (10Doc_James) Okay tools appear up again. A few small things: 1) Annotation is not appear for SVG files such as this https://commons.wikimedia.org/wiki... [02:00:33] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate maps-osmdb from bullseye to .. - https://phabricator.wikimedia.org/T432955#12475765 (10saper) PostgreSQL 13 and PostgreSQL 18 installed in parallel on `caaa4a23-7e32-40e4-b737-03a3402c9b74` (maps-osmdb-2026-g1-template) for migration [02:01:22] 10Cloud-VPS (Debian Bullseye Deprecation): Temporarily add 12 TB to the Cloud VPS project quota for psql 13 to 18 migration - https://phabricator.wikimedia.org/T442480#12475769 (10saper) [02:04:32] 10Cloud-VPS (Quota-requests): Temporarily add 12 TB to the Cloud VPS project quota for psql 13 to 18 migration - https://phabricator.wikimedia.org/T442480#12475783 (10JJMC89) [02:04:35] 10Cloud-VPS (Quota-requests): Temporarily add 12 TB to the Cloud VPS project quota for psql 13 to 18 migration - https://phabricator.wikimedia.org/T442480#12475785 (10JJMC89) a:05saper→03None [02:11:00] (03open) 10ibrahemqasim: Add broad unit and integration test coverage [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/17 [02:13:15] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell - https://phabricator.wikimedia.org/T442482 (10Andrew) 03NEW [02:13:29] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12475799 (10Andrew) [02:22:21] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12475803 (10Andrew) Here is the raw output from the cumin run of the audit script on all VMs. Interpretation to follow... ` ===== NODE GROUP ====... [02:23:52] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12475804 (10Andrew) [02:25:37] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12475805 (10Andrew) [02:26:31] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12475806 (10Andrew) [02:48:51] (03update) 10ibrahemqasim: Add broad unit and integration test coverage [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/17 [02:49:34] (03update) 10ibrahemqasim: Add broad unit and integration test coverage [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/17 [02:49:44] (03update) 10ibrahemqasim: Add broad unit and integration test coverage [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/17 [02:50:07] FIRING: [5x] SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:53:57] 10Tools: Bring up apersonbot after the PHP vulns - https://phabricator.wikimedia.org/T442477#12475836 (10Bugreporter) [02:57:20] (03update) 10ibrahemqasim: Add broad unit and integration test coverage [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/17 [03:02:56] (03update) 10ibrahemqasim: Add broad unit and integration test coverage [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/17 [03:02:57] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40441575561 (https://github.com/cluebotng/component-configs/commits/457585647825bf9845a8c2a5e6ab8dba5aca3763) [03:02:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [03:07:16] 10Tool-commons-delinquent: Tool-commons-delinquent: HTTP Code 500 error - https://phabricator.wikimedia.org/T437416#12475840 (10SD_hehua) >>! 在T437416#12464395中，@SD_hehua写道： > Now it shows "no healthy upstream". Maybe because of T441908? But I don't know if this will work then. [03:07:26] (03update) 10ibrahemqasim: Add broad unit and integration test coverage [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/17 [03:08:33] (03update) 10ibrahemqasim: Add broad unit and integration test coverage [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/17 [03:09:45] (03merge) 10ibrahemqasim: Add broad unit and integration test coverage [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/17 [03:14:39] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442736579 (https://github.com/cluebotng/component-configs/commits/44d3670909d5f5c7236df807b8156fa2cf0d81e6) [03:14:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [03:15:02] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442736567 (https://github.com/cluebotng/component-configs/commits/44d3670909d5f5c7236df807b8156fa2cf0d81e6) [03:15:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [03:16:55] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442895976 (https://github.com/cluebotng/component-configs/commits/22ef6144c3a3323d2054a04b4a2fa02c7c9c2cee) [03:16:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [03:17:58] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442895988 (https://github.com/cluebotng/component-configs/commits/22ef6144c3a3323d2054a04b4a2fa02c7c9c2cee) [03:17:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [03:18:54] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442895975 (https://github.com/cluebotng/component-configs/commits/22ef6144c3a3323d2054a04b4a2fa02c7c9c2cee) [03:18:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [03:20:10] 10Cloud-VPS (Debian Bullseye Deprecation): Access maps DB via osmdb.svc.maps.eqiad1.wikimedia.cloud. CNAME - https://phabricator.wikimedia.org/T442484 (10saper) 03NEW [03:20:46] 10Cloud-VPS (Debian Bullseye Deprecation): Access maps DB via osmdb.svc.maps.eqiad1.wikimedia.cloud. CNAME - https://phabricator.wikimedia.org/T442484#12475859 (10saper) [03:25:48] 10Cloud-VPS (Debian Bullseye Deprecation): Access maps DB via osmdb.svc.maps.eqiad1.wikimedia.cloud. CNAME - https://phabricator.wikimedia.org/T442484#12475862 (10JJMC89) a:05saper→03None [03:25:54] 10Cloud-VPS (Debian Bullseye Deprecation): Access maps DB via osmdb.svc.maps.eqiad1.wikimedia.cloud. CNAME - https://phabricator.wikimedia.org/T442484#12475863 (10saper) Connections example from maps-wma2: ` root@maps-wma2:/var/www# psql -h osmdb.svc.maps.eqiad1.wikimedia.cloud. -U osm_dschwen gis Password for... [03:29:51] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444183977 (https://github.com/cluebotng/component-configs/commits/368024e9769a6b45b65ce73920e2f880a173e978) [03:29:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [03:30:22] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444184085 (https://github.com/cluebotng/component-configs/commits/368024e9769a6b45b65ce73920e2f880a173e978) [03:30:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [03:30:54] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444183898 (https://github.com/cluebotng/component-configs/commits/368024e9769a6b45b65ce73920e2f880a173e978) [03:30:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [03:36:19] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444706514 (https://github.com/cluebotng/component-configs/commits/a9515adb6b57ca73316c1c6b8559bc6b49255d7f) [03:36:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [03:38:37] !log tools.cluebot-syncer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40445004144 (https://github.com/cluebotng/component-configs/commits/ba4dac835cbaed39f73a3456f14e9543c2fe63ac) [03:38:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [03:40:57] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40445124134 (https://github.com/cluebotng/component-configs/commits/f7026cd7409b14b6865323d50c6942fde358921e) [03:40:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [03:41:09] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40445221759 (https://github.com/cluebotng/component-configs/commits/e6c8388c5259b03b82e9139c4d465a9c0084729f) [03:41:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [03:41:25] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40445221810 (https://github.com/cluebotng/component-configs/commits/e6c8388c5259b03b82e9139c4d465a9c0084729f) [03:41:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [03:55:08] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527178 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663) [03:55:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [03:55:10] !log tools.toolforge-functional-runner Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527044 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663) [03:55:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolforge-functional-runner/SAL [03:55:12] !log tools.cluebot-syncer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527408 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663) [03:55:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [03:55:25] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527024 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663) [03:55:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [03:55:27] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527051 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663) [03:55:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [03:55:47] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527142 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663) [03:55:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [03:55:54] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527027 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663) [03:55:55] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446526995 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663) [03:55:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [03:55:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [03:56:12] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446526960 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663) [03:56:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [04:04:34] !log tools.cluebot-syncer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375447 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b) [04:04:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [04:04:39] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375517 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b) [04:04:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [04:04:42] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375417 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b) [04:04:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [04:04:56] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375409 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b) [04:04:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [04:05:28] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375498 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b) [04:05:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [04:05:30] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375410 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b) [04:05:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [04:05:31] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375412 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b) [04:05:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [04:06:04] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375770 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b) [04:06:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [04:08:46] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447571824 (https://github.com/cluebotng/component-configs/commits/7d10e42f436c05ef56f6f317648388d592c3c05f) [04:08:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [04:16:43] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40448492483 (https://github.com/cluebotng/component-configs/commits/d09fba9de2a42159fa1c899076588e5408ccc19b) [04:16:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [04:42:10] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate maps-osmdb from bullseye to .. - https://phabricator.wikimedia.org/T432955#12475908 (10saper) I was able to do the following: 1. (debian 11) shutdown postgres 13 and the replication 2. (debian 11) umount && detach https://horizon.wikimedia.org/project/volumes/e9... [04:49:16] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451550237 (https://github.com/cluebotng/component-configs/commits/d19590e442b664819343d4373714879e039ae49f) [04:49:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [04:51:48] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451765837 (https://github.com/cluebotng/component-configs/commits/b95cee1256fe85e86858e9c35dbf825c29ae24dc) [04:51:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [04:52:36] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451765772 (https://github.com/cluebotng/component-configs/commits/b95cee1256fe85e86858e9c35dbf825c29ae24dc) [04:52:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [04:52:39] 10Tools: Bring up apersonbot after the PHP vulns - https://phabricator.wikimedia.org/T442477#12475911 (10Novem_Linguae) {F110923575} Guess I should take a look at this one. [04:52:58] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451765768 (https://github.com/cluebotng/component-configs/commits/b95cee1256fe85e86858e9c35dbf825c29ae24dc) [04:53:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [04:55:16] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451826228 (https://github.com/cluebotng/component-configs/commits/264fe43c86ed52f85594cf77c7f12208c886bae9) [04:55:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [04:56:43] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451826128 (https://github.com/cluebotng/component-configs/commits/264fe43c86ed52f85594cf77c7f12208c886bae9) [04:56:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [04:59:41] !log tools.cluebotng-trainer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518133 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982) [04:59:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [04:59:47] !log tools.cluebotng-staging Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518212 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982) [04:59:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [04:59:57] !log tools.toolforge-functional-runner Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40452518077 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982) [04:59:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolforge-functional-runner/SAL [05:00:04] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40452518051 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982) [05:00:04] !log tools.cluebot-syncer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40452518081 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982) [05:00:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [05:00:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [05:00:15] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40452518152 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982) [05:00:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [05:00:53] !log tools.cluebotng-monitoring Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518130 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982) [05:00:55] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518044 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982) [05:00:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [05:00:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [05:01:05] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518090 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982) [05:01:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [05:06:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [05:06:41] 10Tools: Bring up apersonbot after the PHP vulns - https://phabricator.wikimedia.org/T442477#12475926 (10Novem_Linguae) > AFD history afdstats has a separate toolforge repo: https://afdstats.toolforge.org. It is up. The apersonbot account has the following tools in the public_html folder: {F110926187} The we... [05:11:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [05:11:43] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40453565998 (https://github.com/cluebotng/component-configs/commits/0b40c20b317be8b21d7e6b4a9c452f2be1a63558) [05:11:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [05:15:37] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40453802690 (https://github.com/cluebotng/component-configs/commits/4186d7d35e3637e5f7722d66c028994026229d84) [05:15:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [05:33:35] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455721145 (https://github.com/cluebotng/component-configs/commits/33785ac45923c22c4e14a8b54268e25c98b83f89) [05:33:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [05:34:46] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40455721193 (https://github.com/cluebotng/component-configs/commits/33785ac45923c22c4e14a8b54268e25c98b83f89) [05:34:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [05:35:01] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455721244 (https://github.com/cluebotng/component-configs/commits/33785ac45923c22c4e14a8b54268e25c98b83f89) [05:35:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [05:35:05] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455798034 (https://github.com/cluebotng/component-configs/commits/80c03e09cf4126cf14f61eae6feded9feafbb264) [05:35:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [05:38:25] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455798083 (https://github.com/cluebotng/component-configs/commits/80c03e09cf4126cf14f61eae6feded9feafbb264) [05:38:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [05:39:06] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455798133 (https://github.com/cluebotng/component-configs/commits/80c03e09cf4126cf14f61eae6feded9feafbb264) [05:39:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [05:42:16] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40456450428 (https://github.com/cluebotng/component-configs/commits/b0fce42989bc739f2992831884538d17c4220a07) [05:42:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [05:50:00] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40457189661 (https://github.com/cluebotng/component-configs/commits/30587ffc7ef251c2e1a6bcd1d55b7a3b2493d65c) [05:50:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [05:53:01] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [05:56:02] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40457745340 (https://github.com/cluebotng/component-configs/commits/c09fe8cd0f8bc989e3798ba689cbffc5c5cb362a) [05:56:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [06:14:05] 10Cloud-VPS, 06tools-infrastructure-team, 06tools-platform-team: Reactivate azwikimedia - https://phabricator.wikimedia.org/T442496 (10Nemoralis) 03NEW [06:14:22] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40459540079 (https://github.com/cluebotng/component-configs/commits/bc660b38ce5052d918a9c8048fded9d70267c171) [06:14:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [06:14:33] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40459540107 (https://github.com/cluebotng/component-configs/commits/bc660b38ce5052d918a9c8048fded9d70267c171) [06:14:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [06:15:08] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40459540119 (https://github.com/cluebotng/component-configs/commits/bc660b38ce5052d918a9c8048fded9d70267c171) [06:15:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [06:17:14] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40459831650 (https://github.com/cluebotng/component-configs/commits/bfbe8dc9171d388710c1d5eface2d00809cd79dd) [06:17:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [06:44:42] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40462539944 (https://github.com/cluebotng/component-configs/commits/e4c07d7c9925e5e434298a95070b8f40a7db75c8) [06:44:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [06:46:53] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40462601965 (https://github.com/cluebotng/component-configs/commits/b4741b807618f702050fab2dca0bd5bbf12941ed) [06:46:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [06:47:04] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40462601906 (https://github.com/cluebotng/component-configs/commits/b4741b807618f702050fab2dca0bd5bbf12941ed) [06:47:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [06:49:56] FIRING: [5x] SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:08:44] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [jobs-api] Allow toolforge job schedule to specify a time zone - https://phabricator.wikimedia.org/T408371#12476169 (10Tkarcher) As just had to manually adjust the jobs again: Any update on this? [07:10:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:20:56] 10Cloud-VPS, 06tools-infrastructure-team: Reactivate wlm-it-visual - https://phabricator.wikimedia.org/T442473#12476186 (10Ysogo) Please note that "horizon" site (https://horizon.wikimedia.org/project/instances/) is still not available; this makes troublesome to manage VMs; have you any recovery time exitimate... [07:25:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:45:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:50:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:52:13] 10Tool-itwiki-bot-cancellazioni: Migrate itwiki BotCancellazioni from tool itwiki to dedicated tool - https://phabricator.wikimedia.org/T442396#12476255 (10valerio.bozzolan) I do not remember anymore which API token I've reset, anyway I've reset it again since ~8:33 Europe/Rome. [07:54:06] FIRING: HarborDown: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [07:56:48] 10Tool-itwiki-bot-cancellazioni: Migrate itwiki BotCancellazioni from tool itwiki to dedicated tool - https://phabricator.wikimedia.org/T442396#12476260 (10valerio.bozzolan) The bot is running again but not writing anything: > [2026-10-27 08:40:19][INFO] start > [2026-10-27 08:40:20][INFO] work on 2026/10/2... [07:59:40] 10Tool-itwiki-bot-cancellazioni: Migrate itwiki BotCancellazioni from tool itwiki to dedicated tool - https://phabricator.wikimedia.org/T442396#12476262 (10valerio.bozzolan) 05Resolved→03Open Re-opening to keep this visible. [08:01:54] 10Tool-itwiki: Migrate itwiki's "markadmins" job from tool itwiki to dedicated tool - https://phabricator.wikimedia.org/T442501 (10valerio.bozzolan) 03NEW [08:02:37] 10Cloud-VPS, 06tools-infrastructure-team: Reactivate wlm-it-visual - https://phabricator.wikimedia.org/T442473#12476289 (10fgiunchedi) According to audit wlm-staging-app.wlm-it-visual.eqiad1.wikimedia.cloud does run php in containers, specifically 8.1.33 and 8.3.32 ` root@wlm-staging-app:~# docker ps CONTAINE... [08:25:52] 10Tool-itwiki-bot-cancellazioni: Migrate itwiki BotCancellazioni from tool itwiki to dedicated tool - https://phabricator.wikimedia.org/T442396#12476386 (10valerio.bozzolan) I expect this edit to be undone by the bot in less than 8 minutes. https://it.wikipedia.org/w/index.php?title=Wikipedia:Pagine_da_cancella... [08:39:10] 10Tool-itwiki-bot-cancellazioni: Migrate itwiki BotCancellazioni from tool itwiki to dedicated tool - https://phabricator.wikimedia.org/T442396#12476422 (10valerio.bozzolan) 05Open→03Resolved ✅ https://it.wikipedia.org/w/index.php?title=Wikipedia:Pagine_da_cancellare/Conta/2026_ottobre_27&diff=next&oldid... [08:52:26] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40476249856 (https://github.com/cluebotng/component-configs/commits/838abafe65c57870163e3118ce8a093debe0d2e6) [08:52:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [08:53:21] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40476249547 (https://github.com/cluebotng/component-configs/commits/838abafe65c57870163e3118ce8a093debe0d2e6) [08:53:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [08:54:03] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40476249549 (https://github.com/cluebotng/component-configs/commits/838abafe65c57870163e3118ce8a093debe0d2e6) [08:54:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [09:00:02] 10Cloud-VPS (Quota-requests): Temporarily add 12 TB to the Cloud VPS project quota for psql 13 to 18 migration - https://phabricator.wikimedia.org/T442480#12476509 (10taavi) Which project is this for? [09:04:36] 10Striker, 06tools-platform-team, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12476513 (10Sfaci) [09:07:08] 10Cloud-VPS, 06tools-infrastructure-team: [Email response] wmcz-stats has no PHP - https://phabricator.wikimedia.org/T442504 (10Urbanecm) 03NEW [09:10:53] 10Cloud-VPS, 06tools-infrastructure-team: [Email response] wmcz-stats has no PHP - https://phabricator.wikimedia.org/T442504#12476531 (10Urbanecm) Same applies for the two Docker containers it runs. [09:13:41] 10Cloud-VPS, 06tools-infrastructure-team: [Email response] wmcz-stats has no PHP - https://phabricator.wikimedia.org/T442504#12476543 (10taavi) 05Open→03Resolved a:03taavi Indeed, all is ok here. [09:24:56] FIRING: [5x] SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:27:25] 10Cloud-VPS, 06tools-infrastructure-team: Reactivate wlm-it-visual - https://phabricator.wikimedia.org/T442473#12476596 (10Ysogo) @fgiunchedi we can get ride of them: we will update containers and be back to you. [09:52:17] (03PS1) 10Samtar: Review access change [labs/xtools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1359215 [09:52:56] (03PS2) 10Samtar: Review access change [labs/xtools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1359215 (https://phabricator.wikimedia.org/T441761) [09:53:02] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [09:53:45] 10Cloud-VPS, 06tools-infrastructure-team, 10CirrusSearch, 06Discovery-Search: cirrus-integ5.search VM busted, unable to finish booting - https://phabricator.wikimedia.org/T442510 (10fgiunchedi) 03NEW [09:55:15] (03PS3) 10Samtar: Add l10n-bot-watcher to labs/xtools [labs/xtools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1359215 (https://phabricator.wikimedia.org/T441761) [09:57:40] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40484261191 (https://github.com/cluebotng/component-configs/commits/698a9bac73dd63611a0686bc1785f80ca9a6683e) [09:57:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [09:57:53] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40484261201 (https://github.com/cluebotng/component-configs/commits/698a9bac73dd63611a0686bc1785f80ca9a6683e) [09:57:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [09:58:10] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40484261158 (https://github.com/cluebotng/component-configs/commits/698a9bac73dd63611a0686bc1785f80ca9a6683e) [09:58:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [09:58:17] (03CR) 10Hashar: [V:03+2 C:03+2] Add l10n-bot-watcher to labs/xtools [labs/xtools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1359215 (https://phabricator.wikimedia.org/T441761) (owner: 10Samtar) [09:58:31] 10Tool-wiki-ocr, 10All-and-every-Wikisource: OCR tool in Wikisource not working - https://phabricator.wikimedia.org/T442268#12476679 (10taavi) 05Open→03Resolved a:03taavi The `wikisource` project was brought up yesterday. [09:59:27] 10Cloud-VPS, 06tools-infrastructure-team, 06tools-platform-team: Reactivate azwikimedia - https://phabricator.wikimedia.org/T442496#12476695 (10taavi) 05Open→03Resolved a:03taavi Done. [10:01:19] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.reboot for all workers [10:01:20] !log taavi@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all workers [10:17:21] (03update) 10renovatebot: Update python Docker tag to v3.15.2 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [10:26:45] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40487986665 (https://github.com/cluebotng/component-configs/commits/7cc193ac0999f846b516b843914c70a13924f99f) [10:26:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [10:40:40] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40489900248 (https://github.com/cluebotng/component-configs/commits/8007804f0273db3ecaf618e4e3723dbe1bf0fb67) [10:40:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [10:47:11] RESOLVED: SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:47:50] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Bring back toolsbeta - https://phabricator.wikimedia.org/T442516 (10taavi) 03NEW [10:50:43] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40491080407 (https://github.com/cluebotng/component-configs/commits/d72ea811ba6c14ea73c3b76fe67040cdac4fa091) [10:50:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [11:02:21] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40492675207 (https://github.com/cluebotng/component-configs/commits/677efd2e385dcbac779a72a750798b6b7fa31221) [11:02:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [11:10:15] 10Cloud-VPS (Quota-requests): Temporarily add 12 TB to the Cloud VPS project quota for psql 13 to 18 migration - https://phabricator.wikimedia.org/T442480#12476897 (10saper) maps on cloud-vps, thank you! [11:32:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:37:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:39:10] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40497231000 (https://github.com/cluebotng/component-configs/commits/ddd9bcc85b24abfbec8521371d7289c73517fb21) [11:39:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [11:39:16] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40497231026 (https://github.com/cluebotng/component-configs/commits/ddd9bcc85b24abfbec8521371d7289c73517fb21) [11:39:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [11:50:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:55:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:57:14] 10Striker, 06tools-platform-team, 07good first task: Alerts: new icon for "Mark as read" - https://phabricator.wikimedia.org/T442526 (10valerio.bozzolan) 03NEW Thank you for tagging this task with #good_first_task for Wikimedia newcomers! Newcomers often may not be aware of things that may seem obvious to... [12:01:02] 10Striker, 06tools-platform-team, 07good first task: Alerts: new icon for "Mark as read" - https://phabricator.wikimedia.org/T442526#12477240 (10valerio.bozzolan) [12:03:05] 10Striker, 06tools-platform-team, 07good first task: Alerts: new icon for "Mark as read" - https://phabricator.wikimedia.org/T442526#12477245 (10valerio.bozzolan) [12:16:17] 10Toolforge, 06tools-platform-team: [builds-api] very slow to response to image list / [components-api] has constant timeouts to api.svc.tools.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T442194#12477325 (10DamianZaremba) This does not appear to be entirely solved by T442364 While not constantl... [12:16:45] (03update) 10renovatebot: Update python Docker tag to v3.15.2 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [12:31:29] 10Cloud-VPS, 06tools-infrastructure-team, 10wikimaps-warper: maps-warper5 VPS on CloudVPS project maps is good to be restored - https://phabricator.wikimedia.org/T442529 (10Chippyy) 03NEW [12:31:57] 10Cloud-VPS, 06tools-infrastructure-team, 10wikimaps-warper: maps-warper5 VPS on CloudVPS project maps is good to be restored - https://phabricator.wikimedia.org/T442529#12477418 (10Chippyy) [12:41:23] 10Cloud-VPS, 06tools-infrastructure-team, 10wikimaps-warper: maps-warper5 VPS on CloudVPS project maps is good to be restored - https://phabricator.wikimedia.org/T442529#12477443 (10taavi) Hello! Unfortunately we can only restore entire projects at a time, not individual instances. In the `maps` project ther... [12:47:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:48:43] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12477505 (10mikez-WMF) > Annotation is not appear for SVG files such as this https://commons.wikimedia.org/wiki/File:Gemstones-production.svg though should as y... [12:52:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:04:18] 10Cloud-VPS, 06tools-infrastructure-team, 10wikimaps-warper: maps-warper5 VPS on CloudVPS project maps is good to be restored - https://phabricator.wikimedia.org/T442529#12477619 (10Chippyy) Sounds very wise. I believe @dschwen might be the contact for maps-wma2 at least. Once all the instances are green... [13:05:56] !log tools.cluebot-syncer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508592543 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2) [13:05:57] !log tools.toolforge-functional-runner Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508591170 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2) [13:05:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [13:05:59] !log tools.cluebotng-staging Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591043 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2) [13:05:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolforge-functional-runner/SAL [13:06:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [13:06:05] !log tools.cluebotng-trainer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591039 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2) [13:06:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [13:06:10] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508591062 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2) [13:06:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [13:06:27] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508590936 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2) [13:06:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [13:06:29] !log tools.cluebotng-monitoring Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508590861 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2) [13:06:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [13:06:35] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591155 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2) [13:06:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [13:07:06] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591099 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2) [13:07:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [13:08:30] 10Toolforge, 06tools-platform-team: [webservice-cli] set default webservice as php 8.4 - https://phabricator.wikimedia.org/T442199#12477649 (10taavi) →14Duplicate dup:03T401715 [13:08:31] 10Toolforge, 06tools-platform-team: Remove or replace "default" web service image - https://phabricator.wikimedia.org/T401715#12477651 (10taavi) [13:10:18] (03open) 10taavi: Remove default webservice type entirely [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/127 (https://phabricator.wikimedia.org/T401715) [13:17:41] 10Striker, 06tools-platform-team: Alerts: new icon for "Mark as read" - https://phabricator.wikimedia.org/T442526#12477701 (10taavi) Please do not tag #good-first-bug on tasks on codebases where you do not have the ability to commit to speedy reviews and deployments. I'm not convinced this is worth it tweakin... [13:27:27] 10Tool-clip2commons: Decoding error shown with no explanation when the browser cannot decode the source video - https://phabricator.wikimedia.org/T442539 (10Dev) 03NEW [13:27:50] 10Tool-clip2commons: Decoding error shown with no explanation when the browser cannot decode the source video - https://phabricator.wikimedia.org/T442539#12477762 (10Dev) p:05Triage→03Medium [13:47:40] 10Toolforge, 06tools-platform-team: Heads-up: A lot of traffic in one of my tools - https://phabricator.wikimedia.org/T442544 (10magnusmanske) 03NEW [13:50:46] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40515108521 (https://github.com/cluebotng/component-configs/commits/e55b6738ff115956fc47c673e403b2d6f91b24a5) [13:50:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [13:51:51] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40515108700 (https://github.com/cluebotng/component-configs/commits/e55b6738ff115956fc47c673e403b2d6f91b24a5) [13:51:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [13:53:02] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [14:09:12] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40517784751 (https://github.com/cluebotng/component-configs/commits/3c4486b403207ab117cadd33785b90d078a4f016) [14:09:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [14:09:15] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40517784252 (https://github.com/cluebotng/component-configs/commits/3c4486b403207ab117cadd33785b90d078a4f016) [14:09:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [14:09:50] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40517784076 (https://github.com/cluebotng/component-configs/commits/3c4486b403207ab117cadd33785b90d078a4f016) [14:09:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [14:12:48] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40517885090 (https://github.com/cluebotng/component-configs/commits/fbcf68ea18c187fbb8af37ed04a674837d121903) [14:12:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [14:15:12] 10PAWS, 10Quarry, 06tools-infrastructure-team, 06tools-platform-team: Move canonical Paws and Quarry code source from github to gitlab - https://phabricator.wikimedia.org/T442548 (10Andrew) 03NEW [14:15:14] !log tools.cluebotng-staging Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40518783500 (https://github.com/cluebotng/component-configs/commits/931b620f8c154ef5e68d899a25f218c171d6de5d) [14:15:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [14:15:49] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40518783532 (https://github.com/cluebotng/component-configs/commits/931b620f8c154ef5e68d899a25f218c171d6de5d) [14:15:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [14:16:01] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40518783783 (https://github.com/cluebotng/component-configs/commits/931b620f8c154ef5e68d899a25f218c171d6de5d) [14:16:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [14:30:04] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12478167 (10Volans) > I don't know what "php=yes-unknown" means but hopefully apt upgrade will resolve them as well. That's where the php version... [14:31:28] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12478186 (10Volans) [14:39:57] 06tools-infrastructure-team: Create temporary secgroupadmin keystone role - https://phabricator.wikimedia.org/T442554 (10Andrew) 03NEW [14:50:42] 10Tool-itwiki-bot-cancellazioni: BotCancellazioni: rollback "cllimit=500" workaround after T433922 has been resolved - https://phabricator.wikimedia.org/T434598#12478348 (10valerio.bozzolan) 05Open→03Declined > It could be okay, if there are not many categories for the 100 category members, but when each... [14:51:44] 10Cloud-VPS, 06tools-infrastructure-team, 10CirrusSearch, 06Discovery-Search (2026.09.01 - 2026.10.02): cirrus-integ5.search VM busted, unable to finish booting - https://phabricator.wikimedia.org/T442510#12478359 (10dcausse) a:03dcausse [14:59:32] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12478478 (10Zabe) [15:02:47] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40526418448 (https://github.com/cluebotng/component-configs/commits/9c1e7dfb720d9ef57fe0e066dcebe9ccd5701eaf) [15:02:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [15:04:36] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12478534 (10thcipriani) p:05Triage→03High [15:05:31] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12478541 (10thcipriani) Thanks @Volans/@Andrew ! Broke down to specific boxes/tasks in the description: lots of folks should be able to help [15:07:46] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40527035651 (https://github.com/cluebotng/component-configs/commits/7dd9e7d26ac0141af54d06f81163a130e5a402d3) [15:07:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [15:10:16] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12478584 (10thcipriani) I think https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning should only affect puppet roles/hiera... [15:13:58] 10Striker, 06tools-platform-team: Add a web shell allowing people to perform actions as their tool from striker - https://phabricator.wikimedia.org/T144713#12478609 (10valerio.bozzolan) This feature sounds pretty much scaring to me. [15:24:27] (03approved) 10fnegri: Remove default webservice type entirely [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/127 (https://phabricator.wikimedia.org/T401715) (owner: 10taavi) [15:40:52] 06tools-infrastructure-team: Create temporary secgroupadmin keystone role - https://phabricator.wikimedia.org/T442554#12478825 (10Andrew) [15:41:23] 06cloud-services-team, 10Striker, 06tools-platform-team, 07Design: Striker should use Codex - https://phabricator.wikimedia.org/T380114#12478835 (10valerio.bozzolan) Nice! As I believe this could take years, maybe we can start preparing at least a branch called "codex", and improving the README, asking con... [15:42:07] 10Striker, 06tools-platform-team: Alerts: new icon for "Mark as read" - https://phabricator.wikimedia.org/T442526#12478840 (10valerio.bozzolan) >>! In T442526#12477701, @taavi wrote: > Please do not tag #good-first-bug on tasks on codebases where you do not have the ability to commit to speedy reviews and depl... [15:52:29] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12478904 (10Southparkfan) Thanks Andrew! Let's get the process started. >>! In T442482#12478584, @thcipriani wrote: > I think https://gitlab.wik... [15:53:06] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40534588912 (https://github.com/cluebotng/component-configs/commits/a9e566fe0aa786659f3ee0801327ee92e5f26106) [15:53:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [15:53:29] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40534588887 (https://github.com/cluebotng/component-configs/commits/a9e566fe0aa786659f3ee0801327ee92e5f26106) [15:53:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [16:05:20] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40536663681 (https://github.com/cluebotng/component-configs/commits/d02fac71cd9a7c00d0b2cba2a245b95677c46c4d) [16:05:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [16:14:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:14:47] 10Striker, 06tools-platform-team, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12479053 (10JVanderhoop-WMF) [16:16:56] 10Toolforge, 06tools-platform-team: [lima-kilo,logging,logs-api] find a replacement for minio (s3 for loki) - https://phabricator.wikimedia.org/T439720#12479119 (10fnegri) [16:17:20] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40538846790 (https://github.com/cluebotng/component-configs/commits/3f4a71e8450918f10ab12a9f4b18506226d32f4c) [16:17:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [16:17:46] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40538846848 (https://github.com/cluebotng/component-configs/commits/3f4a71e8450918f10ab12a9f4b18506226d32f4c) [16:17:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [16:18:18] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40538846798 (https://github.com/cluebotng/component-configs/commits/3f4a71e8450918f10ab12a9f4b18506226d32f4c) [16:18:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [16:18:58] 10Toolforge, 06tools-platform-team: [lima-kilo,logging,logs-api] find a replacement for minio (s3 for loki) - https://phabricator.wikimedia.org/T439720#12479132 (10fnegri) [16:19:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:23:41] (03update) 10deerbee: Extend matching pattern [toolforge-repos/wmde-github-phab-bot] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-github-phab-bot/-/merge_requests/2 (https://phabricator.wikimedia.org/T1 https://phabricator.wikimedia.org/T410608) [16:32:09] (03update) 10deerbee: Extend matching pattern [toolforge-repos/wmde-github-phab-bot] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-github-phab-bot/-/merge_requests/2 (https://phabricator.wikimedia.org/T1 https://phabricator.wikimedia.org/T410608) [16:33:38] (03update) 10deerbee: Extend matching pattern [toolforge-repos/wmde-github-phab-bot] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-github-phab-bot/-/merge_requests/2 (https://phabricator.wikimedia.org/T1 https://phabricator.wikimedia.org/T410608) [16:34:45] (03open) 10dtorsani: Draft: Added initial examples to adoption gallery [toolforge-repos/wikimedia-attribution] - 10https://gitlab.wikimedia.org/toolforge-repos/wikimedia-attribution/-/merge_requests/9 [16:36:49] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40542112336 (https://github.com/cluebotng/component-configs/commits/e3b42405f7637feca5178261ba1e5662c1009020) [16:36:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [16:36:53] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40542112168 (https://github.com/cluebotng/component-configs/commits/e3b42405f7637feca5178261ba1e5662c1009020) [16:36:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [16:37:38] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40542112186 (https://github.com/cluebotng/component-configs/commits/e3b42405f7637feca5178261ba1e5662c1009020) [16:37:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [16:38:55] 10Toolforge, 06tools-platform-team: Deploying built static apps to toolforge - https://phabricator.wikimedia.org/T435088#12479449 (10Gouvernathor) >>! In T435088#12471587, @DamianZaremba wrote: > In the context of build service/push to deploy - I would counter this argument and say the tool author is the best... [16:47:50] 10Cloud-VPS (Debian Bullseye Deprecation), 10SRE-swift-storage: Migrate swift away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435783#12479549 (10Ladsgroup) It looks like it should be closed. I leave it to you. [16:48:25] 10Toolforge, 06tools-platform-team: Deploying built static apps to toolforge - https://phabricator.wikimedia.org/T435088#12479576 (10DamianZaremba) >>! In T435088#12479449, @Gouvernathor wrote: >>>! In T435088#12471587, @DamianZaremba wrote: > But I'm not talking about the build phase, which indeed depends a l... [16:58:57] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40545550695 (https://github.com/cluebotng/component-configs/commits/9a4c6ecd52da46f16ac808838c87f2b4608e2987) [16:59:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [17:04:04] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12479834 (10thcipriani) [17:05:52] 14cloud-services-team (Kanban), 10Cloud-VPS, 06tools-infrastructure-team, 06SRE, 07LDAP: investigate slapd memory leak - https://phabricator.wikimedia.org/T130593#12479870 (10hnowlan) [17:06:20] 10Tools, 10observability: Find replacement for graphite generated graphs of prometheus data - https://phabricator.wikimedia.org/T214336#12479880 (10hnowlan) 05Open→03Declined graphite is no longer in use. [17:10:31] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40547448775 (https://github.com/cluebotng/component-configs/commits/d8e5c2b81841263635a002cdb005aaa7328042b8) [17:10:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [17:16:58] 10Cloud-VPS, 06tools-infrastructure-team, 10CirrusSearch, 06Discovery-Search (2026.09.01 - 2026.10.02): cirrus-integ5.search VM busted, unable to finish booting - https://phabricator.wikimedia.org/T442510#12480025 (10dcausse) Re-created a cirrus-integ6.search instance (deleted cirrus-integ5), seems to work... [17:53:02] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [18:01:32] 10PAWS, 06tools-platform-team: [paws] Upgrade images to supported os - https://phabricator.wikimedia.org/T442389#12480473 (10dcaro) 05Open→03In progress [18:01:37] 10PAWS, 06tools-platform-team: [paws] Upgrade images to supported os - https://phabricator.wikimedia.org/T442389#12480476 (10dcaro) p:05Triage→03High [18:03:45] 10PAWS, 06tools-platform-team: [paws] Upgrade images to supported os - https://phabricator.wikimedia.org/T442389#12480482 (10dcaro) [18:04:28] 10Toolforge, 06tools-platform-team: [infra,k8s] Some pods are stuck in Error, preventing new cronjobs from starting - https://phabricator.wikimedia.org/T442169#12480489 (10dcaro) [18:06:21] 10Toolforge, 06tools-platform-team: [builds-api] very slow to response to image list / [components-api] has constant timeouts to api.svc.tools.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T442194#12480501 (10dcaro) [18:06:31] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Bring back toolsbeta - https://phabricator.wikimedia.org/T442516#12480505 (10dcaro) [18:18:07] 10Tool-toolwatch, 06Indic MediaWiki Developers UG, 06Indic-TechCom: Sort tools based on tool Title - https://phabricator.wikimedia.org/T353579#12480534 (10Viserion7) 05Open→03Resolved Thanks @MahimaSinghal Merged at https://github.com/gopavasanth/ToolWatch/pull/19 [18:19:03] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12480537 (10thcipriani) [18:28:03] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12480582 (10thcipriani) [18:35:48] (03CR) 10Jean-Frédéric: [C:03+2] Stop the Toolforge deploy when a step fails [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1358529 (owner: 10Lokal Profil) [18:37:55] (03Merged) 10jenkins-bot: Stop the Toolforge deploy when a step fails [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1358529 (owner: 10Lokal Profil) [19:34:42] 10Striker, 06tools-platform-team: It should be possible to mark tools as redirects/deprecated in Striker/Toolsadmin - https://phabricator.wikimedia.org/T442634 (10AntiCompositeNumber) 03NEW [20:09:19] 10Quarry, 06tools-platform-team: [quarry] update the database password - https://phabricator.wikimedia.org/T440979#12481080 (10github-toolforge-bot) andrewbogott closed https://github.com/toolforge/quarry/pull/105 [20:23:08] 06tools-infrastructure-team: Create temporary secgroupadmin keystone role - https://phabricator.wikimedia.org/T442554#12481154 (10github-toolforge-bot) andrewbogott opened https://github.com/toolforge/quarry/pull/107 [20:27:47] 06tools-infrastructure-team: Create temporary secgroupadmin keystone role - https://phabricator.wikimedia.org/T442554#12481165 (10github-toolforge-bot) andrewbogott closed https://github.com/toolforge/quarry/pull/107 [20:30:40] 10Cloud-VPS (Project-requests): Request creation of globaltemplates VPS project - https://phabricator.wikimedia.org/T442642 (10bvibber) 03NEW [20:32:29] 10Cloud-VPS (Project-requests): Request creation of globaltemplates VPS project - https://phabricator.wikimedia.org/T442642#12481195 (10bvibber) 05Open→03Invalid p:05Triage→03Low [20:34:49] 10Cloud-VPS, 06tools-infrastructure-team, 10CirrusSearch, 06Discovery-Search (2026.09.01 - 2026.10.02): cirrus-integ5.search VM busted, unable to finish booting - https://phabricator.wikimedia.org/T442510#12481200 (10taavi) 05Open→03Resolved Restored the security group rules allowing that traffic a... [20:49:44] 10Cloud-VPS, 06tools-infrastructure-team, 06Content-Platform-Team: wikitextexp: Temporarily bump quote to create a new g4.cores32.ram64.disk20 VM to replace ctt-qa-03 - https://phabricator.wikimedia.org/T442643 (10ssastry) 03NEW [20:50:23] 10Cloud-VPS (Quota-requests), 06tools-infrastructure-team, 06Content-Platform-Team: wikitextexp: Temporarily bump quote to create a new g4.cores32.ram64.disk20 VM to replace ctt-qa-03 - https://phabricator.wikimedia.org/T442643#12481233 (10ssastry) [20:50:53] 10Cloud-VPS (Quota-requests), 06tools-infrastructure-team, 06Content-Platform-Team: wikitextexp: Temporarily bump quote to create a new g4.cores32.ram64.disk20 VM to replace ctt-qa-03 - https://phabricator.wikimedia.org/T442643#12481236 (10ssastry) [20:53:01] 10Cloud-VPS (Quota-requests), 06tools-infrastructure-team, 06Content-Platform-Team: wikitextexp: Temporarily bump quote to create a new g4.cores32.ram64.disk20 VM to replace ctt-qa-03 - https://phabricator.wikimedia.org/T442643#12481240 (10ssastry) [20:53:41] 10Cloud-VPS (Quota-requests), 06tools-infrastructure-team, 06Content-Platform-Team: wikitextexp: Temporarily bump quote to create a new g4.cores32.ram64.disk20 VM to replace ctt-qa-03 - https://phabricator.wikimedia.org/T442643#12481244 (10Andrew) +1 [20:54:43] 10Tool-global-search: Upgrade Global Search to a supported PHP version - https://phabricator.wikimedia.org/T442332#12481246 (10MusikAnimal) 05In progress→03Resolved [20:59:27] 10Cloud-VPS (Quota-requests), 06tools-infrastructure-team, 06Content-Platform-Team: wikitextexp: Temporarily bump quote to create a new g4.cores32.ram64.disk20 VM to replace ctt-qa-03 - https://phabricator.wikimedia.org/T442643#12481255 (10Andrew) Lots of things are broken today (including the quota tool and... [21:02:31] 10Cloud-VPS (Quota-requests), 06tools-infrastructure-team, 06Content-Platform-Team: wikitextexp: Temporarily bump quote to create a new g4.cores32.ram64.disk20 VM to replace ctt-qa-03 - https://phabricator.wikimedia.org/T442643#12481264 (10BLiviero-WMF) +1 [21:44:11] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12481792 (10Southparkfan) [21:48:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [21:53:02] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [21:53:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [21:59:25] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12481891 (10Southparkfan) >>! In T442482#12475803, @Andrew wrote: > Here is the raw output from the cumin run of the audit script on all VMs. Inter... [22:02:49] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12481913 (10Southparkfan) [22:12:24] FIRING: SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:35:48] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12481997 (10Southparkfan) [22:46:04] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12482048 (10thcipriani) [22:54:55] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12482102 (10thcipriani) [22:56:51] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12482104 (10Southparkfan) [22:58:06] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12482107 (10Southparkfan) [23:03:36] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12482120 (10Southparkfan) [23:03:37] 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10Puppet CI: PCC: no facts for Cloud VPS hosts on the cloudinfra puppetserver (codesearch10) - https://phabricator.wikimedia.org/T441067#12482121 (10Dzahn) thanks to taavi for reviewing my patch and setting a realm override in Hiera.... [23:11:12] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12482187 (10Southparkfan) [23:33:40] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12482268 (10Southparkfan) [23:48:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:52:05] (03PS1) 10Dzahn: frontend: disable session.upload_progress.enabled in the image [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1359418 (https://phabricator.wikimedia.org/T268199) [23:53:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:53:19] (03CR) 10Dzahn: [C:03+2] frontend: disable session.upload_progress.enabled in the image [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1359418 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [23:54:00] (03CR) 10Dzahn: "doing the same in the image https://gerrit.wikimedia.org/r/c/labs/codesearch/+/1359418" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1357835 (owner: 10Ladsgroup) [23:54:08] 10Toolforge, 06tools-platform-team: Broken mariadb credentials for globaltemplatestest on Toolforge - https://phabricator.wikimedia.org/T442660 (10bvibber) 03NEW [23:55:07] (03Merged) 10jenkins-bot: frontend: disable session.upload_progress.enabled in the image [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1359418 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn)