[00:06:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:11:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:19:38] 10Cloud-VPS, 06tools-infrastructure-team: Restore the mediawiki-quickstart Cloud VPS project after PHP2Shell - https://phabricator.wikimedia.org/T442661 (10Mhurd) 03NEW [00:20:34] (03update) 10renovatebot: Update dependency @types/node to v26 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/37 [00:20:36] (03open) 10renovatebot: Update dependency @types/node to v26 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/37 [00:20:50] (03update) 10renovatebot: Update dependency @types/node to v26 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/46 [00:20:51] (03open) 10renovatebot: Update dependency @types/node to v26 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/46 [00:21:06] (03update) 10renovatebot: Update dependency @types/node to v26 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/44 [00:21:10] (03open) 10renovatebot: Update dependency @types/node to v26 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/44 [00:35:44] 10Cloud-VPS, 06tools-infrastructure-team: please enable wikiapiary - https://phabricator.wikimedia.org/T442662 (10MarkAHershberger) 03NEW [01:03:32] 10Striker, 06tools-platform-team: It should be possible to mark tools as redirects/deprecated in Striker/Toolsadmin - https://phabricator.wikimedia.org/T442634#12482513 (10bd808) > but since Toolhub doesn't have complete coverage of Toolforge tools or up-to-date maintainer information Toolhub has data on all... [01:50:18] 10Cloud-VPS, 06tools-infrastructure-team, 10Catalyst, 06Release-Engineering-Team: Restore ingress to patchdemo / catalyst - https://phabricator.wikimedia.org/T442664 (10brennen) 03NEW [01:53:02] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [02:12:24] FIRING: SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:32:31] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40621021123 (https://github.com/cluebotng/component-configs/commits/e68d7e50924b4c26fda7a56ae5fbfb784bdf14bf) [02:32:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [02:32:54] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40621021329 (https://github.com/cluebotng/component-configs/commits/e68d7e50924b4c26fda7a56ae5fbfb784bdf14bf) [02:32:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [02:57:52] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12482631 (10Raymond_Ndibe) @komla we can move this out of draft [02:58:56] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create code change and redeploy tutorial - https://phabricator.wikimedia.org/T437900#12482632 (10Raymond_Ndibe) yeaa we can also move this out of draft @komla [03:06:29] (03update) 10raymond-ndibe: Draft: feat: support jobs exec [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/368 (https://phabricator.wikimedia.org/T431276) [03:06:55] (03update) 10raymond-ndibe: Draft: feat: support jobs exec [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/174 (https://phabricator.wikimedia.org/T431276) [03:07:41] (03update) 10raymond-ndibe: feature: support jobs exec [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/174 (https://phabricator.wikimedia.org/T431276) [03:07:50] (03update) 10raymond-ndibe: feature: support jobs exec [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/174 (https://phabricator.wikimedia.org/T431276) [03:08:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [03:08:21] (03update) 10raymond-ndibe: feature: support jobs exec [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/368 (https://phabricator.wikimedia.org/T431276) [03:08:48] (03update) 10raymond-ndibe: feature: support jobs exec [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/368 (https://phabricator.wikimedia.org/T431276) [03:09:21] (03update) 10raymond-ndibe: feature: support interactive shell sessions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/368 (https://phabricator.wikimedia.org/T431276) [03:09:33] (03update) 10raymond-ndibe: feature: support interactive shell sessions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/368 (https://phabricator.wikimedia.org/T431276) [03:11:06] (03update) 10raymond-ndibe: feature: support interactive shell sessions [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/174 (https://phabricator.wikimedia.org/T431276) [03:12:08] 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api, jobs-cli: investigate shell / exec into jobs-api containers - https://phabricator.wikimedia.org/T431276#12482668 (10Raymond_Ndibe) >>! In T431276#12167622, @taavi wrote: > Dupe of {T311917}? it's a bit different. This is about adding it to jobs-api [03:12:15] 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api, jobs-cli: investigate shell / exec into jobs-api containers - https://phabricator.wikimedia.org/T431276#12482670 (10Raymond_Ndibe) 05Open→03In progress [03:13:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [03:15:09] (03update) 10raymond-ndibe: feature: support interactive shell sessions [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/174 (https://phabricator.wikimedia.org/T431276) [03:50:21] (03open) 10raymond-ndibe: nginx: add websocket support [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/110 (https://phabricator.wikimedia.org/T431276) [03:50:48] (03update) 10raymond-ndibe: nginx: add websocket support [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/110 (https://phabricator.wikimedia.org/T431276) [04:33:12] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40632642704 (https://github.com/cluebotng/component-configs/commits/611facf7667c39a7f5b1712e5b174684a0649ea4) [04:38:22] (03update) 10raymond-ndibe: feature: support interactive shell sessions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/368 (https://phabricator.wikimedia.org/T431276) [05:18:21] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40636879022 (https://github.com/cluebotng/component-configs/commits/1519387bf38d5ffaec5380641581f2b9f842af99) [05:18:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [05:18:34] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40636878933 (https://github.com/cluebotng/component-configs/commits/1519387bf38d5ffaec5380641581f2b9f842af99) [05:18:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [05:19:08] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40636879060 (https://github.com/cluebotng/component-configs/commits/1519387bf38d5ffaec5380641581f2b9f842af99) [05:19:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [05:41:30] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40638987512 (https://github.com/cluebotng/component-configs/commits/ed84094285672ba9b9621030959dffcfe3f2fcbc) [05:41:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [05:42:20] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40638987481 (https://github.com/cluebotng/component-configs/commits/ed84094285672ba9b9621030959dffcfe3f2fcbc) [05:42:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [05:53:02] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [05:54:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [05:59:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:08:23] (03update) 10raymond-ndibe: feature: support interactive shell sessions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/368 (https://phabricator.wikimedia.org/T431276) [06:12:25] FIRING: SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:22:49] (03update) 10raymond-ndibe: feature: support interactive shell sessions [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/174 (https://phabricator.wikimedia.org/T431276) [06:30:10] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40643749649 (https://github.com/cluebotng/component-configs/commits/6aa169a5ad67427223a008319e43e00be9db16c5) [06:30:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [06:33:48] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Remove or replace "default" web service image - https://phabricator.wikimedia.org/T401715#12482896 (10GPSLeo) I think there should be an easy to use default that only supports static html/js sites without any server component. [06:47:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:52:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:54:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:59:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:05:11] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [jobs] Allow configuration of Promethus scraping of a specific endpoint for publication in grafana.wmcloud.org - https://phabricator.wikimedia.org/T362012#12482926 (10fgiunchedi) [07:54:07] FIRING: HarborDown: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [08:04:10] 10Cloud-VPS, 06tools-infrastructure-team: please enable wikiapiary - https://phabricator.wikimedia.org/T442662#12482993 (10taavi) Hello! Your project still has various anchient PHP versions installed (as well as an unpatched version of 8.5): ` binary /usr/bin/php7.4 PHP 7.4.33 (cli) (built: Jul 3 2026 10:36:... [08:05:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [08:09:14] 10Cloud-VPS, 06tools-infrastructure-team: Restore the mediawiki-quickstart Cloud VPS project after PHP2Shell - https://phabricator.wikimedia.org/T442661#12482999 (10taavi) 05Open→03Resolved a:03taavi Done. [08:10:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [08:13:18] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: 2026-10-21 Toolforge and Cloud VPS down - https://phabricator.wikimedia.org/T441908#12483010 (10Aklapper) [08:17:25] 10Cloud-VPS, 06tools-infrastructure-team, 10Catalyst, 06Release-Engineering-Team: Restore ingress to patchdemo / catalyst - https://phabricator.wikimedia.org/T442664#12483035 (10taavi) I cannot look further at the moment why, but our audit script is flagging the presence of an outdated PHP 8.2 version on C... [08:25:31] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Remove or replace "default" web service image - https://phabricator.wikimedia.org/T401715#12483045 (10Tacsipacsi) That would actually make sense. I argued against just bumping the result in T442199, since that would mean that we have the same problem ev... [08:44:28] 10VPS-project-Extdist, 10ExtensionDistributor, 13Patch-For-Review: Polish outdated issues in labs/tools/extdist - https://phabricator.wikimedia.org/T442167#12483079 (10hashar) 05Open→03Resolved [08:54:20] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788301 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd) [08:54:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [08:54:31] !log tools.toolforge-functional-runner Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788311 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd) [08:54:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolforge-functional-runner/SAL [08:54:43] !log tools.cluebotng-staging Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788367 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd) [08:54:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [08:54:55] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788418 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd) [08:54:57] !log tools.cluebotng-trainer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788352 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd) [08:54:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [08:54:58] !log tools.cluebot-syncer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788309 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd) [08:55:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [08:55:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [08:55:20] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788312 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd) [08:55:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [08:55:46] !log tools.cluebotng-monitoring Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788397 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd) [08:55:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [08:56:01] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788346 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd) [08:56:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [09:39:49] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40665456675 (https://github.com/cluebotng/component-configs/commits/cbd8bb224c15c6dacdf93632ef29a11586a0b50b) [09:39:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [09:53:02] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [09:53:19] 10Toolforge, 06tools-platform-team: [maintain-dbusers] new credentials are not working - https://phabricator.wikimedia.org/T442683 (10dcaro) 03NEW [09:53:56] 10PAWS, 06tools-platform-team: [maintain-dbusers] getting errors when listing paws users - https://phabricator.wikimedia.org/T442684 (10dcaro) 03NEW [09:54:19] 10Toolforge, 06tools-platform-team: [maintain-dbusers] new credentials are not working - https://phabricator.wikimedia.org/T442683#12483323 (10dcaro) [09:54:20] 10Toolforge, 06tools-platform-team: Broken mariadb credentials for globaltemplatestest on Toolforge - https://phabricator.wikimedia.org/T442660#12483324 (10dcaro) [10:12:25] FIRING: SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:24:21] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [maintain-dbusers] new credentials are not working - https://phabricator.wikimedia.org/T442683#12483446 (10dcaro) 05Open→03In progress p:05Triage→03High a:03dcaro [10:24:47] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [maintain-dbusers] new credentials are not working - https://phabricator.wikimedia.org/T442683#12483451 (10dcaro) Manually deployed in coludcontrol1007, accounts are being created again. [10:25:00] 10Toolforge, 06tools-platform-team: Broken mariadb credentials for globaltemplatestest on Toolforge - https://phabricator.wikimedia.org/T442660#12483453 (10dcaro) 05Open→03In progress p:05Triage→03High a:03dcaro [10:25:12] 10PAWS, 06tools-platform-team: [maintain-dbusers] getting errors when listing paws users - https://phabricator.wikimedia.org/T442684#12483459 (10dcaro) 05Open→03In progress [10:25:22] 10PAWS, 06tools-platform-team: [maintain-dbusers] getting errors when listing paws users - https://phabricator.wikimedia.org/T442684#12483460 (10dcaro) p:05Triage→03High a:03dcaro [10:25:52] 10PAWS, 06tools-platform-team: [maintain-dbusers] getting errors when listing paws users - https://phabricator.wikimedia.org/T442684#12483462 (10dcaro) Manually enabled the port 80 for cloudcontrols (copied the ip ranges from the tools sec group) on the nfs sec group for paws [10:29:02] 10Toolforge, 06tools-platform-team: [maintain-dbusers] add alert to flag for account population errors - https://phabricator.wikimedia.org/T442688 (10dcaro) 03NEW [10:40:53] 10Cloud-VPS, 06tools-infrastructure-team: wmcs-backup failure on cloudbackup2003 - https://phabricator.wikimedia.org/T442690 (10fgiunchedi) 03NEW [10:47:08] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [maintain-dbusers] new credentials are not working - https://phabricator.wikimedia.org/T442683#12483582 (10dcaro) Related to {T442684} [10:48:36] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [maintain-dbusers] add alert to flag for account population errors - https://phabricator.wikimedia.org/T442688#12483586 (10dcaro) p:05Triage→03High [10:58:56] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [maintain-dbusers] add alert to flag for account population errors - https://phabricator.wikimedia.org/T442688#12483627 (10dcaro) 05Open→03In progress Added to [[ https://grafana-rw.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-... [11:06:31] 10Toolforge, 06tools-platform-team: Broken mariadb credentials for globaltemplatestest on Toolforge - https://phabricator.wikimedia.org/T442660#12483675 (10dcaro) 05In progress→03Resolved [11:07:13] 10Tool-spacemedia, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Update Toolforge Deploy to Heroku 24_0.21.20+ to get openjdk 27 support - https://phabricator.wikimedia.org/T439154#12483679 (10dcaro) I think this might have gotten fixed with the last update (we should double check) [11:12:08] 10Toolforge, 06tools-platform-team: Heads-up: A lot of traffic in one of my tools - https://phabricator.wikimedia.org/T442544#12483706 (10dcaro) Thanks for heads up! We have a per-tool limiting in place that should avoid affecting other tools, there's also an effort for adding (opt-in for now) scraper blockin... [11:13:18] 10Toolforge, 06tools-platform-team: Heads-up: A lot of traffic in one of my tools - https://phabricator.wikimedia.org/T442544#12483712 (10dcaro) 05Open→03Resolved p:05Triage→03Low a:03dcaro Consider us informed :), I'll close this. Let us know though if you are having other unexpected issues or... [11:21:50] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [11:27:41] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [maintain-dbusers] new credentials are not working - https://phabricator.wikimedia.org/T442683#12483784 (10dcaro) 05In progress→03Resolved [11:27:46] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [maintain-dbusers] add alert to flag for account population errors - https://phabricator.wikimedia.org/T442688#12483787 (10dcaro) 05In progress→03Resolved [11:27:48] 10PAWS, 06tools-platform-team: [maintain-dbusers] getting errors when listing paws users - https://phabricator.wikimedia.org/T442684#12483790 (10dcaro) 05In progress→03Resolved [11:28:27] 10Quarry, 06tools-platform-team: [quarry] update the database password - https://phabricator.wikimedia.org/T440979#12483792 (10dcaro) 05In progress→03Resolved [11:28:36] 10PAWS, 06tools-platform-team: [paws] Upgrade images to supported os - https://phabricator.wikimedia.org/T442389#12483795 (10dcaro) 05In progress→03Resolved [11:29:26] 10PAWS, 06tools-platform-team: [paws] Upgrade images to supported os - https://phabricator.wikimedia.org/T442389#12483800 (10dcaro) New cluster up and running, old clusters deleted, there might be other minor errors happening due to the jupyterhub version change but closing this for now. [11:36:19] 10Toolforge, 06tools-platform-team: [all] rebuild containers to pull updated packages - https://phabricator.wikimedia.org/T442696 (10dcaro) 03NEW [11:36:54] 10Toolforge, 06tools-platform-team: [all] rebuild containers to pull updated packages - https://phabricator.wikimedia.org/T442696#12483827 (10dcaro) [11:38:24] 10Toolforge, 06tools-platform-team: [harbor,o11y] The harbor down alert on tools keeps triggering - https://phabricator.wikimedia.org/T442697 (10dcaro) 03NEW [11:38:33] 10Toolforge, 06tools-platform-team: [harbor,o11y] The harbor down alert on tools keeps triggering - https://phabricator.wikimedia.org/T442697#12483858 (10dcaro) [11:47:17] FIRING: JobUnavailable: Reduced availability for job pdns_rec in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:47:44] PROBLEM - Host clouddb1022 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:22] FIRING: [2x] ProbeDown: Service dumps-https:443 has failed probes (http_dumps-https_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:49:22] FIRING: [2x] HAProxyBackendUnavailable: HAProxy service wikireplica-db-web-s3 backend clouddb1022.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [11:49:22] FIRING: [2x] ProbeDown: Service dumps-https:443 has failed probes (http_dumps-https_ip4) #page - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:50:19] FIRING: [10x] ProbeDown: Service clouddumps1001:443 has failed probes (http_dumps_wikimedia_org_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:50:39] FIRING: [5x] ProbeDown: Service virt.cloudgw.codfw1dev.wikimediacloud.org:0 has failed probes (icmp_virt_cloudgw_codfw1dev_wikimediacloud_org_from_eqiad_ip4) - https://wikitech.wikimedia.org/wiki/Network_monitoring#ProbeDown - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:59:34] RESOLVED: [2x] ProbeDown: Service dumps-https:443 has failed probes (http_dumps-https_ip4) #page - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:01:07] RESOLVED: [7x] JobUnavailable: Reduced availability for job ceph_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:01:42] RESOLVED: [3x] ProbeDown: Service dumps-https:443 has failed probes (http_dumps-https_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:02:12] RESOLVED: [2x] HAProxyBackendUnavailable: HAProxy service wikireplica-db-web-s3 backend clouddb1022.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [12:02:47] RESOLVED: [12x] ProbeDown: Service clouddumps1001:443 has failed probes (http_dumps_wikimedia_org_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:21:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [12:33:28] 10Cloud-VPS, 06tools-infrastructure-team: Reactivate wlm-it-visual - https://phabricator.wikimedia.org/T442473#12484019 (10Ysogo) @fgiunchedi we have completed upgrade of php, can you please rescan and crosscheck? Thank you [12:40:22] 10Tool-wikinewsie: More user-generated metadata for Wikinewsie headlines - https://phabricator.wikimedia.org/T435237#12484036 (10Pharos) Another possible application of this feature could be for T182076 in the Top Read widget. [13:20:48] 10Cloud-VPS, 06tools-infrastructure-team: Reactivate wlm-it-visual - https://phabricator.wikimedia.org/T442473#12484280 (10Volans) @Ysogo, great, I had a quick look but there are still some unpatched PHPs in `wlm-staging-app.wlm-it-visual.eqiad1.wikimedia.cloud`, if not needed could you purge them? ` layer-pa... [13:32:30] 10Cloud-VPS, 06tools-infrastructure-team: Reactivate wlm-it-visual - https://phabricator.wikimedia.org/T442473#12484318 (10Ysogo) Checking. Keep you posted. [13:34:52] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40697046893 (https://github.com/cluebotng/component-configs/commits/65c2635e0eca77c926d6ecfdd3bc4243290cb2de) [13:34:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [13:41:00] 10Toolforge, 06tools-platform-team: [all] remove mentions and usages of non-php8 pre-built images - https://phabricator.wikimedia.org/T442716 (10dcaro) 03NEW [13:41:48] 10Toolforge, 06tools-platform-team: [all] remove mentions and usages of non-php8 pre-built images - https://phabricator.wikimedia.org/T442716#12484405 (10dcaro) [13:44:09] 10PAWS, 06tools-platform-team: PAWS should not show update notifications for users - https://phabricator.wikimedia.org/T442717 (10taavi) 03NEW [13:45:05] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40698633627 (https://github.com/cluebotng/component-configs/commits/29d35958a8230b19c67327fae4cc1a2c55f8f4f0) [13:45:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [13:53:02] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [13:55:00] (03merge) 10taavi: Remove default webservice type entirely [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/127 (https://phabricator.wikimedia.org/T401715) [13:55:31] FIRING: Primary cloud switch port utilisation over 90%: Alert for device cloudsw1-c8-eqiad.mgmt.eqiad.wmnet - Primary cloud switch port utilisation over 90% - https://alerts.wikimedia.org/?q=alertname%3DPrimary+cloud+switch+port+utilisation+over+90%25 [13:57:42] (03open) 10taavi: d/changelog: bump to 0.103.28 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/128 (https://phabricator.wikimedia.org/T401715) [13:59:48] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Bring back toolsbeta - https://phabricator.wikimedia.org/T442516#12484524 (10dcaro) p:05Triage→03High a:03dcaro [13:59:55] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Bring back toolsbeta - https://phabricator.wikimedia.org/T442516#12484527 (10dcaro) 05Open→03In progress [14:00:03] 10Cloud-VPS, 06tools-infrastructure-team: Reactivate wlm-it-visual - https://phabricator.wikimedia.org/T442473#12484534 (10Ysogo) @fgiunchedi We have just removed old mediwiki images which were no longer required. Can you please rescan and crosscheck? [14:00:31] RESOLVED: Primary cloud switch port utilisation over 90%: Device cloudsw1-c8-eqiad.mgmt.eqiad.wmnet recovered from Primary cloud switch port utilisation over 90% - https://alerts.wikimedia.org/?q=alertname%3DPrimary+cloud+switch+port+utilisation+over+90%25 [14:08:12] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component toolforge-webservice [14:08:16] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-webservice [14:08:24] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [14:09:11] (03open) 10r4356th: Replace timestamps and disable animations when taking screenshots [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/43 (https://phabricator.wikimedia.org/T415882) [14:09:13] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli [14:10:59] 10Cloud-VPS, 06tools-infrastructure-team: Reactivate wlm-it-visual - https://phabricator.wikimedia.org/T442473#12484588 (10Volans) @Ysogo great, all good, I'll proceed with restoring ingress traffic. FYI the procedure includes a reboot of the VMs to ensure nothing got stuck in a bad state while the traffic was... [14:13:40] (03open) 10taavi: Support 'webservice status' without a type [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/129 (https://phabricator.wikimedia.org/T401715) [14:14:30] (03update) 10taavi: Support 'webservice status' without a type [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/129 (https://phabricator.wikimedia.org/T401715) [14:15:35] 10Toolforge, 06tools-platform-team: [all] remove mentions and usages of non-php8 pre-built images - https://phabricator.wikimedia.org/T442716#12484608 (10taavi) [14:15:37] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Remove or replace "default" web service image - https://phabricator.wikimedia.org/T401715#12484607 (10taavi) [14:16:19] (03PS2) 10Klausman: hiera/k8s: Add pseudo-secrets stanza for llm-batch namespace on LW [labs/private] - 10https://gerrit.wikimedia.org/r/1359864 (https://phabricator.wikimedia.org/T442150) [14:16:29] (03CR) 10Klausman: [V:03+2 C:03+2] hiera/k8s: Add pseudo-secrets stanza for llm-batch namespace on LW [labs/private] - 10https://gerrit.wikimedia.org/r/1359864 (https://phabricator.wikimedia.org/T442150) (owner: 10Klausman) [14:18:17] (03approved) 10dcaro: Support 'webservice status' without a type [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/129 (https://phabricator.wikimedia.org/T401715) (owner: 10taavi) [14:18:34] (03merge) 10taavi: Support 'webservice status' without a type [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/129 (https://phabricator.wikimedia.org/T401715) [14:19:19] (03update) 10taavi: d/changelog: bump to 0.103.28 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/128 (https://phabricator.wikimedia.org/T401715) [14:21:04] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [14:21:48] 10Cloud-VPS, 06tools-infrastructure-team: Reactivate wlm-it-visual - https://phabricator.wikimedia.org/T442473#12484644 (10Volans) 05Open→03Resolved a:03Volans All done. Everything should be back to normal operations. Resolving this task, feel free to re-open it in case something is amiss. [14:22:29] !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli [14:22:58] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Bring back toolsbeta - https://phabricator.wikimedia.org/T442516#12484656 (10dcaro) all VMs are up and security groups restored, doing some checks before closing [14:25:23] (03open) 10taavi: functional-tests: Update to use modern PHP versions [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1480 (https://phabricator.wikimedia.org/T401715 https://phabricator.wikimedia.org/T442716) [14:25:59] (03approved) 10dcaro: functional-tests: Update to use modern PHP versions [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1480 (https://phabricator.wikimedia.org/T401715 https://phabricator.wikimedia.org/T442716) (owner: 10taavi) [14:27:07] (03merge) 10taavi: functional-tests: Update to use modern PHP versions [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1480 (https://phabricator.wikimedia.org/T401715 https://phabricator.wikimedia.org/T442716) [14:27:12] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [14:28:04] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli [14:28:23] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [14:29:11] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli [14:30:51] (03update) 10r4356th: Replace timestamps and disable animations when taking screenshots [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/43 (https://phabricator.wikimedia.org/T415882) [14:30:57] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [14:32:02] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli [14:32:47] (03update) 10r4356th: Replace timestamps and disable animations when taking screenshots [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/43 (https://phabricator.wikimedia.org/T415882) [14:33:01] (03merge) 10taavi: d/changelog: bump to 0.103.28 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/128 (https://phabricator.wikimedia.org/T401715) [14:33:49] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Remove or replace "default" web service image - https://phabricator.wikimedia.org/T401715#12484747 (10taavi) 05Open→03Resolved a:03taavi [14:35:15] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Remove or replace "default" web service image - https://phabricator.wikimedia.org/T401715#12484758 (10taavi) Please see T241817 / T435088 / T442122 for the static app use case. [14:36:11] (03update) 10r4356th: Replace timestamps and disable animations when taking screenshots [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/43 (https://phabricator.wikimedia.org/T415882) [14:40:37] (03update) 10r4356th: Replace timestamps and disable animations when taking screenshots [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/43 (https://phabricator.wikimedia.org/T415882) [14:42:11] (03merge) 10r4356th: Replace timestamps and disable animations when taking screenshots [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/43 (https://phabricator.wikimedia.org/T415882) [14:44:01] 10Tool-delintbot, 13Patch-For-Review: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12484807 (10Redmin) [14:44:34] 10Tool-delintbot: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12484810 (10Redmin) [14:45:38] (03update) 10dcaro: builds-builder - set insecureRegistries [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1479 (owner: 10damian) [14:45:40] (03update) 10dcaro: builds-builder - set insecureRegistries [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1479 (owner: 10damian) [14:45:41] (03approved) 10dcaro: builds-builder - set insecureRegistries [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1479 (owner: 10damian) [14:46:21] (03merge) 10dcaro: builds-builder - set insecureRegistries [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1479 (owner: 10damian) [14:57:12] 10Tool-inteGraality, 06Wikidata Platform Team: Reference column drill-down queries time out on QLever - https://phabricator.wikimedia.org/T435101#12484870 (10trueg) @Hannah_Bast , could you please have a look at this performance issue related to nested `NOT EXISTS` filters. Is that a known issue, it is expected? [15:00:35] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Bring back toolsbeta - https://phabricator.wikimedia.org/T442516#12484897 (10dcaro) 05In progress→03Resolved [15:01:39] 10Tool-inteGraality, 06Wikidata Platform Team: Reference column drill-down queries time out on QLever - https://phabricator.wikimedia.org/T435101#12484910 (10BTracy-WMF) Hi @JeanFred , thank you for reporting the issue. We will take this to our partners in QLever for further investigation. In the meantime, I... [15:08:21] 10Cloud-VPS, 06tools-infrastructure-team: Reactivate wlm-it-visual - https://phabricator.wikimedia.org/T442473#12484938 (10Ysogo) @fgiunchedi so far, so good. Thank you for the swift support. [15:15:06] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate pcc-db1001.puppet-diffs.eqiad1.wikimedia.cloud is about to expire in 27d 23h 58m 34s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [15:39:15] 10Openstack-Magnum, 10Quarry, 06tools-infrastructure-team: Rebuild quarry with the new magnum-cluster-api driver - https://phabricator.wikimedia.org/T437501#12485089 (10Andrew) 05Open→03Resolved Now this is actually done. I think. [15:40:39] 06cloud-services-team, 10PAWS, 06tools-platform-team: replace ingress-nginx in PAWS - https://phabricator.wikimedia.org/T427848#12485098 (10Andrew) 05Open→03Resolved a:03dcaro After procrastinating on this for ages, David fixed it in a few hours as part of the php2shell response. Thank you @dcaro! [15:41:15] 10Openstack-Magnum, 06tools-infrastructure-team: Replace all heat-based magnum clusters - https://phabricator.wikimedia.org/T437496#12485107 (10Andrew) [15:50:18] 10Cloud-VPS (Debian Bullseye Deprecation), 10SRE-swift-storage: Migrate swift away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435783#12485164 (10komla) 05Open→03Resolved I've marked this as closed. Thanks! [15:53:46] 10Tool-spacemedia, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Update Toolforge Deploy to Heroku 24_0.21.20+ to get openjdk 27 support - https://phabricator.wikimedia.org/T439154#12485184 (10Don-vip) >>! In T439154#12483679, @dcaro wrote: > I think this might have gotten fixed with the last upd... [15:54:07] RESOLVED: HarborDown: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [15:55:34] 10Toolforge, 06tools-platform-team: [harbor,o11y] The harbor down alert on tools keeps triggering - https://phabricator.wikimedia.org/T442697#12485190 (10dcaro) From irc: > taavi> Taavi Väänänen welp, I found why the harbor alerts have been firing > target=https://tools-harbor.wmcloud.org/api/v2.0/ping err="... [15:56:14] 10Toolforge, 06tools-platform-team: [harbor,o11y] The harbor down alert on tools keeps triggering - https://phabricator.wikimedia.org/T442697#12485192 (10dcaro) 05Open→03Resolved p:05Triage→03High a:03taavi [16:08:22] 10Cloud-VPS, 06tools-infrastructure-team, 06Wikibase Suite Team, 10Wikidata Vector Database: Re-enable wikidata-dev cloud vps project - https://phabricator.wikimedia.org/T442730 (10WMDE-leszek) 03NEW [16:10:39] 10Cloud-VPS, 06tools-infrastructure-team, 06Wikibase Suite Team, 10Wikidata Vector Database: Re-enable wikidata-dev cloud vps project - https://phabricator.wikimedia.org/T442730#12485266 (10taavi) Hello! `wd-embedding.wikidata-dev.eqiad1.wikimedia.cloud` seems to be running an outdated MediaWiki container... [16:12:13] (03open) 10andrew: Update magnum default templates to use the latest worker image [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/369 [16:15:18] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/369 [16:15:19] 10Cloud-Services, 06tools-infrastructure-team, 10Pywikibot, 07Pywikibot-tests: WMCS SOCKS tunnel establishment fails for all Pywikibot GitHub Actions - https://phabricator.wikimedia.org/T442175#12485320 (10MGChecker) > An explicit exception for GitHub Actions has been granted, with conditions. See T260746... [16:15:48] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/369 [16:16:10] (03merge) 10andrew: Update magnum default templates to use the latest worker image [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/369 [16:16:18] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [16:16:23] (03open) 10taavi: eqiad1: paws: Manage default security group [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/370 [16:16:36] (03update) 10taavi: eqiad1: paws: Manage default security group [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/370 [16:16:57] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [16:17:46] 10Toolforge, 06tools-platform-team: [toolsdb] Upgrade to MariaDB 10.11 - https://phabricator.wikimedia.org/T434679#12485358 (10bvibber) This is blocking our use of a MediaWiki installation on Toolforge for Global Templates testing; if it's going to take time to fix I'll have to move to a VPS instead and manage... [16:18:00] 10Striker, 06tools-platform-team, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12485354 (10Milimetric) a:03tchin [16:18:22] 10Striker, 06tools-platform-team, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12485356 (10Milimetric) [16:18:28] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [16:18:57] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [16:19:37] 10Cloud-VPS (Project-requests): Request creation of globaltemplates VPS project - https://phabricator.wikimedia.org/T442642#12485376 (10bvibber) 05Invalid→03Open p:05Low→03Triage [16:20:26] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [16:20:53] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [16:22:09] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [16:22:35] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [16:24:02] (03approved) 10dcaro: eqiad1: paws: Manage default security group [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/370 (owner: 10taavi) [16:25:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [16:45:51] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [16:46:24] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [16:49:15] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/370 [16:49:30] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/370 [16:51:45] 10Toolforge, 06tools-platform-team: Remove or replace "default" web service image - https://phabricator.wikimedia.org/T401715#12485618 (10Tacsipacsi) >>! In T401715#12484747, @taavi wrote: > Please see T241817 / T435088 / T442122 for the static app use case. Which one should I see? ☺ (Maybe the two open t... [16:55:23] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/370 [16:55:39] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/370 [16:56:27] (03merge) 10taavi: eqiad1: paws: Manage default security group [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/370 [16:56:50] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [16:57:56] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [16:59:57] 10Striker, 06tools-platform-team, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12485671 (10Jdforrester-WMF) This is probably not all being done by @tchin for all ~22 outstanding services across 10 teams. Maybe you wan... [17:01:34] 10Striker, 06tools-platform-team, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12485675 (10taavi) [17:09:15] 10Cloud-VPS, 06tools-infrastructure-team: Reactivate wlm-it-visual - https://phabricator.wikimedia.org/T442473#12485726 (10fgiunchedi) @Ysogo you are welcome! I wish I could take credit, @Volans did all the work (thank you !) [17:12:14] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [17:13:58] (03open) 10r4356th: Fix pwrap-bug-workaround errors [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/44 (https://phabricator.wikimedia.org/T418464) [17:14:29] (03merge) 10r4356th: Fix pwrap-bug-workaround errors [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/44 (https://phabricator.wikimedia.org/T418464) [17:14:49] 10Tool-delintbot, 13Patch-For-Review: Fix pwrap-bug-workaround errors - https://phabricator.wikimedia.org/T418464#12485740 (10Redmin) 05Open→03Resolved a:03Redmin [17:16:27] 10Toolforge, 06tools-platform-team: [infra,k8s] Some pods are stuck in Error, preventing new cronjobs from starting - https://phabricator.wikimedia.org/T442169#12485748 (10dcaro) @Superyetkin is this fixed for you? I don't see any stuck pods on your superyetkin tool namespace [17:31:24] 10Cloud-Services, 06tools-infrastructure-team, 10Pywikibot, 07Pywikibot-tests: WMCS SOCKS tunnel establishment fails for all Pywikibot GitHub Actions - https://phabricator.wikimedia.org/T442175#12485808 (10taavi) It is not clear what is actually broken here. Toolforge and Cloud VPS bastions are up and have... [17:31:49] 10Tool-delintbot: Fix pwrap-bug-workaround errors - https://phabricator.wikimedia.org/T418464#12485812 (10Redmin) [17:31:58] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12485815 (10Volans) If it helps I've run the audit for the whole project just now and those are the results: ` # With PHP unable to check the vers... [17:38:25] 10Tool-delintbot: Store data about blocked edit attempts - https://phabricator.wikimedia.org/T440299#12485843 (10Redmin) The dbname column turned out to be too short for `zh_min_nan_wiktionary` so I just increased increased its limit. ` ALTER TABLE blocked_attempts MODIFY dbname VARCHAR(21); Query OK, 0 rows aff... [17:39:15] 10Tool-delintbot: Store data about blocked edit attempts - https://phabricator.wikimedia.org/T440299#12485844 (10Redmin) p:05Medium→03High [17:42:30] 10PAWS, 06tools-platform-team: [paws] make sure we can deploy in codfw1dev - https://phabricator.wikimedia.org/T442743 (10dcaro) 03NEW [17:46:24] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12485893 (10taavi) >>! In T442482#12485815, @Volans wrote: > If it helps I've run the audit for the whole project just now and those are the result... [17:47:21] 10Cloud-VPS, 06tools-infrastructure-team, 10Catalyst, 06Release-Engineering-Team: Restore ingress to patchdemo / catalyst - https://phabricator.wikimedia.org/T442664#12485897 (10hashar) [17:47:52] 10Tool-delintbot: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12485898 (10Redmin) p:05Medium→03Low [17:48:45] 10Tool-delintbot: Mention what was fixed in the edit summary - https://phabricator.wikimedia.org/T416068#12485919 (10Redmin) p:05Medium→03High [17:53:02] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [17:57:43] 10Striker, 06tools-platform-team, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12485957 (10karapayneWMDE) [18:06:54] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40741508078 (https://github.com/cluebotng/component-configs/commits/af850f315b5d274b83e396634fbb4577f83c0c60) [18:06:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [18:07:00] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40741507911 (https://github.com/cluebotng/component-configs/commits/af850f315b5d274b83e396634fbb4577f83c0c60) [18:07:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [18:07:57] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40741507977 (https://github.com/cluebotng/component-configs/commits/af850f315b5d274b83e396634fbb4577f83c0c60) [18:07:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [18:16:24] 10Cloud-VPS, 06tools-infrastructure-team, 06Test Platform (San Miguel de Allende 31): Restore the mediawiki-quickstart Cloud VPS project after PHP2Shell - https://phabricator.wikimedia.org/T442661#12486045 (10Mhurd) [18:22:07] 10Tool-wikinewsie, 10GitLab (Account Approval): Requesting GitLab account activation for Wikinewsie Columbia students - https://phabricator.wikimedia.org/T442748 (10Pharos) 03NEW [18:23:52] 10Tool-wikinewsie, 10GitLab (Account Approval): Requesting GitLab account activation for Wikinewsie Columbia students - https://phabricator.wikimedia.org/T442748#12486103 (10Pharos) [18:33:38] 10Cloud-VPS, 06tools-infrastructure-team, 06Wikibase Suite Team, 10Wikidata Vector Database: Re-enable wikidata-dev cloud vps project - https://phabricator.wikimedia.org/T442730#12486139 (10philippe.saade.WMDE) Thank you for flagging this. We just updated MediaWiki to 1.46.2 with PHP 8.3.36 [18:50:17] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40748217679 (https://github.com/cluebotng/component-configs/commits/eceec121315d1ae29499bd0543ff1a373edb1efe) [18:50:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [18:50:51] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40748217792 (https://github.com/cluebotng/component-configs/commits/eceec121315d1ae29499bd0543ff1a373edb1efe) [18:50:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [18:51:45] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12486204 (10Volans) @taavi I trust your knowledge of those services more than the audit script 😉 if you run it without the `-s` it will tell you wh... [18:57:47] 10Cloud-VPS, 06tools-infrastructure-team, 10Catalyst, 06Release-Engineering-Team, 13Patch-For-Review: Restore ingress to patchdemo / catalyst - https://phabricator.wikimedia.org/T442664#12486233 (10brennen) We deleted a lingering failed job and some old images from the registry. We think the script shoul... [19:13:08] 10Tool-delintbot: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#12486260 (10Redmin) [19:18:14] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40752366984 (https://github.com/cluebotng/component-configs/commits/b792020c142bd09c08d70ae0f40684c9c3f81079) [19:18:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [19:40:07] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40755839963 (https://github.com/cluebotng/component-configs/commits/43e1fcf7f4599d40cc9b9bd87abfeba5baede79b) [19:40:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [20:11:15] (03merge) 10sadrettin: Bir ilçedeki aynı türden anıtları birlikte öner [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/7 (owner: 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7) [21:20:14] 10Cloud-VPS, 06tools-infrastructure-team, 06Test Platform (San Miguel de Allende 31): Restore the pixel Cloud VPS project after PHP2Shell - https://phabricator.wikimedia.org/T442770 (10Mhurd) 03NEW [21:45:02] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12486969 (10Southparkfan) @Volans @taavi Thanks for your help. >>! In T442482#12485893, @taavi wrote: > [...] > The shellboxes are showing up as... [21:53:02] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [22:03:35] 10Cloud-VPS (Debian Bullseye Deprecation), 06Collaboration-Services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12487050 (10komla) Hello Team, We are maintaining the October 31 deadline for the migration. We recognize that the recent platform-wide security iss... [22:03:46] 10Cloud-VPS (Debian Bullseye Deprecation), 10Continuous-Integration-Infrastructure: Re-build integration-cumin.integration.eqiad1.wikimedia.cloud on something newer than bullseye - https://phabricator.wikimedia.org/T433592#12487051 (10komla) Could you please confirm whether you still intend to complete the mig... [22:03:50] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate text-to-speech away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435785#12487052 (10komla) Hello @TheresNoTime Could you please confirm whether you still intend to complete the migration and restore your project, and provide a b... [22:04:01] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wikidocumentaries away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435789#12487053 (10komla) Could you please confirm whether you still intend to complete the migration and restore your project, and provide a brief status update?... [22:04:04] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wmcs-uptime away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435759#12487054 (10komla) Hello @TheresNoTime Could you please confirm whether you still intend to complete the migration and restore your project, and provide a brie... [22:08:54] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate etytree away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435770#12487060 (10komla) @Epantaleo, as you may be aware, the team has been dealing with the recent php security vulnerability. Once the dust settles, this will be looked at. [22:14:14] (03update) 10raymond-ndibe: kubernetes.py: deepcopy KubernetesBackend.DEFAULT_RESOURCES before modification [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/123 (https://phabricator.wikimedia.org/T440414) [22:14:52] (03update) 10raymond-ndibe: kubernetes.py: support the new image-config format [repos/cloud/toolforge/webservice-cli] (deepcopy_resources_dict) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/124 (https://phabricator.wikimedia.org/T440415) [22:15:12] (03approved) 10raymond-ndibe: kubernetes.py: deepcopy KubernetesBackend.DEFAULT_RESOURCES before modification [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/123 (https://phabricator.wikimedia.org/T440414) [22:18:36] (03merge) 10raymond-ndibe: kubernetes.py: deepcopy KubernetesBackend.DEFAULT_RESOURCES before modification [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/123 (https://phabricator.wikimedia.org/T440414) [22:18:37] (03update) 10raymond-ndibe: kubernetes.py: support the new image-config format [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/124 (https://phabricator.wikimedia.org/T440415) [22:18:51] (03approved) 10raymond-ndibe: kubernetes.py: support the new image-config format [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/124 (https://phabricator.wikimedia.org/T440415) [22:26:31] (03merge) 10raymond-ndibe: kubernetes.py: support the new image-config format [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/124 (https://phabricator.wikimedia.org/T440415) [22:26:32] (03update) 10raymond-ndibe: add jobs-api backend to webservice-cli [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T431421) [22:26:41] (03update) 10raymond-ndibe: kubernetes.py: support the new image-config format [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/124 (https://phabricator.wikimedia.org/T440415) [22:36:16] (03open) 10raymond-ndibe: d/changelog: bump to 0.103.29 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/130 (https://phabricator.wikimedia.org/T440414 https://phabricator.wikimedia.org/T440415) [22:36:24] (03update) 10raymond-ndibe: d/changelog: bump to 0.103.29 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/130 (https://phabricator.wikimedia.org/T440414 https://phabricator.wikimedia.org/T440415) [22:37:35] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [22:38:33] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli [22:40:23] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [22:41:21] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli [22:42:27] (03approved) 10raymond-ndibe: d/changelog: bump to 0.103.29 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/130 (https://phabricator.wikimedia.org/T440414 https://phabricator.wikimedia.org/T440415) [22:42:38] (03merge) 10raymond-ndibe: d/changelog: bump to 0.103.29 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/130 (https://phabricator.wikimedia.org/T440414 https://phabricator.wikimedia.org/T440415) [22:43:47] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: webservice-cli: support the new image-config format - https://phabricator.wikimedia.org/T440415#12487174 (10Raymond_Ndibe) 05In progress→03Resolved [22:44:05] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: webservice-cli: deepcopy KubernetesBackend.DEFAULT_RESOURCES before modification - https://phabricator.wikimedia.org/T440414#12487177 (10Raymond_Ndibe) 05In progress→03Resolved [22:44:30] (03update) 10raymond-ndibe: Draft: Increase memory quota for oka-bot [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1474 (https://phabricator.wikimedia.org/T441425) (owner: 10tlepage) [22:45:27] (03approved) 10raymond-ndibe: Draft: Increase memory quota for oka-bot [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1474 (https://phabricator.wikimedia.org/T441425) (owner: 10tlepage) [23:28:09] (03open) 10ibrahemqasim: Draft: Resolve "Handle badtoken CSRF Errors in Link Result Validation" [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/18 [23:29:05] (03update) 10ibrahemqasim: Draft: Resolve "Handle badtoken CSRF Errors in Link Result Validation" [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/18 [23:36:51] 10Cloud-VPS, 06tools-infrastructure-team: please enable wikiapiary - https://phabricator.wikimedia.org/T442662#12487323 (10MarkAHershberger) Sorry 'bout that: ` mah@legacy:~$ dpkg -S /usr/bin/php* dpkg-query: no path found matching pattern /usr/bin/php php8.5-cli: /usr/bin/php8.5 dpkg-query: no path found mat... [23:40:03] 10Tool-wsindex, 10Wikisource Reader App (Android): WSIndex returns 500 instead of 404 when a book is not found - https://phabricator.wikimedia.org/T442783 (10RCodes) 03NEW [23:46:34] (03update) 10ibrahemqasim: Draft: Resolve "Handle badtoken CSRF Errors in Link Result Validation" [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/18 [23:50:56] (03open) 10rabuma: books: Return 404 when a book is not found (T442783) [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/16 (https://phabricator.wikimedia.org/T442783) [23:51:45] 10Striker, 06tools-platform-team, 10ChangeProp, 06Data-Engineering, and 14 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12487406 (10Jdforrester-WMF) [23:59:42] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: WSIndex returns 500 instead of 404 when a book is not found - https://phabricator.wikimedia.org/T442783#12487432 (10RCodes) a:03RCodes Hi @Saiphani02 @Bodhisattwa @Muguro I created this task after finding the bug and opened a fix for i... [23:59:52] (03update) 10ibrahemqasim: Resolve "Handle badtoken CSRF Errors in Link Result Validation" [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/18 [23:59:56] (03approved) 10ibrahemqasim: Resolve "Handle badtoken CSRF Errors in Link Result Validation" [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/18 [23:59:59] (03merge) 10ibrahemqasim: Resolve "Handle badtoken CSRF Errors in Link Result Validation" [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/18