[00:38:37] 10Tools, 06MediaWiki-Core-Platform-Team (Radar): coverage.toolforge.org: Down since 14 July 2026 - https://phabricator.wikimedia.org/T435033#12342937 (10Legoktm) OK, so two issues: * serde-xml-rs upgrade broke the charts: fixed in https://gitlab.wikimedia.org/toolforge-repos/coverage/-/commit/614ce32b94a2ad16a... [01:53:01] 10Toolforge, 06tools-platform-team: [jobs-api] Allow storing one-off jobs - https://phabricator.wikimedia.org/T433868#12342951 (10connorshea) FWIW in case it's useful: I am using one-off jobs in Toolforge to run db migrations and to bootstrap the data in my app (it's pulling data from the wikidata dumps), so I... [02:00:15] 10Tools, 06MediaWiki-Core-Platform-Team (Radar): coverage.toolforge.org: Down since 14 July 2026 - https://phabricator.wikimedia.org/T435033#12342952 (10Legoktm) 05Open→03Resolved OK, everything is backfilled as far as we can, mediawiki.svg is updated, the rest of the charts will update on the next dai... [02:10:43] 10Tool-coverage, 10Continuous-Integration-Infrastructure, 07Test-Coverage: Migrate https://coverage.toolforge.org/charts/ to CI infrastructure - https://phabricator.wikimedia.org/T182751#12342954 (10Legoktm) [02:10:50] 10Tool-coverage, 10MediaWiki-Core-Tests, 07Code-Health, 07Test-Coverage: Track test code coverage long term - https://phabricator.wikimedia.org/T182749#12342955 (10Legoktm) [02:11:20] 10Tool-coverage, 06MediaWiki-Core-Platform-Team (Radar): coverage.toolforge.org: Down since 14 July 2026 - https://phabricator.wikimedia.org/T435033#12342956 (10Legoktm) [02:58:36] 10Tool-wikinewsie: Wikinewsie recent changes should link to article card - https://phabricator.wikimedia.org/T438640 (10Pharos) 03NEW [04:25:24] FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-70 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [05:17:00] 10Tool-coverage: Create a proper frontend for coverage.toolforge.org - https://phabricator.wikimedia.org/T438645 (10Legoktm) 03NEW [05:22:41] 10Tool-coverage, 10Continuous-Integration-Infrastructure, 07Test-Coverage: Migrate https://coverage.toolforge.org/charts/ to CI infrastructure - https://phabricator.wikimedia.org/T182751#12343046 (10Legoktm) 05Open→03Declined I don't think there was ever a good plan or interest in doing this; in {T43... [05:26:01] 10Tool-coverage, 10MediaWiki-Core-Tests, 07Code-Health, 07Test-Coverage: Track test code coverage long term - https://phabricator.wikimedia.org/T182749#12343054 (10Legoktm) 05Open→03Resolved a:03Legoktm https://coverage.toolforge.org/ has been a mostly effective "proof of concept" for nearly 9 ye... [05:36:22] 10Toolforge, 06tools-platform-team: Toolforge works unstable now - https://phabricator.wikimedia.org/T438624#12343079 (10MBH) @dcaro could you see? The problem persists. [06:50:42] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T435537) [06:50:46] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T435537) [06:50:51] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [06:50:59] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1077' (T435537) [07:05:03] 10Toolforge, 06tools-platform-team: [jobs-api,regression] Can't delete one-off jobs anymore - https://phabricator.wikimedia.org/T438649 (10dcaro) 03NEW [07:05:12] 10Toolforge, 06tools-platform-team: [jobs-api,regression] Can't delete one-off jobs anymore - https://phabricator.wikimedia.org/T438649#12343164 (10dcaro) p:05Triage→03High [07:05:16] 10Toolforge, 06tools-platform-team: [jobs-api,regression] Can't delete one-off jobs anymore - https://phabricator.wikimedia.org/T438649#12343166 (10dcaro) 05Open→03In progress [07:05:29] 10Toolforge, 06tools-platform-team: [jobs-api,regression] Can't delete one-off jobs anymore - https://phabricator.wikimedia.org/T438649#12343168 (10dcaro) [07:05:31] 10Toolforge, 06tools-platform-team: Toolforge works unstable now - https://phabricator.wikimedia.org/T438624#12343167 (10dcaro) [07:11:11] !log filippo@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1077' (T435537) [07:11:20] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [07:11:29] 10Toolforge, 06tools-platform-team: [jobs-api,regression] Can't delete one-off jobs anymore - https://phabricator.wikimedia.org/T438649#12343189 (10dcaro) We don't have one-off jobs in storage, it should not be trying to delete them. The issue is not the lack of permissions, it's that it should not be trying... [07:11:59] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T435537) [07:12:02] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T435537) [07:12:15] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1077' (T435537) [07:12:22] !log filippo@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1077' (T435537) [07:14:03] FIRING: InstanceDown: Project metricsinfra instance metricsinfra-puppetserver-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:14:04] FIRING: InstanceDown: Project cloudinfra instance mx-out05 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:14:04] FIRING: TargetDown: Job mtail-mx-out is unreachable in project cloudinfra instance mx-out05 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:15:04] FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-39 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:17:14] 10Cloud-VPS, 06tools-infrastructure-team: Disable cloudvps instance 'suspend' - https://phabricator.wikimedia.org/T438651 (10fgiunchedi) 03NEW [07:20:20] 10Toolforge, 06tools-platform-team: [jobs-api,regression] Can't delete one-off jobs anymore - https://phabricator.wikimedia.org/T438649#12343207 (10dcaro) Ok, so the issue is that as we try to delete anyhow, and handle the one-off only on fallback, now that the clusterrole is changed it returns 401 instead of... [07:20:24] FIRING: [2x] ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-39 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [07:30:26] (03open) 10dcaro: delete: don't try to delete oneoffs from k8s [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/378 (https://phabricator.wikimedia.org/T438649) [07:36:04] !log dcaro@acme tools START - Cookbook wmcs.openstack.cloudvirt.vm_console [07:36:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [07:38:13] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T435537) [07:38:16] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T435537) [07:38:18] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [07:41:12] !log dcaro@acme tools END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) [07:41:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [07:43:39] PROBLEM - Host cloudvirt1077 is DOWN: PING CRITICAL - Packet loss = 100% [07:44:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1077 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [07:48:21] RECOVERY - Host cloudvirt1077 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [07:48:51] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance [07:48:55] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) [07:49:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1077 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [07:52:48] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T435537) [07:52:51] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T435537) [07:52:55] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T435537) [07:52:56] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [07:52:58] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T435537) [07:53:02] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T435537) [07:53:05] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T435537) [07:53:35] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T435537) [07:53:38] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T435537) [07:53:56] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1063' (T435537) [07:59:03] FIRING: PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-prometheus-5 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:02:06] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1063' (T435537) [08:02:13] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [08:04:03] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-meta-monitor-1 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:14:03] FIRING: [7x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:18:05] (03approved) 10tlepage: delete: don't try to delete oneoffs from k8s [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/378 (https://phabricator.wikimedia.org/T438649) (owner: 10dcaro) [08:19:03] FIRING: [9x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:23:49] !log dcaro@acme tools START - Cookbook wmcs.openstack.cloudvirt.vm_console [08:23:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:28:58] !log dcaro@acme tools END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) [08:29:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:30:07] !log dcaro@acme tools START - Cookbook wmcs.openstack.cloudvirt.vm_console [08:30:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:30:45] (03merge) 10dcaro: delete: don't try to delete oneoffs from k8s [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/378 (https://phabricator.wikimedia.org/T438649) [08:33:38] !log dcaro@acme tools END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [08:33:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:33:43] !log dcaro@acme tools START - Cookbook wmcs.openstack.cloudvirt.vm_console [08:33:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:33:45] !log dcaro@acme tools END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) [08:33:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:33:52] !log dcaro@acme tools START - Cookbook wmcs.openstack.cloudvirt.vm_console [08:33:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:33:55] !log dcaro@acme tools END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) [08:33:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:34:13] !log dcaro@acme tools START - Cookbook wmcs.openstack.cloudvirt.vm_console [08:34:15] !log dcaro@acme tools END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) [08:34:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:34:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:34:26] !log dcaro@acme tools START - Cookbook wmcs.openstack.cloudvirt.vm_console [08:34:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:34:53] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.571-20260921083057-16c3bc3a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1425 (https://phabricator.wikimedia.org/T438649) [08:34:55] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.571-20260921083057-16c3bc3a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1425 (https://phabricator.wikimedia.org/T438649) [08:35:18] 10Tool-wikitermbase: [Gadget] Group each result's sources by dictionary type (terminology, language, thesaurus) - https://phabricator.wikimedia.org/T438664 (10ForzaGreen) 03NEW [08:36:05] 10Cloud-VPS, 06tools-infrastructure-team: Change mount propagation for /mnt/nfs in wikiqlever - https://phabricator.wikimedia.org/T433671#12343504 (10gmodena) >>! In T433671#12328925, @Physikerwelt wrote: > I was hoping to wind down the WikiQlever project as soon as we have a non-split Qlever instance in produ... [08:37:25] 10Tool-wikitermbase: [Gadget] Group each result's sources by dictionary type (terminology, language, thesaurus) - https://phabricator.wikimedia.org/T438664#12343511 (10ForzaGreen) @Michel_Bakni , could you confirm the functional requirement above, and add any details you think are missing? [08:39:28] !log dcaro@acme tools END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) [08:39:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:50:55] 10Cloud-VPS, 06tools-infrastructure-team: Change mount propagation for /mnt/nfs in wikiqlever - https://phabricator.wikimedia.org/T433671#12343731 (10Physikerwelt) >>! In T433671#12343504, @gmodena wrote: > The QLever team maintains a full graph endpoint at https://qlever.dev/wikidata, is it an avenue y... [09:02:57] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [09:08:35] 10Cloud-VPS, 06tools-infrastructure-team: Establish an automated cinder volume restore procedure and testing - https://phabricator.wikimedia.org/T438051#12343990 (10Volans) For a full detach (you probably want to change it to redirect the output to a file if backy2 logs to stdout/err) you can see the content... [09:10:04] FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-39 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:16:03] 10Toolforge, 06tools-platform-team: [jobs-api] Allow storing one-off jobs - https://phabricator.wikimedia.org/T433868#12344028 (10dcaro) >>! In T433868#12342951, @connorshea wrote: > FWIW in case it's useful: I am using one-off jobs in Toolforge to run db migrations and to bootstrap the data in my app (it's pu... [09:18:22] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [09:19:28] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Documentation: [phase 1]Write tutorial for migrating build service jobs to push-to-deploy - https://phabricator.wikimedia.org/T437477#12344038 (10komla) [09:32:57] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Documentation: [phase 1]Write tutorial for migrating build service jobs to push-to-deploy - https://phabricator.wikimedia.org/T437477#12344090 (10komla) @dcaro config generation warns my job appears not to be a build-service based job. ` tools.komla-te... [09:37:06] 10Cloud-VPS, 06tools-infrastructure-team: Instances migrated to cloudvirt1073 lost their network following a live migration - https://phabricator.wikimedia.org/T438697 (10fgiunchedi) 03NEW [09:46:02] 10Toolforge, 06tools-platform-team: [toolsdb] 2026-08-18 Transaction History Length growing too much - https://phabricator.wikimedia.org/T438504#12344209 (10fnegri) @magnus thanks for having a look. I can confirm that I can see some moments where no SQL queries are running from mixnmatch, so maybe it is not yo... [09:51:02] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [09:54:03] RESOLVED: InstanceDown: Project metricsinfra instance metricsinfra-puppetserver-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:54:04] RESOLVED: InstanceDown: Project cloudinfra instance mx-out05 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:54:04] RESOLVED: TargetDown: Job mtail-mx-out is unreachable in project cloudinfra instance mx-out05 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:55:04] FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-39 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:55:24] FIRING: [2x] ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-39 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [09:57:38] 10Cloud-VPS, 06tools-infrastructure-team: Instances migrated to cloudvirt1073 lost their network following a live migration - https://phabricator.wikimedia.org/T438697#12344253 (10fgiunchedi) @taavi bounced the neutron agent on 1073 and the network came back, we were unable to understand what went wrong though [10:03:18] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T435537) [10:03:21] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T435537) [10:03:27] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [10:04:03] FIRING: [9x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:05:07] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T435537) [10:05:10] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T435537) [10:05:34] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge, 06tools-platform-team: Design mechanism and process for upgrading Kubernetes container runtimes - https://phabricator.wikimedia.org/T213641#12344335 (10dcaro) [10:06:20] 06cloud-services-team, 10Toolforge, 06tools-platform-team: Remove jessie + stretch containers from Toolforge - https://phabricator.wikimedia.org/T286798#12344336 (10dcaro) [10:10:27] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [10:12:03] (03CR) 10David Caro: [C:03+1] "Just did a check and we are not really removing any language, so this is ok imo, people can upgrade to the newer version of their language" [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1342772 (https://phabricator.wikimedia.org/T400258) (owner: 10Majavah) [10:12:22] (03CR) 10Majavah: [C:03+2] Remove bullseye based images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1342772 (https://phabricator.wikimedia.org/T400258) (owner: 10Majavah) [10:12:48] (03approved) 10dcaro: Mark remaining bullseye images as deprecated [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/24 (https://phabricator.wikimedia.org/T400258) (owner: 10taavi) [10:13:16] (03approved) 10dcaro: jobs-api: bump to 0.0.571-20260921083057-16c3bc3a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1425 (https://phabricator.wikimedia.org/T438649) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:13:20] (03update) 10taavi: Mark remaining bullseye images as deprecated [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/24 (https://phabricator.wikimedia.org/T400258) [10:13:29] (03merge) 10dcaro: jobs-api: bump to 0.0.571-20260921083057-16c3bc3a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1425 (https://phabricator.wikimedia.org/T438649) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:13:33] (03Merged) 10jenkins-bot: Remove bullseye based images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1342772 (https://phabricator.wikimedia.org/T400258) (owner: 10Majavah) [10:13:37] (03merge) 10taavi: Mark remaining bullseye images as deprecated [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/24 (https://phabricator.wikimedia.org/T400258) [10:15:04] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: image-config: bump to 0.0.29-20260921101352-341123b4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1426 (https://phabricator.wikimedia.org/T400258) [10:16:02] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api,regression] Can't delete one-off jobs anymore - https://phabricator.wikimedia.org/T438649#12344361 (10dcaro) 05In progress→03Resolved [10:16:38] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component image-config [10:17:13] 10Toolforge, 06tools-platform-team: Toolforge works unstable now - https://phabricator.wikimedia.org/T438624#12344364 (10dcaro) The failures trying to delete one-off jobs are due to {T438649}, to debug the connectivity issues seem to have been the bastion stuck on D processes (no cause found yet), after a... [10:17:19] 10Toolforge, 06tools-platform-team: Toolforge works unstable now - https://phabricator.wikimedia.org/T438624#12344366 (10dcaro) 05Open→03Resolved a:03dcaro [10:19:03] FIRING: [9x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:21:03] 10Toolforge, 06tools-platform-team: [toolforge-cli,jobs-cli] Potential tech-debt improvements - https://phabricator.wikimedia.org/T438391#12344377 (10dcaro) [10:25:04] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-70 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [10:25:24] RESOLVED: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-70 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [10:29:03] RESOLVED: [7x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:29:37] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config [11:14:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [11:17:07] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component image-config [11:19:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [11:32:36] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config [11:33:16] (03merge) 10taavi: image-config: bump to 0.0.29-20260921101352-341123b4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1426 (https://phabricator.wikimedia.org/T400258) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [11:33:30] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Stop building Bullseye based Toolforge prebuilt images - https://phabricator.wikimedia.org/T400258#12344724 (10taavi) 05Open→03Resolved a:03taavi [12:06:30] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/78 [12:13:23] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/78 (owner: 10l10n-bot) [12:14:02] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/78 (owner: 10l10n-bot) [12:46:20] 06tools-platform-team: [jobs-api] Error with mounts on one-off jobs - https://phabricator.wikimedia.org/T438730 (10dcaro) 03NEW [12:46:31] (03open) 10dcaro: ping_clinic_duty: use thursday for the etherpad [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/11 [12:47:20] (03update) 10dcaro: ping_clinic_duty: use thursday for the etherpad [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/11 [12:50:51] 10Toolforge, 06tools-platform-team: [jobs-api] Error with mounts on one-off jobs - https://phabricator.wikimedia.org/T438730#12345031 (10taavi) [12:53:02] 10Cloud-VPS, 06tools-infrastructure-team: Come up with a backup mechanism for P:wmcs::etcd clusters - https://phabricator.wikimedia.org/T438731 (10taavi) 03NEW [12:54:13] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project: [etcd,infra] Find a backup solution for the etcd database - https://phabricator.wikimedia.org/T339934#12345063 (10taavi) a:03taavi [12:54:16] 10Cloud-VPS, 06tools-infrastructure-team: Come up with a backup mechanism for P:wmcs::etcd clusters - https://phabricator.wikimedia.org/T438731#12345066 (10taavi) [12:54:19] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project: [etcd,infra] Find a backup solution for the etcd database - https://phabricator.wikimedia.org/T339934#12345065 (10taavi) [13:00:52] 10Cloud-VPS, 06tools-infrastructure-team: Instances migrated to cloudvirt1073 lost their network following a live migration - https://phabricator.wikimedia.org/T438697#12345079 (10fgiunchedi) 05Open→03Declined I'm tentatively declining just because I don't think we'll be actively looking at this proble... [13:01:10] 10Tool-wikiloves: Wikiloves statistics are not being updated - https://phabricator.wikimedia.org/T437822#12345099 (10This_is_Andy) >>! In T437822#12316012, @JeanFred wrote: > (Not a perfect one, but https://ptools.toolforge.org/uploadersincat.php?category=Images_from_Wiki_Loves_Monuments_2026_in_Poland might be... [13:11:14] 10Toolforge, 06tools-platform-team: [toolsdb] 2026-08-18 Transaction History Length growing too much - https://phabricator.wikimedia.org/T438504#12345142 (10fnegri) 05Open→03In progress [13:16:03] 10Data-Services, 06tools-infrastructure-team: clouddb1023 memory alert - https://phabricator.wikimedia.org/T438200#12345187 (10fnegri) 05Open→03Resolved a:03fnegri Thanks @Marostegui for looking into this. I will mark it as Resolved for now, we can reopen it if the issue re-occurs. [13:16:06] (03open) 10dcaro: add noop and time param [repos/cloud/wmcs/utils] (fix_etherpad_date) - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/12 [13:16:39] (03update) 10dcaro: add noop and time param [repos/cloud/wmcs/utils] (fix_etherpad_date) - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/12 [13:18:28] 10Toolforge, 06tools-infrastructure-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project: [etcd,infra] Find a backup solution for the etcd database - https://phabricator.wikimedia.org/T339934#12345214 (10aputhin) [13:18:37] (03update) 10dcaro: rotate_clinic_duty: add noop and time param [repos/cloud/wmcs/utils] (fix_etherpad_date) - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/12 [13:19:09] 10Toolforge, 06tools-infrastructure-team: Stop building Bullseye based Toolforge prebuilt images - https://phabricator.wikimedia.org/T400258#12345216 (10aputhin) [13:31:05] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-api] Implement an endpoint to retrieve system-level event logs - https://phabricator.wikimedia.org/T432566#12345287 (10dcaro) >>! In T432566#12333108, @TLepage-WMF wrote: > The latter approach requires logs-api to be aware of the dep... [13:46:40] 10Cloud-VPS, 06tools-infrastructure-team: cloudinfra-db backups - https://phabricator.wikimedia.org/T301640#12345355 (10taavi) 05Open→03Resolved And documented at https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Cloudinfra#cloudinfra-db. [13:47:41] 10Toolforge, 06tools-platform-team: [toolsdb] 2026-08-18 Transaction History Length growing too much - https://phabricator.wikimedia.org/T438504#12345359 (10aputhin) p:05Triage→03Medium [13:56:15] (03open) 10taavi: kubernetes: Add VM to hold cluster backups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/127 (https://phabricator.wikimedia.org/T339934) [13:58:27] (03update) 10taavi: kubernetes: Add VM to hold cluster backups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/127 (https://phabricator.wikimedia.org/T339934) [14:14:44] 10Cloud-VPS, 06tools-infrastructure-team: Disable cloudvps instance 'suspend' - https://phabricator.wikimedia.org/T438651#12345497 (10fgiunchedi) →14Duplicate dup:03T431307 [14:14:46] 10Cloud-VPS, 06tools-infrastructure-team: Consider deprecating the 'suspend' VM feature - https://phabricator.wikimedia.org/T431307#12345499 (10fgiunchedi) [14:38:34] (03update) 10taavi: kubernetes: Add VM to hold cluster backups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/127 (https://phabricator.wikimedia.org/T339934) [14:43:06] (03approved) 10volans: kubernetes: Add VM to hold cluster backups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/127 (https://phabricator.wikimedia.org/T339934) (owner: 10taavi) [14:45:23] (03merge) 10taavi: kubernetes: Add VM to hold cluster backups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/127 (https://phabricator.wikimedia.org/T339934) [14:46:43] 10Cloud-VPS (Project-requests): Request creation of Trove Database for Broad Interwiki Enhanced Retrieval VPS project - https://phabricator.wikimedia.org/T438363#12345713 (10dcaro) @WikiBayer Hi! For the project name, we will need it to match the openstack project naming standards, essentially: > Note that ther... [14:49:52] 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Arusha 30): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12345729 (10dcaro) Hi, is this already sorted? Or there's anything on t... [14:50:56] 10Cloud-VPS (Project-requests): Request creation of Trove Database for Broad Interwiki Enhanced Retrieval VPS project - https://phabricator.wikimedia.org/T438363#12345734 (10WikiBayer) Since it's just one database, the name doesn't matter a shortened version will do, e.g., "bier-db" [14:54:29] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-backup-1.toolsbeta.eqiad1.wikimedia.cloud [14:55:50] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-backup-1.toolsbeta.eqiad1.wikimedia.cloud [14:57:47] (03open) 10taavi: tools: Add k8s backup host [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/128 (https://phabricator.wikimedia.org/T339934) [14:58:33] 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure: Disk quota increase for the integration project - https://phabricator.wikimedia.org/T438516#12345770 (10Andrew) Seems fine to me, but please check back and revert if the cache turns out to not help! +1 [15:06:26] 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure: Disk quota increase for the integration project - https://phabricator.wikimedia.org/T438516#12345818 (10fnegri) +1 [15:07:02] 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure: Disk quota increase for the integration project - https://phabricator.wikimedia.org/T438516#12345821 (10dcaro) 05Open→03In progress a:03dcaro [15:07:14] 10Cloud-VPS (Quota-requests), 06tools-platform-team, 10Continuous-Integration-Infrastructure: Disk quota increase for the integration project - https://phabricator.wikimedia.org/T438516#12345824 (10dcaro) [15:10:31] 10Cloud-VPS (Project-requests): Request creation of Trove Database for Broad Interwiki Enhanced Retrieval VPS project - https://phabricator.wikimedia.org/T438363#12345845 (10Andrew) +1 sgtm [15:12:21] !log dcaro@cloudcumin1001 integration START - Cookbook wmcs.openstack.quota_increase by 400 gigabytes (T438516) [15:12:25] T438516: Disk quota increase for the integration project - https://phabricator.wikimedia.org/T438516 [15:12:29] !log dcaro@cloudcumin1001 integration END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 400 gigabytes (T438516) [15:13:33] 10Cloud-VPS (Quota-requests), 06tools-platform-team, 10Continuous-Integration-Infrastructure: Disk quota increase for the integration project - https://phabricator.wikimedia.org/T438516#12345867 (10dcaro) 05In progress→03Resolved This should be up and running :) {F104136872} [15:14:01] 10Cloud-VPS (Project-requests), 06tools-platform-team: Request creation of Trove Database for Broad Interwiki Enhanced Retrieval VPS project - https://phabricator.wikimedia.org/T438363#12345880 (10dcaro) a:03dcaro [15:15:00] !log dcaro@cloudcumin1001 bier-db START - Cookbook wmcs.vps.create_project for project bier-db in eqiad1 (T438363) [15:15:00] dcaro@cloudcumin1001: Unknown project "bier-db" [15:15:01] T438363: Request creation of Trove Database for Broad Interwiki Enhanced Retrieval VPS project - https://phabricator.wikimedia.org/T438363 [15:15:49] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project bier-db [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/359 (https://phabricator.wikimedia.org/T438363) [15:19:05] dcaro@cloudcumin1001 create_project (PID 715546) is awaiting input [15:31:10] 10Toolforge, 06tools-platform-team: [toolsdb] 2026-08-18 Transaction History Length growing too much - https://phabricator.wikimedia.org/T438504#12346018 (10fnegri) `s53685` (`tools.editgroups`) could be related to this spike, as it logged a few slow queries on 2026-09-12 right when the history length started... [15:33:20] (03approved) 10dcaro: projects: added project bier-db [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/359 (https://phabricator.wikimedia.org/T438363) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [15:33:29] (03merge) 10dcaro: projects: added project bier-db [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/359 (https://phabricator.wikimedia.org/T438363) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [15:34:04] 10Toolforge, 06tools-platform-team: [toolsdb] 2026-08-18 Transaction History Length growing too much - https://phabricator.wikimedia.org/T438504#12346048 (10fnegri) p:05Medium→03High [15:34:52] (03open) 10dcaro: Revert "projects: added project bier-db" [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/360 [15:34:57] (03approved) 10dcaro: Revert "projects: added project bier-db" [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/360 [15:35:14] !log dcaro@cloudcumin1001 bier-db END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project bier-db in eqiad1 (T438363) [15:35:15] (03merge) 10dcaro: Revert "projects: added project bier-db" [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/360 [15:35:16] dcaro@cloudcumin1001: Unknown project "bier-db" [15:35:16] T438363: Request creation of Trove Database for Broad Interwiki Enhanced Retrieval VPS project - https://phabricator.wikimedia.org/T438363 [15:35:28] !log dcaro@cloudcumin1001 bier-db START - Cookbook wmcs.vps.create_project for trove-only project bier-db in eqiad1 (T438363) [15:35:28] dcaro@cloudcumin1001: Unknown project "bier-db" [15:36:15] (03update) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project bier-db [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/361 (https://phabricator.wikimedia.org/T438363) [15:36:18] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project bier-db [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/361 (https://phabricator.wikimedia.org/T438363) [15:38:50] 10Toolforge, 06tools-infrastructure-team: [tofu,ci] Make the ci use the vendored providers directory in the git repo - https://phabricator.wikimedia.org/T438754 (10dcaro) 03NEW [15:39:12] 10Toolforge, 06tools-infrastructure-team: [tofu-infra,ci] Make the ci use the vendored providers directory in the git repo - https://phabricator.wikimedia.org/T438754#12346106 (10dcaro) [15:39:30] dcaro@cloudcumin1001 create_project (PID 717929) is awaiting input [15:39:35] (03approved) 10dcaro: projects: added project bier-db [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/361 (https://phabricator.wikimedia.org/T438363) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [15:39:40] (03merge) 10dcaro: projects: added project bier-db [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/361 (https://phabricator.wikimedia.org/T438363) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [15:39:41] 10Cloud-VPS, 06tools-infrastructure-team: [tofu-infra,ci] Make the ci use the vendored providers directory in the git repo - https://phabricator.wikimedia.org/T438754#12346108 (10taavi) [15:41:00] !log dcaro@cloudcumin1001 bier-db END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for trove-only project bier-db in eqiad1 (T438363) [15:41:04] T438363: Request creation of Trove Database for Broad Interwiki Enhanced Retrieval VPS project - https://phabricator.wikimedia.org/T438363 [15:42:39] 10Cloud-VPS (Project-requests), 06tools-platform-team, 13Patch-For-Review: Request creation of Trove Database for Broad Interwiki Enhanced Retrieval VPS project - https://phabricator.wikimedia.org/T438363#12346122 (10dcaro) 05Open→03Resolved @WikiBayer Done! Make sure to subscribe to cloud-announce (... [15:43:34] 10Cloud-VPS, 06tools-infrastructure-team: Consider deprecating the 'suspend' VM feature - https://phabricator.wikimedia.org/T431307#12346126 (10Andrew) a:03Andrew As I feared, this is easy to disable in the API policies but Horizon still displays the option. The right/slow solution is to fix upstream horizo... [15:47:55] 10Cloud-VPS, 06tools-infrastructure-team: Consider deprecating the 'suspend' VM feature - https://phabricator.wikimedia.org/T431307#12346143 (10Andrew) >>! In T431307#12346126, @Andrew wrote: > As I feared, this is easy to disable in the API policies but Horizon still displays the option. Wrong! If I disable... [16:17:29] 10Toolforge, 06tools-platform-team: [toolsdb] 2026-08-18 Transaction History Length growing too much - https://phabricator.wikimedia.org/T438504#12346297 (10fnegri) Index created successfully: `lang=mysql MariaDB [(none)]> SET SESSION max_statement_time = 0; Query OK, 0 rows affected (0.000 sec) MariaDB [(no... [16:19:55] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Consider deprecating the 'suspend' VM feature - https://phabricator.wikimedia.org/T431307#12346304 (10Andrew) Attached patch should disable 'suspend' and 'pause' (both of which prevent live migration.) I've also sent an email to cloud@lists.wikime... [16:25:44] (03approved) 10dcaro: refactor: Parse Loki result streams into logs [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) (owner: 10tlepage) [16:28:03] 10PAWS, 06tools-platform-team: [paws] Do a 1h check with @fnegri on bad behavior detection - https://phabricator.wikimedia.org/T438364#12346349 (10dcaro) >>! In T438364#12334818, @dcaro wrote: > @fnegri do you want to do this during the monday pairing slot? I forgot! Damn xd, will send an invite [16:28:17] 10Tool-wsindex, 06MediaWiki-Media-Platform-Team, 10Wikipedia-iOS-App, 06WikipediaApp, and 2 others: Update apps to support for the move from upload.wikimedia.org to thumb.wikimedia.org - https://phabricator.wikimedia.org/T434821#12346350 (10Seddon) @Ladsgroup are all the changes in place? If so we want to... [16:29:12] 10PAWS, 06tools-platform-team: [paws] Do a 1h check with @fnegri on bad behavior detection - https://phabricator.wikimedia.org/T438364#12346362 (10dcaro) 05Open→03In progress [16:40:01] 06Toolforge-standards-committee, 06LPL Onboarding and Development, 10Starter Kit, 07Privacy: starterkit.toolforge.org loads third-party resources - https://phabricator.wikimedia.org/T438458#12346410 (10eamedina) Thanks for flagging, this has been fixed and deployed now. Also took the opportunity to fix /ro... [16:46:49] 10Toolforge, 06tools-platform-team: [toolsdb] 2026-09-18 Transaction History Length growing too much - https://phabricator.wikimedia.org/T438504#12346445 (10fnegri) [16:54:39] (03open) 10r4356th: Implement visual regression testing [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [16:55:11] (03update) 10r4356th: Use REST API to verify whether any lint errors were actually fixed before saving edit [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/20 (https://phabricator.wikimedia.org/T420294) [16:55:17] (03update) 10r4356th: Fix REST API path [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/21 (https://phabricator.wikimedia.org/T420294) [17:03:56] 10Tool-delintbot, 13Patch-For-Review: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12346569 (10Redmin) [17:05:12] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Consider deprecating the 'suspend' VM feature - https://phabricator.wikimedia.org/T431307#12346576 (10Andrew) On the email thread, Martin asked about 'locking' which turns out to also pause, so I've added that feature to the patch. [17:07:07] FIRING: ToolsDBReplicationLagIsTooHigh: ToolsDB replication is lagging - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBReplicationLagIsTooHigh [17:11:00] 10Data-Services, 06tools-infrastructure-team: clouddb1023 memory alert - https://phabricator.wikimedia.org/T438200#12346627 (10fnegri) 05Resolved→03Open clouddb1022 is now alerting :/ > WARN Memory 95% used. Largest process: mysqld (1497) = 72.7% [17:11:11] 10Data-Services, 06tools-infrastructure-team: clouddb memory alert - https://phabricator.wikimedia.org/T438200#12346633 (10fnegri) [17:14:30] 10Data-Services, 06tools-infrastructure-team: clouddb memory alert - https://phabricator.wikimedia.org/T438200#12346670 (10fnegri) Interestingly, clouddb1023 is a `analytics` host, while clouddb1022 is a `web` host so the workloads should be different: `lang=shell-session fnegri@cumin1004:~$ sudo confctl sele... [17:18:00] 10Data-Services, 06tools-infrastructure-team: clouddb memory alert - https://phabricator.wikimedia.org/T438200#12346695 (10fnegri) Current memory usage on clouddb1022: `lang=shell-session top - 17:16:55 up 34 days, 7:41, 1 user, load average: 0.32, 0.29, 0.44 Tasks: 498 total, 2 running, 496 sleeping,... [17:23:37] RESOLVED: ToolsDBReplicationLagIsTooHigh: ToolsDB replication is lagging - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBReplicationLagIsTooHigh [17:28:30] 10Data-Services, 06tools-infrastructure-team: clouddb memory alert - https://phabricator.wikimedia.org/T438200#12346792 (10fnegri) Processlist for s3 (the instance at 72.7% mem): `lang=mysql root@clouddb1022:s3[(none)]> SHOW PROCESSLIST; +----------+--------------+--------------------+----------------+-------... [17:44:59] 10Data-Services, 06tools-infrastructure-team: clouddb memory alert - https://phabricator.wikimedia.org/T438200#12346854 (10Andrew) ` root@clouddb1022:/etc/mysql# grep -ir innodb_buffer_pool_size * mysqld.conf.d/s3.cnf:innodb_buffer_pool_size = 100G mysqld.conf.d/x3.cnf:innodb_buffer_pool_size = 105G ` [17:46:25] FIRING: [2x] SystemdUnitFailed: wmf-pt-kill@s3.service on clouddb1022:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:55:22] (03PS2) 10Krinkle: write_config: Add mediawiki/vendor to "Everything" preset [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1336608 (https://phabricator.wikimedia.org/T227704) [17:55:27] (03CR) 10Krinkle: write_config: Add mediawiki/vendor to "Everything" preset [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1336608 (https://phabricator.wikimedia.org/T227704) (owner: 10Krinkle) [17:55:30] (03CR) 10Krinkle: [C:03+2] write_config: Add mediawiki/vendor to "Everything" preset [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1336608 (https://phabricator.wikimedia.org/T227704) (owner: 10Krinkle) [17:56:25] RESOLVED: [2x] SystemdUnitFailed: wmf-pt-kill@s3.service on clouddb1022:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:56:51] (03CR) 10CI reject: [V:04-1] write_config: Add mediawiki/vendor to "Everything" preset [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1336608 (https://phabricator.wikimedia.org/T227704) (owner: 10Krinkle) [17:58:09] (03CR) 10Krinkle: "CI is broken it seems" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1336608 (https://phabricator.wikimedia.org/T227704) (owner: 10Krinkle) [18:13:30] 10wikitech.wikimedia.org: Add tabbed window gadget to wikitech to support multi-language code sample display - https://phabricator.wikimedia.org/T438775 (10TBurmeister) 03NEW [18:14:07] 10wikitech.wikimedia.org, 07Documentation: Add tabbed window gadget to wikitech to support multi-language code sample display - https://phabricator.wikimedia.org/T438775#12347006 (10TBurmeister) p:05Triage→03Low [18:18:01] 10VPS-project-Codesearch, 06Infrastructure-Foundations: Codesearch can't build: ERROR: golang1.15 not found - https://phabricator.wikimedia.org/T438777#12347049 (10Krinkle) [18:21:21] 10VPS-project-Codesearch, 06Infrastructure-Foundations: Codesearch can't build: ERROR: golang1.15 not found - https://phabricator.wikimedia.org/T438777#12347061 (10Krinkle) At https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/production-images/+/refs/heads/master/images/golang/, we curren... [18:24:03] (03PS1) 10Krinkle: hound: Upgrade from golang1.15 to golang1.19 [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1343614 (https://phabricator.wikimedia.org/T438777) [18:25:02] (03CR) 10CI reject: [V:04-1] hound: Upgrade from golang1.15 to golang1.19 [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1343614 (https://phabricator.wikimedia.org/T438777) (owner: 10Krinkle) [19:20:28] 10Openstack-Magnum, 10Quarry, 06tools-infrastructure-team: Rebuild quarry with the new magnum-cluster-api driver - https://phabricator.wikimedia.org/T437501#12347242 (10Andrew) 05Open→03Resolved a:03Andrew [19:32:15] 10Toolforge, 06tools-platform-team: [toolsdb] 2026-09-18 Transaction History Length growing too much - https://phabricator.wikimedia.org/T438504#12347283 (10Arcstur) Thank you @fnegri EditGroups is currently being spammed by bots and the only viable solution right now was to restrict visualization for logged... [19:36:23] 06tools-infrastructure-team, 10wikitech.wikimedia.org: wikitech-static: get better at cleaning up old containers and/or establish a cadence for rebuilding the host - https://phabricator.wikimedia.org/T432391#12347303 (10Andrew) 05Open→03Resolved ` root@ip-172-31-28-4:/var/log/nginx# uptime 19:34:55 up... [19:45:53] (03CR) 10Krinkle: "`" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1343614 (https://phabricator.wikimedia.org/T438777) (owner: 10Krinkle) [19:45:57] (03CR) 10Krinkle: "recheck" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1343614 (https://phabricator.wikimedia.org/T438777) (owner: 10Krinkle) [20:25:11] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: cloudbackup100[34] still do not actually do 'backy2 cleanup - https://phabricator.wikimedia.org/T394618#12347545 (10Andrew) 05Open→03Resolved ...much later... This was failing due to backup jobs running for more than 24 hours and overl... [20:26:43] 06Toolforge-standards-committee, 06LPL Onboarding and Development, 10Starter Kit, 07Privacy: starterkit.toolforge.org loads third-party resources - https://phabricator.wikimedia.org/T438458#12347558 (10srishakatux) 05Open→03Resolved [22:07:28] (03open) 10tmoney: Use published selection picker and core packages [toolforge-repos/findlinkfast] - 10https://gitlab.wikimedia.org/toolforge-repos/findlinkfast/-/merge_requests/5 [22:07:49] (03merge) 10tmoney: Use published selection picker and core packages [toolforge-repos/findlinkfast] - 10https://gitlab.wikimedia.org/toolforge-repos/findlinkfast/-/merge_requests/5