[00:06:22] FIRING: [2x] PrometheusK8sCertExpirySoon: Prometheus k8s certificate is about to expire - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/PrometheusK8sCertExpirySoon - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusK8sCertExpirySoon [00:16:56] RESOLVED: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1002 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [00:48:34] FIRING: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [01:09:42] (03update) 10raymond-ndibe: builds-api: update heroku builder and runner images [repos/cloud/toolforge/toolforge-deploy] (fix_istio_issue) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1225 (https://phabricator.wikimedia.org/T423417 https://phabricator.wikimedia.org/T424362) (owner: 10fnegri) [01:09:45] (03approved) 10raymond-ndibe: builds-api: update heroku builder and runner images [repos/cloud/toolforge/toolforge-deploy] (fix_istio_issue) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1225 (https://phabricator.wikimedia.org/T423417 https://phabricator.wikimedia.org/T424362) (owner: 10fnegri) [01:14:39] (03update) 10raymond-ndibe: k8s: adjust default memory requests [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/102 (https://phabricator.wikimedia.org/T420565) (owner: 10filippo) [01:19:48] (03update) 10raymond-ndibe: replace job images with web images [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/263 (https://phabricator.wikimedia.org/T415322) [01:22:19] (03close) 10raymond-ndibe: [maintain-harbor.jobs] manage policies and robot accounts [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/47 (https://phabricator.wikimedia.org/T360509) [01:29:02] (03update) 10raymond-ndibe: replace job images with web images [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/263 (https://phabricator.wikimedia.org/T415322) [01:29:50] (03approved) 10raymond-ndibe: istio-system: Disable X-Envoy-Peer-Metadata headers [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235 (https://phabricator.wikimedia.org/T392356) (owner: 10taavi) [01:30:19] (03update) 10raymond-ndibe: loki.py: fix logs ordering [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/18 (https://phabricator.wikimedia.org/T401552) [01:30:20] (03update) 10raymond-ndibe: loki.py: fix logs ordering [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/18 (https://phabricator.wikimedia.org/T401552) [01:32:17] (03close) 10raymond-ndibe: [build, api] support build queueing beyond max_parallel build config [repos/cloud/toolforge/builds-api] (run_pipeline_cleanup_per_repo) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T402568) [01:32:33] (03close) 10raymond-ndibe: [deploy_task, tool_handlers] queue deployments to allow creation of multiple deployments at once [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/131 (https://phabricator.wikimedia.org/T402568) [01:32:52] (03close) 10raymond-ndibe: [build] run pipeline cleanup per repo [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/142 (https://phabricator.wikimedia.org/T404157) [01:33:15] (03close) 10raymond-ndibe: [maintain-harbor] add tests and configurations for new maintain-harbor jobs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/881 (https://phabricator.wikimedia.org/T360509) [01:33:30] (03close) 10raymond-ndibe: [jobs-cli] quota refactor [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/97 (https://phabricator.wikimedia.org/T389118) [01:33:50] (03close) 10raymond-ndibe: Draft: job: clean up orphaned s3 objects [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/63 (https://phabricator.wikimedia.org/T418528) [01:35:46] (03close) 10raymond-ndibe: [jobs-api] refactor quota models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/164 (https://phabricator.wikimedia.org/T389118) [02:22:23] (03open) 10raymond-ndibe: upgrade pytest [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/94 (https://phabricator.wikimedia.org/T424368) [02:41:43] (03update) 10raymond-ndibe: upgrade pytest [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/94 (https://phabricator.wikimedia.org/T424368) [03:07:54] 10Tool-humaniki-2: Favicon - https://phabricator.wikimedia.org/T423004#11892644 (10Tiisu) a:03Tiisu Hi there @Danya, do you have a favicon/icon design for Humaniki that I should add, or should I create one? [03:25:38] (03open) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/109 (https://phabricator.wikimedia.org/T422184) [03:26:27] (03open) 10raymond-ndibe: upgrade pytest [repos/cloud/toolforge/envvars-cli] (upgrade_python_to_3.13) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/110 (https://phabricator.wikimedia.org/T424368) [03:26:38] (03update) 10raymond-ndibe: upgrade pytest [repos/cloud/toolforge/envvars-cli] (upgrade_python_to_3.13) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/110 (https://phabricator.wikimedia.org/T424368) [03:28:18] (03open) 10raymond-ndibe: upgrade pytest [repos/cloud/toolforge/builds-cli] (upgrade_python_to_3_13) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/135 (https://phabricator.wikimedia.org/T424368) [03:29:44] (03open) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/136 (https://phabricator.wikimedia.org/T422184) [03:31:26] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/136 (https://phabricator.wikimedia.org/T422184) [03:32:11] (03update) 10raymond-ndibe: upgrade pytest [repos/cloud/toolforge/builds-cli] (upgrade_python_to_3_13) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/135 (https://phabricator.wikimedia.org/T424368) [03:41:53] (03open) 10raymond-ndibe: upgrade pytest [repos/cloud/toolforge/jobs-api] (upgrade_python_to_3_13) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/296 (https://phabricator.wikimedia.org/T424368) [03:43:12] (03open) 10raymond-ndibe: upgrade pytest [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/80 (https://phabricator.wikimedia.org/T424368) [03:44:33] (03update) 10raymond-ndibe: upgrade pytest [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/80 (https://phabricator.wikimedia.org/T424368) [03:46:14] (03open) 10raymond-ndibe: upgrade pytest [repos/cloud/toolforge/jobs-emailer] (upgrade_python_to_3.13) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/49 (https://phabricator.wikimedia.org/T424368) [03:46:19] (03update) 10raymond-ndibe: upgrade pytest [repos/cloud/toolforge/jobs-emailer] (upgrade_python_to_3.13) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/49 (https://phabricator.wikimedia.org/T424368) [03:47:23] (03open) 10raymond-ndibe: upgrade pytest [repos/cloud/toolforge/components-api] (upgrade_python_to_3_13) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/168 (https://phabricator.wikimedia.org/T424368) [03:50:17] FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [03:53:20] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/136 (https://phabricator.wikimedia.org/T422184) [03:54:39] (03update) 10raymond-ndibe: upgrade pytest [repos/cloud/toolforge/builds-cli] (upgrade_python_to_3_13) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/135 (https://phabricator.wikimedia.org/T424368) [03:55:17] RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:45:56] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-33 (2026-05-05 to 2026-05-19)): Add tests to ensure consistency between OAD example and OpenAPI linter - https://phabricator.wikimedia.org/T419576#11892850 (10KBach) From the technical writers' perspective, it's OK to move the OAD example to the linte... [07:45:59] 06cloud-services-team, 10Data-Services, 06Data-Persistence: imagelinks table missing in pplwiki, several other tables missing in testcommonswiki - https://phabricator.wikimedia.org/T425384#11892976 (10Marostegui) pplwiki seems to have had some weird issues before, we found incorrect schema definitions on `im... [07:59:21] (03open) 10simon04: Fix source code link, fix utm_source=commons.wikimedia.org problem [toolforge-repos/mvc] - 10https://gitlab.wikimedia.org/toolforge-repos/mvc/-/merge_requests/1 [08:16:44] FIRING: IstioGatewayPodMisplaced: istio-gateway pod misplaced - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/IstioGatewayPodMisplaced - https://prometheus-alerts.wmcloud.org/?q=alertname%3DIstioGatewayPodMisplaced [08:21:26] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-system [08:21:41] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system [08:21:44] RESOLVED: IstioGatewayPodMisplaced: istio-gateway pod misplaced - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/IstioGatewayPodMisplaced - https://prometheus-alerts.wmcloud.org/?q=alertname%3DIstioGatewayPodMisplaced [08:29:52] (03update) 10taavi: istio-system: Disable X-Envoy-Peer-Metadata headers [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235 (https://phabricator.wikimedia.org/T392356) [08:30:09] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [08:30:15] (03open) 10simon04: Update UI library to use Wikimedia Codex [toolforge-repos/mvc] - 10https://gitlab.wikimedia.org/toolforge-repos/mvc/-/merge_requests/2 [08:30:21] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway [08:48:45] FIRING: Toolforge Kyverno no policy resources: Toolforge Kyverno has no policy resources - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/Toolforge_Kyverno_no_policy_resources - https://grafana.wmcloud.org/d/kyverno/kyverno?orgId=1&var-DS_PROMETHEUS_KYVERNO=prometheus-tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforge+Kyverno+no+policy+resources [08:48:45] FIRING: Toolforge Kyverno unknown state: Toolforge Kyverno has unknown state. Kyverno might be down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/Toolforge_Kyverno_unknown_state - https://grafana.wmcloud.org/d/kyverno/kyverno?orgId=1&var-DS_PROMETHEUS_KYVERNO=prometheus-tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforge+Kyverno+unknown+state [08:50:27] (03PS1) 10Majavah: t1: Fix f-string MW template syntax [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1283642 [08:50:53] (03CR) 10Majavah: [C:03+2] t1: Fix f-string MW template syntax [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1283642 (owner: 10Majavah) [08:52:19] FIRING: TektonDown: Tekton is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/TektonDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTektonDown [08:52:21] FIRING: MaintainKubeusersDown: maintain-kubeusers is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersDown [08:52:58] FIRING: JobsEmailerDown: JobsEmailer is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/JobsEmailerDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DJobsEmailerDown [08:53:13] (03Merged) 10jenkins-bot: t1: Fix f-string MW template syntax [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1283642 (owner: 10Majavah) [08:53:18] FIRING: EnvvarsApiDown: EnvvarsApi is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/EnvvarsApiDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DEnvvarsApiDown [08:53:23] FIRING: ToolforgeKubernetesNodeNotReady: (no data) Multiple Kubernetes nodes are not ready #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [08:53:25] FIRING: JobsApiDown: JobsApi is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/JobsApiDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DJobsApiDown [08:53:26] FIRING: BuildsApiDown: BuildsApi is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/BuildsApiDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DBuildsApiDown [08:53:34] FIRING: ComponentsApiDown: ComponentsApi is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ComponentsApiDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DComponentsApiDown [08:53:34] FIRING: EnvvarsAdmissionDown: EnvvarsAdmission is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/EnvvarsAdmissionDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DEnvvarsAdmissionDown [08:58:17] FIRING: PrometheusRestarted: Prometheus instance toolsbeta-prometheus-2:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [09:01:22] RESOLVED: [2x] PrometheusK8sCertExpirySoon: Prometheus k8s certificate is about to expire - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/PrometheusK8sCertExpirySoon - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusK8sCertExpirySoon [09:02:19] RESOLVED: TektonDown: Tekton is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/TektonDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTektonDown [09:02:21] RESOLVED: MaintainKubeusersDown: maintain-kubeusers is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersDown [09:02:58] RESOLVED: JobsEmailerDown: JobsEmailer is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/JobsEmailerDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DJobsEmailerDown [09:03:18] RESOLVED: EnvvarsApiDown: EnvvarsApi is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/EnvvarsApiDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DEnvvarsApiDown [09:03:20] FIRING: [2x] PrometheusK8sCertExpirySoon: Prometheus k8s certificate is about to expire - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/PrometheusK8sCertExpirySoon - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusK8sCertExpirySoon [09:03:23] RESOLVED: ToolforgeKubernetesNodeNotReady: (no data) Multiple Kubernetes nodes are not ready #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [09:03:25] RESOLVED: JobsApiDown: JobsApi is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/JobsApiDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DJobsApiDown [09:03:26] RESOLVED: BuildsApiDown: BuildsApi is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/BuildsApiDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DBuildsApiDown [09:03:34] RESOLVED: ComponentsApiDown: ComponentsApi is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ComponentsApiDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DComponentsApiDown [09:03:38] RESOLVED: EnvvarsAdmissionDown: EnvvarsAdmission is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/EnvvarsAdmissionDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DEnvvarsAdmissionDown [09:03:45] RESOLVED: Toolforge Kyverno unknown state: Toolforge Kyverno has unknown state. Kyverno might be down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/Toolforge_Kyverno_unknown_state - https://grafana.wmcloud.org/d/kyverno/kyverno?orgId=1&var-DS_PROMETHEUS_KYVERNO=prometheus-tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforge+Kyverno+unknown+state [09:03:51] RESOLVED: Toolforge Kyverno no policy resources: Toolforge Kyverno has no policy resources - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/Toolforge_Kyverno_no_policy_resources - https://grafana.wmcloud.org/d/kyverno/kyverno?orgId=1&var-DS_PROMETHEUS_KYVERNO=prometheus-tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforge+Kyverno+no+policy+resources [09:26:12] 06cloud-services-team, 10Data-Services, 06Data-Persistence: imagelinks table missing in pplwiki, several other tables missing in testcommonswiki - https://phabricator.wikimedia.org/T425384#11893411 (10Zabe) As @Marostegui said there was an incorrect schema for pplwiki. I would assume that was given by the fa... [09:27:23] 06cloud-services-team, 10Data-Services, 06Data-Persistence: imagelinks table missing in pplwiki, several other tables missing in testcommonswiki - https://phabricator.wikimedia.org/T425384#11893416 (10Marostegui) @fnegri can you take care of the update-views part? thanks! [09:28:17] RESOLVED: PrometheusRestarted: Prometheus instance toolsbeta-prometheus-2:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [09:41:06] 06cloud-services-team, 10Toolforge: Toolforge web egress with Istio/Envoy seems slow - https://phabricator.wikimedia.org/T425172#11893465 (10taavi) This seems to be reproducible from any in-K8s traffic from the new Gateway-specific worker nodes: `lang=shell-session taavi@tools-bastion-14:~ $ kubectl get po -o... [09:48:23] 06cloud-services-team, 10Toolforge: Toolforge web egress with Istio/Envoy seems slow - https://phabricator.wikimedia.org/T425172#11893501 (10taavi) I can also reproduce this from a worker in the VXLAN Cloud VPS network, but not from one in the legacy VLAN network. My first thought was that this was going to b... [10:19:09] (03update) 10fnegri: values.yaml: hoist web image variants to top of config [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/18 (https://phabricator.wikimedia.org/T415322) (owner: 10raymond-ndibe) [10:19:15] (03update) 10fnegri: values.yaml: add image variant name to aliases [repos/cloud/toolforge/image-config] (replace_job_with_webservice_image_variants) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/21 (https://phabricator.wikimedia.org/T415322) (owner: 10raymond-ndibe) [10:23:10] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence: imagelinks table missing in pplwiki, several other tables missing in testcommonswiki - https://phabricator.wikimedia.org/T425384#11893613 (10fnegri) a:03fnegri [10:25:49] 10Cloud-VPS (Quota-requests), 06Content-Transform-Team: Increase quota for wikitextexp project - https://phabricator.wikimedia.org/T425420#11893616 (10fnegri) a:03fnegri [10:25:59] 10Cloud-VPS (Quota-requests): Quota increase for development metrics - https://phabricator.wikimedia.org/T425387#11893618 (10fnegri) a:03fnegri [10:30:54] !log fnegri@cloudcumin1001 wikitextexp START - Cookbook wmcs.openstack.quota_increase by 16 cores, 100 gigabytes, 32778 ram (T425420) [10:30:58] T425420: Increase quota for wikitextexp project - https://phabricator.wikimedia.org/T425420 [10:31:02] !log fnegri@cloudcumin1001 wikitextexp END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 16 cores, 100 gigabytes, 32778 ram (T425420) [10:32:07] 10Cloud-VPS (Quota-requests), 06Content-Transform-Team: Increase quota for wikitextexp project - https://phabricator.wikimedia.org/T425420#11893636 (10fnegri) 05Open→03Resolved [10:33:23] !log fnegri@cloudcumin1001 development-metrics START - Cookbook wmcs.openstack.quota_increase by 12 cores, 8192 ram (T425387) [10:33:25] T425387: Quota increase for development metrics - https://phabricator.wikimedia.org/T425387 [10:33:31] !log fnegri@cloudcumin1001 development-metrics END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 12 cores, 8192 ram (T425387) [10:33:34] RESOLVED: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [10:34:01] 10Cloud-VPS (Quota-requests): Quota increase for development metrics - https://phabricator.wikimedia.org/T425387#11893641 (10fnegri) 05Open→03Resolved [10:42:33] FIRING: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [10:44:27] (03open) 10taavi: kubernetes: Add grace period for misplaced pod alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/62 [10:44:35] (03update) 10taavi: kubernetes: Add grace period for misplaced pod alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/62 [10:49:42] 10Cloud-VPS (Project-requests): Request creation of wolf-a Cloud VPS project - https://phabricator.wikimedia.org/T424192#11893666 (10Aklapper) 05Stalled→03Declined Unfortunately closing this Phabricator task as no further information has been provided. @Moh777tyjer3c: After you have provided all the info... [10:52:33] RESOLVED: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [11:28:49] !log fnegri@cloudcumin1001 thelounge START - Cookbook wmcs.vps.create_project for project thelounge in eqiad1 (T424443) [11:28:51] fnegri@cloudcumin1001: Unknown project "thelounge" [11:28:51] T424443: Request creation of thelounge Cloud VPS project - https://phabricator.wikimedia.org/T424443 [11:29:36] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project thelounge [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/311 (https://phabricator.wikimedia.org/T424443) [11:32:49] fnegri@cloudcumin1001 create_project (PID 2054077) is awaiting input [11:36:42] 10Cloud-VPS (Quota-requests), 06Content-Transform-Team: Increase quota for wikitextexp project - https://phabricator.wikimedia.org/T425420#11893748 (10Jgiannelos) Any chance we can migrate testreduce-1 from mobileappsperformance to wikitextexp ? [11:57:33] 10Tool-wikimonitor: MariaDB Driver Migration - https://phabricator.wikimedia.org/T425534 (10Gerges) 03NEW [11:57:54] 10Tool-wikimonitor: MariaDB Driver Migration - https://phabricator.wikimedia.org/T425534#11893799 (10Gerges) 05Open→03In progress a:03Gerges [12:02:41] (03approved) 10volans: projects: added project thelounge [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/311 (https://phabricator.wikimedia.org/T424443) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [12:14:21] 10Toolforge, 06tools-platform-team: fix bugs in istio-gateway deployment - https://phabricator.wikimedia.org/T425479#11893840 (10taavi) > There are a number of issues causing istio-gateway to fail in lima-kilo. Can you be more specific? [12:19:33] (03merge) 10fnegri: projects: added project thelounge [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/311 (https://phabricator.wikimedia.org/T424443) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [12:20:55] !log fnegri@cloudcumin1001 thelounge END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project thelounge in eqiad1 (T424443) [12:20:57] fnegri@cloudcumin1001: Unknown project "thelounge" [12:20:58] T424443: Request creation of thelounge Cloud VPS project - https://phabricator.wikimedia.org/T424443 [12:21:53] 10Cloud-VPS (Project-requests), 13Patch-For-Review: Request creation of thelounge Cloud VPS project - https://phabricator.wikimedia.org/T424443#11893864 (10fnegri) 05Open→03Resolved The project was created. [12:38:19] 06cloud-services-team, 10Data-Services, 06Data-Persistence: imagelinks table missing in pplwiki, several other tables missing in testcommonswiki - https://phabricator.wikimedia.org/T425384#11893913 (10fnegri) a:05fnegri→03None I updated the view `s5.pplwiki_p.imagelinks` on clouddbs (clouddb1015 and clou... [12:46:36] 10Tool-centralnotice-banner-editor: Update Meta-Wiki documentation for CentralNotice Banner Editor with latest changes - https://phabricator.wikimedia.org/T425540 (10Oyelola_Victoria) 03NEW [13:15:48] (03approved) 10filippo: kubernetes: Add grace period for misplaced pod alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/62 (owner: 10taavi) [13:22:44] (03merge) 10taavi: kubernetes: Add grace period for misplaced pod alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/62 [13:53:03] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-04-24 - 2026-05-15): Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11894273 (10Ahoelzl) p:05Triage→03High [14:22:13] 06cloud-services-team, 10Toolforge, 07ContentSecurityPolicy: Add Content-Security-Policy header enforcing 3rd party web interaction restrictions to proxy responses - https://phabricator.wikimedia.org/T130748#11894551 (10sbassett) Just want to confirm that this task is //explicitly //for applications and infr... [14:31:50] (03PS1) 10Wandji collins: Chore: update dependencies versions [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1283798 [14:32:16] (03CR) 10CI reject: [V:04-1] Chore: update dependencies versions [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1283798 (owner: 10Wandji collins) [14:36:16] (03CR) 10Alien4444: [C:03+2] build: Updating dependencies [labs/xtools] - 10https://gerrit.wikimedia.org/r/1283138 (owner: 10Libraryupgrader) [14:37:47] (03CR) 10CI reject: [V:04-1] build: Updating dependencies [labs/xtools] - 10https://gerrit.wikimedia.org/r/1283138 (owner: 10Libraryupgrader) [14:38:31] 10Tool-centralnotice-banner-editor, 03Wikimedia-Hackathon-2026: Wikimedia Hackathon 2026: CentralNotice Banner Editor: What's new and What's next - https://phabricator.wikimedia.org/T424074#11894688 (10Sadads) Congrats @Oyelola_Victoria so excited you got a chance to present this and get some improvements at t... [14:39:17] 10VPS-project-Codesearch, 06Wikidata Platform Team (Sprint 05 (2026/05/05)): request: index wikidata-platform gitlab repos - https://phabricator.wikimedia.org/T424896#11894689 (10lerickson) [14:39:20] (03PS1) 10Wandji collins: Fix image display for collaboration network. [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1283801 (https://phabricator.wikimedia.org/T425559) [14:39:40] 10VPS-project-Codesearch, 06Wikidata Platform Team (Sprint 05 (2026/05/05)): request: index wikidata-platform gitlab repos - https://phabricator.wikimedia.org/T424896#11894693 (10lerickson) 05Open→03In progress [14:39:50] (03CR) 10CI reject: [V:04-1] Fix image display for collaboration network. [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1283801 (https://phabricator.wikimedia.org/T425559) (owner: 10Wandji collins) [14:41:44] (03Abandoned) 10Wandji collins: Chore: update dependencies versions [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1283798 (owner: 10Wandji collins) [14:45:21] (03PS2) 10Wandji collins: Fix image display for collaboration network. [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1283801 (https://phabricator.wikimedia.org/T425559) [14:45:52] (03CR) 10CI reject: [V:04-1] Fix image display for collaboration network. [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1283801 (https://phabricator.wikimedia.org/T425559) (owner: 10Wandji collins) [14:47:24] (03PS3) 10Wandji collins: Fix image display for collaboration network. [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1283801 (https://phabricator.wikimedia.org/T425559) [14:47:32] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11894723 (10BTullis) I've had a first go at this ECS schema in https://gitlab.wikimedia.org/repos/dat... [14:47:54] (03CR) 10CI reject: [V:04-1] Fix image display for collaboration network. [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1283801 (https://phabricator.wikimedia.org/T425559) (owner: 10Wandji collins) [15:00:14] 10VPS-project-Codesearch, 06Wikidata Platform Team (Sprint 05 (2026/05/05)): request: index wikidata-platform gitlab repos - https://phabricator.wikimedia.org/T424896#11894840 (10gmodena) a:03gmodena [15:00:27] 10VPS-project-Codesearch, 06Wikidata Platform Team (Sprint 05 (2026/05/05)): request: index wikidata-platform gitlab repos - https://phabricator.wikimedia.org/T424896#11894846 (10gmodena) 05In progress→03Resolved [15:06:35] 10VPS-project-Codesearch, 07OKR-Work, 06Wikidata Platform Team (Sprint 05 (2026/05/05)): request: index wikidata-platform gitlab repos - https://phabricator.wikimedia.org/T424896#11894889 (10DSantamaria) [15:06:53] 10VPS-project-Codesearch, 07OKR-Work, 06Wikidata Platform Team (Sprint 05 (2026/05/05)): request: index wikidata-platform gitlab repos - https://phabricator.wikimedia.org/T424896#11894891 (10DSantamaria) [15:17:53] (03approved) 10fnegri: [istio-gateway, utils]: fix istio bugs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1244 (https://phabricator.wikimedia.org/T425479) (owner: 10raymond-ndibe) [15:19:53] 10Toolforge, 06tools-platform-team: [lima-kilo] istio failing to start due to lack of resources - https://phabricator.wikimedia.org/T425479#11894995 (10fnegri) [15:20:52] 10Toolforge, 06tools-platform-team: [lima-kilo] istio failing to start due to lack of resources - https://phabricator.wikimedia.org/T425479#11895000 (10fnegri) > These should be happening in prod too, not sure why it's not causing issues. I might be wrong, but I think in prod it's working fine because we hav... [15:21:57] (03unapproved) 10fnegri: [istio-gateway, utils]: fix istio bugs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1244 (https://phabricator.wikimedia.org/T425479) (owner: 10raymond-ndibe) [15:26:22] (03update) 10fnegri: [istio-gateway, utils]: fix istio bugs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1244 (https://phabricator.wikimedia.org/T425479) (owner: 10raymond-ndibe) [15:31:15] (03PS4) 10Wandji collins: Fix image display for collaboration network. [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1283801 (https://phabricator.wikimedia.org/T425559) [15:34:42] (03CR) 10CI reject: [V:04-1] Fix image display for collaboration network. [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1283801 (https://phabricator.wikimedia.org/T425559) (owner: 10Wandji collins) [15:36:07] 06cloud-services-team, 10Toolforge: Make toolsbeta paging alerts less confusing - https://phabricator.wikimedia.org/T424814#11895186 (10fgiunchedi) [15:36:10] (03update) 10fnegri: [istio-gateway, utils]: fix istio bugs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1244 (https://phabricator.wikimedia.org/T425479) (owner: 10raymond-ndibe) [16:35:09] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25447977391 (https://github.com/cluebotng/component-configs/commits/bd39eba234686d41e4d889def1ed91fc28e8fdf6) [16:35:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [16:40:24] 10Tool-wikimedia-attribution: Attribution API: Include wprov parameters in response URLs - https://phabricator.wikimedia.org/T425576 (10Sarai-WMF) 03NEW [16:40:58] (03approved) 10fnegri: main.yaml, kind.yaml: remove the deployment of ingress-nginx [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/321 (https://phabricator.wikimedia.org/T425480) (owner: 10raymond-ndibe) [16:41:03] (03update) 10fnegri: main.yaml, kind.yaml: remove the deployment of ingress-nginx [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/321 (https://phabricator.wikimedia.org/T425480) (owner: 10raymond-ndibe) [16:41:59] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [lima-kilo] remove ingress-nginx deployment - https://phabricator.wikimedia.org/T425480#11895456 (10fnegri) [16:42:38] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [lima-kilo] remove ingress-nginx deployment - https://phabricator.wikimedia.org/T425480#11895457 (10fnegri) 05Open→03In progress [16:44:06] (03approved) 10fnegri: upgrade pytest [repos/cloud/toolforge/builds-cli] (upgrade_python_to_3_13) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/135 (https://phabricator.wikimedia.org/T424368) (owner: 10raymond-ndibe) [16:44:11] (03update) 10fnegri: upgrade pytest [repos/cloud/toolforge/builds-cli] (upgrade_python_to_3_13) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/135 (https://phabricator.wikimedia.org/T424368) (owner: 10raymond-ndibe) [16:44:14] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448365994 (https://github.com/cluebotng/component-configs/commits/7eaf56e56e3eacfd97455f271b152f369ca2b3f4) [16:44:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [16:45:06] (03approved) 10fnegri: upgrade python to 3.13 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/136 (https://phabricator.wikimedia.org/T422184) (owner: 10raymond-ndibe) [16:45:11] 10Tool-wikimedia-attribution: Attribution API: Include wprov parameters in response URLs - https://phabricator.wikimedia.org/T425576#11895467 (10Sarai-WMF) [16:46:02] (03approved) 10fnegri: upgrade pytest [repos/cloud/toolforge/components-api] (upgrade_python_to_3_13) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/168 (https://phabricator.wikimedia.org/T424368) (owner: 10raymond-ndibe) [16:46:09] (03approved) 10fnegri: upgrade pytest [repos/cloud/toolforge/jobs-emailer] (upgrade_python_to_3.13) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/49 (https://phabricator.wikimedia.org/T424368) (owner: 10raymond-ndibe) [16:46:12] (03approved) 10fnegri: upgrade pytest [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/80 (https://phabricator.wikimedia.org/T424368) (owner: 10raymond-ndibe) [16:46:25] (03approved) 10fnegri: upgrade pytest [repos/cloud/toolforge/jobs-api] (upgrade_python_to_3_13) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/296 (https://phabricator.wikimedia.org/T424368) (owner: 10raymond-ndibe) [16:46:36] (03approved) 10fnegri: upgrade pytest [repos/cloud/toolforge/envvars-cli] (upgrade_python_to_3.13) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/110 (https://phabricator.wikimedia.org/T424368) (owner: 10raymond-ndibe) [16:46:38] (03update) 10fnegri: upgrade pytest [repos/cloud/toolforge/envvars-cli] (upgrade_python_to_3.13) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/110 (https://phabricator.wikimedia.org/T424368) (owner: 10raymond-ndibe) [16:46:53] (03update) 10fnegri: upgrade pytest [repos/cloud/toolforge/jobs-api] (upgrade_python_to_3_13) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/296 (https://phabricator.wikimedia.org/T424368) (owner: 10raymond-ndibe) [16:46:55] (03update) 10fnegri: upgrade pytest [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/80 (https://phabricator.wikimedia.org/T424368) (owner: 10raymond-ndibe) [16:46:57] (03update) 10fnegri: upgrade pytest [repos/cloud/toolforge/jobs-emailer] (upgrade_python_to_3.13) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/49 (https://phabricator.wikimedia.org/T424368) (owner: 10raymond-ndibe) [16:46:58] (03update) 10fnegri: upgrade pytest [repos/cloud/toolforge/components-api] (upgrade_python_to_3_13) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/168 (https://phabricator.wikimedia.org/T424368) (owner: 10raymond-ndibe) [16:48:20] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448624822 (https://github.com/cluebotng/component-configs/commits/92c52e117c73389fe04c66d53ad083eef60ef387) [16:48:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [16:51:09] 10Tool-humaniki-2: Favicon - https://phabricator.wikimedia.org/T423004#11895492 (10Danya) Hi, I don’t have one and I don’t have the skill to create one. Feel free to create one if you want, but I will need to have the opinion of //les sans pagEs// before integrating it into the tool, as this is a project directe... [16:51:51] (03approved) 10fnegri: upgrade python to 3.13 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/109 (https://phabricator.wikimedia.org/T422184) (owner: 10raymond-ndibe) [16:51:52] (03update) 10fnegri: upgrade python to 3.13 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/109 (https://phabricator.wikimedia.org/T422184) (owner: 10raymond-ndibe) [16:52:08] (03update) 10fnegri: upgrade python to 3.13 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/136 (https://phabricator.wikimedia.org/T422184) (owner: 10raymond-ndibe) [16:52:29] (03update) 10fnegri: upgrade_blubber_to_v0_24_0 [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/84 (https://phabricator.wikimedia.org/T422184) (owner: 10raymond-ndibe) [16:52:43] (03update) 10fnegri: upgrade_blubber_to_v0_24_0 [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/84 (https://phabricator.wikimedia.org/T422184) (owner: 10raymond-ndibe) [16:53:06] (03approved) 10fnegri: upgrade_blubber_to_v0_24_0 [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/84 (https://phabricator.wikimedia.org/T422184) (owner: 10raymond-ndibe) [16:59:16] 10Tool-humaniki-2: Create a contribution guide - https://phabricator.wikimedia.org/T425486#11895563 (10Danya) 05Open→03Resolved a:03Danya [16:59:46] 10Tool-humaniki-2: Create a contribution guide - https://phabricator.wikimedia.org/T425486#11895565 (10Danya) 05Resolved→03Open [16:59:47] 10Tool-humaniki-2: Create a contribution guide - https://phabricator.wikimedia.org/T425486#11895567 (10Danya) 05Open→03In progress [17:00:38] (03open) 10danyya: Draft: Contribution guide [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/34 [17:02:42] (03update) 10danyya: Draft: Home page [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/33 [17:23:53] 10Tool-wikimonitor: MariaDB Driver Migration - https://phabricator.wikimedia.org/T425534#11895639 (10Gerges) 05In progress→03Resolved [17:27:23] 10Tool-humaniki-2: Header items overflow on mobile screens - https://phabricator.wikimedia.org/T422344#11895646 (10Danya) a:05Danya→03None [17:27:29] 10Tool-humaniki-2: Add button to save page state as GET params - https://phabricator.wikimedia.org/T422330#11895647 (10Danya) a:05Danya→03None [17:27:34] 10Tool-humaniki-2: Localize table texts - https://phabricator.wikimedia.org/T423787#11895648 (10Danya) a:05Danya→03None [17:27:40] 10Tool-humaniki-2: Choose licence for website’s content - https://phabricator.wikimedia.org/T422336#11895649 (10Danya) a:05Danya→03None [17:27:44] 10Tool-humaniki-2: List data sources - https://phabricator.wikimedia.org/T422332#11895650 (10Danya) a:05Danya→03None [17:27:53] 10Tool-humaniki-2: Add field to filter gender tables - https://phabricator.wikimedia.org/T422337#11895652 (10Danya) a:05Danya→03None [17:27:58] 10Tool-humaniki-2: Keep filters states when updating datasets - https://phabricator.wikimedia.org/T422335#11895653 (10Danya) a:05Danya→03None [17:28:02] 10Tool-humaniki-2: Admin page - https://phabricator.wikimedia.org/T425293#11895654 (10Danya) a:05Danya→03None [17:50:18] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25451592437 (https://github.com/cluebotng/component-configs/commits/88790d34de5a0bca24d8685f3f7df749b052647b) [17:50:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [17:56:00] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25451879961 (https://github.com/cluebotng/component-configs/commits/966f5e00a0a1098153ca8b6af11bc1124a119a1b) [17:56:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [17:56:12] !log tools.cluebotng-editsets Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25451879892 (https://github.com/cluebotng/component-configs/commits/966f5e00a0a1098153ca8b6af11bc1124a119a1b) [17:56:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [18:03:04] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452220343 (https://github.com/cluebotng/component-configs/commits/2ba80dd76efd2798ab7d1fedddf2a7baeab1118b) [18:03:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [18:14:40] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452753895 (https://github.com/cluebotng/component-configs/commits/6680b6abb8b5b775f79893981bd070c9e17f0358) [18:14:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [18:19:48] 10Tool-wikimedia-attribution, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Attribution API: Include wprov parameters in response URLs - https://phabricator.wikimedia.org/T425576#11895834 (10pmiazga) [18:30:43] (03update) 10danyya: Draft: Contribution guide [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/34 [18:31:02] (03update) 10danyya: Contribution guide [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/34 [18:33:59] (03update) 10danyya: Contribution guide [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/34 [18:35:43] (03merge) 10danyya: Contribution guide [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/34 [18:35:45] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25453754913 (https://github.com/cluebotng/component-configs/commits/92f164d1ab158aea1f76cd0a787f33ffe4017e85) [18:35:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [18:38:12] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25453754970 (https://github.com/cluebotng/component-configs/commits/92f164d1ab158aea1f76cd0a787f33ffe4017e85) [18:38:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [18:44:11] 10Tool-wikimonitor: [WikiStreamService] 30-Minute SseEmitter Timeout with Event Resumption - https://phabricator.wikimedia.org/T422194#11895894 (10Gerges) Thanks @Praffq. [18:44:16] 10Tool-wikimonitor: [WikiStreamService] 30-Minute SseEmitter Timeout with Event Resumption - https://phabricator.wikimedia.org/T422194#11895895 (10Gerges) 05Open→03Resolved [19:30:14] 10Tool-humaniki-2: Admin page - https://phabricator.wikimedia.org/T425293#11896018 (10Danya) a:03Danya [19:30:21] 10Tool-humaniki-2: About page - https://phabricator.wikimedia.org/T425022#11896020 (10Danya) a:03Danya [19:30:29] 10Tool-humaniki-2: Add page to query data - https://phabricator.wikimedia.org/T425023#11896021 (10Danya) a:03Danya [19:37:25] 10Tool-humaniki-2: Create a contribution guide - https://phabricator.wikimedia.org/T425486#11896051 (10Danya) 05In progress→03Resolved [19:39:11] 10Tool-upload-workbench: Add Beta indicator and bug-report / source links; refresh README - https://phabricator.wikimedia.org/T425587 (10Daanvr) 03NEW p:05Triage→03Medium [19:39:42] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-33 (2026-05-05 to 2026-05-19)), 07OKR-Work: [SPIKE] Add security scheme information to MediaWiki REST API description - https://phabricator.wikimedia.org/T423552#11896065 (10daniel) In general I think the spec of an API module should be the same no... [19:47:34] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11896077 (10Ottomata) Oh! Saw the MR before the Phab. Commented about development/ there too. It m... [20:07:17] 10Tool-wikimedia-attribution, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Attribution API: Include wprov parameters in response URLs - https://phabricator.wikimedia.org/T425576#11896132 (10RHo) [20:13:52] (03update) 10wikigit: Add the example spec to the tests [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/20 (https://phabricator.wikimedia.org/T419576) [20:13:55] (03open) 10wikigit: Add the example spec to the tests [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/20 (https://phabricator.wikimedia.org/T419576) [20:14:14] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-33 (2026-05-05 to 2026-05-19)), 13Patch-For-Review: Add tests to ensure consistency between OAD example and OpenAPI linter - https://phabricator.wikimedia.org/T419576#11896150 (10Mooeypoo) a:03Mooeypoo [20:14:29] 10Tool-schedule-deployment, 06Release-Engineering-Team, 06ServiceOps new, 13Patch-For-Review, 03Wikimedia-Hackathon-2026: Extend functionality to support MediaWiki infrastructure Windows and related repos - https://phabricator.wikimedia.org/T385007#11896153 (10RLazarus) 05Open→03In progress a:03jiji... [20:14:38] 10Tool-schedule-deployment, 06Release-Engineering-Team, 06ServiceOps new, 13Patch-For-Review, 03Wikimedia-Hackathon-2026: Extend functionality to support MediaWiki infrastructure Windows and related repos - https://phabricator.wikimedia.org/T385007#11896165 (10RLazarus) p:05Triage→03Medium [20:15:53] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-33 (2026-05-05 to 2026-05-19)), 13Patch-For-Review: Add tests to ensure consistency between OAD example and OpenAPI linter - https://phabricator.wikimedia.org/T419576#11896168 (10Mooeypoo) I've moved all the fiels from the doc repo (including the RE... [20:40:13] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25459711948 (https://github.com/cluebotng/component-configs/commits/3e0d938cc4bbeb5069a3e0b0d864df7eabf78c23) [20:40:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [20:45:30] 10Tool-humaniki-2: Stop counting elements with no specified gender - https://phabricator.wikimedia.org/T425592 (10Danya) 03NEW [20:45:34] 10Tool-humaniki-2: Stop counting elements with no specified gender - https://phabricator.wikimedia.org/T425592#11896225 (10Danya) [20:45:41] 10Tool-humaniki-2: Stop counting elements with no specified gender - https://phabricator.wikimedia.org/T425592#11896229 (10Danya) p:05Triage→03Medium [20:53:59] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460310332 (https://github.com/cluebotng/component-configs/commits/511feb967af13f77e59803c69bc3447124733524) [20:54:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [20:57:14] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460509016 (https://github.com/cluebotng/component-configs/commits/b9c2dde9382e1a3a5502f5218ecc7eaef6b4f9ec) [20:57:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [21:30:04] (03CR) 10Alien4444: build: Updating dependencies (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1283138 (owner: 10Libraryupgrader) [21:31:15] (03CR) 10Alien4444: "recheck" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1283138 (owner: 10Libraryupgrader) [21:32:50] (03CR) 10Alien4444: [C:03+2] build: Updating dependencies [labs/xtools] - 10https://gerrit.wikimedia.org/r/1283138 (owner: 10Libraryupgrader) [21:33:02] 10Tool-humaniki-2: Favicon - https://phabricator.wikimedia.org/T423004#11896313 (10Tiisu) >>! In T423004#11895492, @Danya wrote: > Hi, I don’t have one and I don’t have the skills to create one. Feel free to create it if you want, but it will need to get the approval of //les sans pagEs// before integrating it i... [21:38:10] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462408819 (https://github.com/cluebotng/component-configs/commits/c905103b2bd6129e76da6cf191567f82913c8349) [21:38:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [21:50:19] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462948750 (https://github.com/cluebotng/component-configs/commits/78047ed0bf8def9f4ff2b181d1710a274328a5b3) [21:50:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [21:52:51] (03PS9) 10Alien4444: Readd third-party suppport [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) [21:58:41] (03update) 10danyya: Draft: Home page [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/33 [21:59:51] (03PS1) 10Alien4444: Repository: fix typo made while retabbing a query [labs/xtools] - 10https://gerrit.wikimedia.org/r/1283974 [22:01:17] (03update) 10danyya: Draft: Home page [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/33 [22:03:56] (03CR) 10Alien4444: [C:03+2] Repository: fix typo made while retabbing a query [labs/xtools] - 10https://gerrit.wikimedia.org/r/1283974 (owner: 10Alien4444) [22:04:57] (03Merged) 10jenkins-bot: Repository: fix typo made while retabbing a query [labs/xtools] - 10https://gerrit.wikimedia.org/r/1283974 (owner: 10Alien4444) [22:12:32] (03PS1) 10Alien4444: 3.24.5 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1283981 [22:16:34] (03CR) 10Alien4444: [C:03+2] 3.24.5 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1283981 (owner: 10Alien4444) [22:18:53] (03Merged) 10jenkins-bot: 3.24.5 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1283981 (owner: 10Alien4444) [22:51:36] (03PS10) 10Alien4444: Readd third-party suppport [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) [23:05:03] 10Tool-humaniki-2: Button to download the whole database? - https://phabricator.wikimedia.org/T424282#11896635 (10Tiisu) a:03Tiisu I am working on this ticket [23:05:33] 10Tool-humaniki-2: Button to download the whole database? - https://phabricator.wikimedia.org/T424282#11896639 (10Tiisu) 05Open→03In progress [23:06:29] (03PS11) 10Alien4444: Add proofreadpage quality support [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) [23:17:48] 10Tool-etherpad-backup, 03Wikimedia-Hackathon-2026: Archive etherpads mentioned in hackathon/techconf tasks - https://phabricator.wikimedia.org/T417207#11896648 (10bd808) The run from T417207#11892073 stopped with a `mwclient.errors.APIError` raised because a pad title hit on an abuse filter on Wikitech while... [23:22:07] 10Tool-etherpad-backup, 03Wikimedia-Hackathon-2026: Archive etherpads mentioned in hackathon/techconf tasks - https://phabricator.wikimedia.org/T417207#11896662 (10bd808) `lang=shell-session tools.etherpad-backup@tools-bastion-14:~$ webservice buildservice shell --mount all I have no name!@shell-1778109079:/wo... [23:27:09] (03open) 10tiisu: Draft: Resolve T424282 "Add a button to download the whole database either the sqlite file or an xml export" [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/35 [23:39:20] (03update) 10tiisu: Draft: Resolve T424282 "Add a button to download the whole database either the sqlite file or an xml export" [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/35 [23:46:32] 06cloud-services-team, 10Toolforge, 07ContentSecurityPolicy: Add Content-Security-Policy header enforcing 3rd party web interaction restrictions to proxy responses - https://phabricator.wikimedia.org/T130748#11896705 (10bd808) >>! In T130748#11894551, @sbassett wrote: > Just want to confirm that this task is...