[02:26:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:31:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:53:33] (03update) 10bd808: Draw the rest of the owl [toolforge-repos/etherpad-backup] - 10https://gitlab.wikimedia.org/toolforge-repos/etherpad-backup/-/merge_requests/1 (https://phabricator.wikimedia.org/T422958 https://phabricator.wikimedia.org/T423014) [04:05:00] (03update) 10bd808: Draw the rest of the owl [toolforge-repos/etherpad-backup] - 10https://gitlab.wikimedia.org/toolforge-repos/etherpad-backup/-/merge_requests/1 (https://phabricator.wikimedia.org/T422958 https://phabricator.wikimedia.org/T423014) [04:24:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:39:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:52:14] (03update) 10bd808: Draw the rest of the owl [toolforge-repos/etherpad-backup] - 10https://gitlab.wikimedia.org/toolforge-repos/etherpad-backup/-/merge_requests/1 (https://phabricator.wikimedia.org/T422958 https://phabricator.wikimedia.org/T423014) [05:09:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [05:14:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [05:55:33] 10Tool-glam-tool-hospital, 03Wikimedia-Hackathon-2026: GLAM Tool Hospital - https://phabricator.wikimedia.org/T424464 (10Dactylantha) 03NEW [05:59:17] 10Tool-glam-tool-hospital, 03Wikimedia-Hackathon-2026: GLAM Tool Hospital - https://phabricator.wikimedia.org/T424464#11859233 (10Dactylantha) [05:59:50] 10Tool-glam-tool-hospital, 03Wikimedia-Hackathon-2026: GLAM Tool Hospital - https://phabricator.wikimedia.org/T424464#11859234 (10Dactylantha) [06:02:22] 10Tool-glam-tool-hospital, 03Wikimedia-Hackathon-2026: GLAM Tool Hospital - https://phabricator.wikimedia.org/T424464#11859235 (10Dactylantha) [06:08:44] (03open) 10samwilson: Fix up MINT_URL env config variable [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/10 (https://phabricator.wikimedia.org/T206436) [06:09:46] (03update) 10samwilson: Fix up MINT_URL env config variable [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/10 (https://phabricator.wikimedia.org/T206436) [06:18:30] (03update) 10samwilson: Fix up MINT_URL env config variable [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/10 (https://phabricator.wikimedia.org/T206436) [07:25:28] (03update) 10vriaa: Draft: fix: spacing overflow bug [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/59 [07:26:34] (03update) 10vriaa: fix: spacing overflow bug [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/59 [07:27:07] (03merge) 10vriaa: fix: spacing overflow bug [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/59 [09:00:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:05:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:43:14] 10superset.wmcloud.org, 10Automoderator, 06Moderator-Tools-Team, 06Product-Analytics: Migrate Automoderator activity dashboard from wmcloud to wikimedia Superset instance - https://phabricator.wikimedia.org/T418836#11860036 (10Samwalton9-WMF) 05Open→03Declined Per the announcement that the public S... [09:46:17] 10superset.wmcloud.org, 10Automoderator, 06Moderator-Tools-Team: Find a new way to provide Wikipedia communities with automated metrics about Automoderator's behaviour - https://phabricator.wikimedia.org/T417388#11860050 (10Samwalton9-WMF) 05Open→03Declined The public Superset instance is being maint... [10:43:16] (03update) 10vriaa: feat: add support for templates with CTAs [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/58 [10:44:35] (03merge) 10vriaa: feat: add support for templates with CTAs [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/58 [10:57:53] (03PS1) 10Krinkle: docroot: Adopt standard thumb width on https://cvn.wmcloud.org page [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1277473 [11:00:26] (03update) 10vriaa: fix: spacing overflow bug [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/59 (https://phabricator.wikimedia.org/T423934) [11:01:21] (03CR) 10Krinkle: [V:03+2 C:03+2] docroot: Adopt standard thumb width on https://cvn.wmcloud.org page [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1277473 (owner: 10Krinkle) [11:09:02] (03update) 10fnegri: builds-api: update heroku builder and runner images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1225 (https://phabricator.wikimedia.org/T423417) [11:09:56] (03update) 10fnegri: README: Clarify process for updating Heroku images [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/88 (https://phabricator.wikimedia.org/T424362) [11:39:43] 10Tool-centralnotice-banner-editor: Text element grows in size when padding is increased and can overflow - https://phabricator.wikimedia.org/T423934#11860578 (10Oyelola_Victoria) 05Open→03Resolved [11:41:41] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Audit tools memory requests vs actual usage - https://phabricator.wikimedia.org/T420565#11860589 (10fgiunchedi) [12:39:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:44:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:48:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:53:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:59:42] 06cloud-services-team, 10Cloud-VPS, 10Lingua-Libre: Configure vanity domain for lingualibre - https://phabricator.wikimedia.org/T419525#11860773 (10Volans) Added: `lang=yaml customlingualibre: CN: dev.lingualibre.org SNI: - dev.lingualibre.org authorized_regexes: - ^.+\.project-proxy\.... [14:01:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:03:08] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 07Epic: Ability to create new views for tools and analytic purposes - https://phabricator.wikimedia.org/T424524 (10saper) 03NEW [14:06:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:19:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:24:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:32:51] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Define update process for Toolforge builder/runner images - https://phabricator.wikimedia.org/T424362#11861393 (10fnegri) @bd808 thanks for the feedback! I think we can wait for @dcaro to be back before deciding on a policy, but we can start collecting... [14:33:59] (03update) 10fnegri: builds-api: update heroku builder and runner images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1225 (https://phabricator.wikimedia.org/T423417 https://phabricator.wikimedia.org/T424362) [14:34:19] (03update) 10fnegri: builds-api: update heroku builder and runner images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1225 (https://phabricator.wikimedia.org/T423417 https://phabricator.wikimedia.org/T424362) [14:37:07] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Define update process for Toolforge builder/runner images - https://phabricator.wikimedia.org/T424362#11861440 (10fnegri) a:03fnegri [14:40:40] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Define update process for Toolforge builder/runner images - https://phabricator.wikimedia.org/T424362#11861470 (10fnegri) 05Open→03In progress [14:45:30] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06tools-platform-team, 06Data-Persistence: Install a clouddb hosts with Debian Trixie - https://phabricator.wikimedia.org/T415165#11861521 (10fnegri) A temporary replacement for clouddb1019 was found (see {T409557}) so we can go back again to the... [14:49:10] 10Tools, 06All-and-every-Wikisource: ws-page-game.toolforge.org does not receive page scan thumbnails (HTTP 429) - https://phabricator.wikimedia.org/T419239#11861560 (10Ladsgroup) [14:50:15] 10Tools, 06All-and-every-Wikisource: ws-page-game.toolforge.org does not receive page scan thumbnails (HTTP 429) - https://phabricator.wikimedia.org/T419239#11861575 (10Ladsgroup) We are not going to fix tools. It's responsibility of the tool owners to fix it (plus it's mostly fixed?) [14:57:55] (03merge) 10pepepiton: Messages for age at death (fr, eo) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/184 (owner: 10maldaxura) [15:10:29] (03open) 10raymond-ndibe: loki.py: fix logs ordering [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/18 (https://phabricator.wikimedia.org/T401552) [15:13:00] 10Toolforge, 06tools-platform-team: Cannot output "Ερευνητής Αλήθειας" - https://phabricator.wikimedia.org/T367663#11861684 (10Raymond_Ndibe) @Soda is this still happening? I can't reproduce it: ` local.tf-test@lima-kilo:~$ cat test.py #!/usr/bin/env python3 print("Ερευνητής Αλήθειας") print("Some other stuff"... [15:16:01] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linting - info.license - https://phabricator.wikimedia.org/T422908#11861698 (10KBach) p:05Triage→03Low [15:16:21] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: [SPIKE] Verify info.license in non-generated specs - https://phabricator.wikimedia.org/T422911#11861700 (10KBach) p:05Triage→03Low [15:17:15] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linting - enum descriptions - https://phabricator.wikimedia.org/T424210#11861702 (10KBach) p:05Medium→03Low [15:17:50] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Essential-Work: Spec caching: Cache validation results for identical OpenAPI specs - https://phabricator.wikimedia.org/T415916#11861714 (10KBach) p:05Medium→03Low [15:17:57] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Essential-Work: [SPIKE] Figure out best approach for spec result caching - https://phabricator.wikimedia.org/T415917#11861715 (10KBach) p:05Medium→03Low [15:18:05] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Essential-Work: Lazy loading of ruleset: Implement caching layer for Spectral ruleset - https://phabricator.wikimedia.org/T415920#11861716 (10KBach) p:05Medium→03Low [15:18:18] 10Cloud-VPS (Project-requests): Request creation of thelounge Cloud VPS project - https://phabricator.wikimedia.org/T424443#11861720 (10bd808) I have other concerns like potential problems with spam management and IP address/network reputation upstream at libera.chat and handling of IRC account credentials by a... [15:29:56] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11861781 (10LSobanski) p:05Triage→03Medium [15:31:18] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 07Epic: Ability to create new views for tools and analytic purposes - https://phabricator.wikimedia.org/T424524#11861789 (10bd808) [15:33:16] (03open) 10raymond-ndibe: toolforge_get_versions.sh: remove ingress-nginx [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1226 [15:33:50] (03update) 10raymond-ndibe: toolforge_get_versions.sh: remove ingress-nginx [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1226 [15:36:41] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 07Epic: Ability to create new views for tools and analytic purposes - https://phabricator.wikimedia.org/T424524#11861819 (10bd808) It isn't exactly the same, but this task does seem to rhyme with {T173511} where we were trying to figure out a sust... [15:36:59] (03open) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/292 (https://phabricator.wikimedia.org/T422184) [15:38:46] (03open) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/45 (https://phabricator.wikimedia.org/T422184) [15:38:58] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/45 (https://phabricator.wikimedia.org/T422184) [15:45:59] (03open) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/91 (https://phabricator.wikimedia.org/T422184) [15:50:02] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/45 (https://phabricator.wikimedia.org/T422184) [16:00:28] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Define update process for Toolforge builder/runner images - https://phabricator.wikimedia.org/T424362#11861931 (10bd808) >>! In T424362#11861392, @fnegri wrote: > Your proposal looks good to me, one caveat is that "minor version bumps" are a bit confusi... [16:47:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:52:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:21:28] 10Tools, 06LPL Onboarding and Development, 10Starter Kit, 10LPL Projects (Starter kit): Host and deploy StarterKit MVP on Toolforge - https://phabricator.wikimedia.org/T420074#11862354 (10MaryMunyoki) [17:21:48] 10Tools, 06LPL Onboarding and Development, 10Starter Kit, 10LPL Projects (Starter kit): Host and deploy StarterKit tool MVP on Toolforge - https://phabricator.wikimedia.org/T420074#11862358 (10MaryMunyoki) [18:02:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [18:04:13] (03open) 10ragesoss: Add batch POST endpoint for counting references across many revisions [toolforge-repos/reference-counter] - 10https://gitlab.wikimedia.org/toolforge-repos/reference-counter/-/merge_requests/1 [18:07:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [18:12:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:16:32] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:16:43] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [18:17:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:21:32] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:26:32] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:26:34] 10Tools, 06LPL Onboarding and Development, 10Starter Kit, 10LPL Projects (Starter kit): Host and deploy StarterKit tool MVP on Toolforge - https://phabricator.wikimedia.org/T420074#11862605 (10eamedina) Leaving here related commit for documenting purposes, we can close this task now https://gitlab.wikimed... [18:31:32] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:31:43] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [18:35:01] 10Cloud-VPS (Quota-requests): Quota increase for development metrics - https://phabricator.wikimedia.org/T424564 (10thcipriani) 03NEW [18:36:08] 10Tools, 06LPL Onboarding and Development, 10Starter Kit, 10LPL Projects (Starter kit): Host and deploy StarterKit tool MVP on Toolforge - https://phabricator.wikimedia.org/T420074#11862645 (10MaryMunyoki) 05Open→03Resolved [18:41:32] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:45:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:50:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:52:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:57:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:38:40] 10Toolforge, 06tools-platform-team: Cannot output "Ερευνητής Αλήθειας" - https://phabricator.wikimedia.org/T367663#11862875 (10Soda) I will check! [19:51:05] (03CR) 10Alien4444: [C:03+2] Localisation updates from https://translatewiki.net. [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277528 (owner: 10L10n-bot) [20:25:04] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Quota increase for development metrics - https://phabricator.wikimedia.org/T424564#11863055 (10bd808) +1 [20:27:18] 10Tool-humaniki-2: Prevent page freeze when loading data - https://phabricator.wikimedia.org/T424578 (10Danya) 03NEW [20:27:21] 10Tool-humaniki-2: Prevent page freeze when loading data - https://phabricator.wikimedia.org/T424578#11863073 (10Danya) p:05Triage→03Medium [20:34:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:44:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:03:40] 10Tool-glam-tool-hospital, 03Wikimedia-Hackathon-2026: Wikimedia Hackathon 2026: GLAM Tool Hospital - https://phabricator.wikimedia.org/T424464#11863193 (10Dactylantha) [21:31:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:36:08] 10Tool-glam-tool-hospital: [TEST] GLAMorgan - Bug - API connectivity test - https://phabricator.wikimedia.org/T424584 (10Dactylantha) 03NEW [21:39:03] 10Tool-glam-tool-hospital: [TEST] OpenRefine - Feature request - feature request - https://phabricator.wikimedia.org/T424585 (10Dactylantha) 03NEW [21:46:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:36:21] 10Tool-glam-tool-hospital: [TEST] Pattypan - Bug - Test - https://phabricator.wikimedia.org/T424589 (10Dactylantha) 03NEW [22:51:13] (03approved) 10aaron: chore(tests): split rule tests by validation domain (T415914) [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/16 (owner: 10aghirelli) [22:52:44] (03merge) 10aaron: chore(tests): split rule tests by validation domain (T415914) [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/16 (owner: 10aghirelli) [22:53:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:08:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:12:51] 10Tool-glam-tool-hospital: [TEST] GLAMorgan - Feature request - API connectivity test - https://phabricator.wikimedia.org/T424594 (10Dactylantha) 03NEW [23:14:48] 10Tool-glam-tool-hospital: [TEST] GLAMorgan - Feature request - API connectivity test - https://phabricator.wikimedia.org/T424594#11863627 (10Dactylantha) 05Open→03Resolved a:03Dactylantha [23:26:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [23:31:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors