[01:46:38] 10Tool-wikimonitor, 07good first task: [WikiStreamService] Fix silent SSE stream freeze on half-open TCP drops - https://phabricator.wikimedia.org/T435612#12244090 (10CodeSparks45) a:03CodeSparks45 [03:23:14] (03Abandoned) 10Krinkle: Build: Update build system [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1143806 (owner: 10Slyngshede) [03:56:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:01:05] 10Tool-wikinewsie: Better category building for AI-curated feeds for Wikinewsie - https://phabricator.wikimedia.org/T435709 (10Pharos) 03NEW [04:01:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:05:00] 10Tool-wikinewsie: Excluded categories nort working in Wikinewsie custom portals - https://phabricator.wikimedia.org/T435710 (10Pharos) 03NEW [04:05:13] 10Tool-wikinewsie: Excluded categories not working in Wikinewsie custom portals - https://phabricator.wikimedia.org/T435710#12244130 (10Pharos) [04:05:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:10:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:15:35] (03PS17) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [04:37:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:42:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:45:10] (03PS18) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [04:47:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:42:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:47:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:02:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:26:36] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [pywikibot-buildservice] New upstream release for Pywikibot - https://phabricator.wikimedia.org/T431056#12244172 (10Xqt) Pywikibot on Toolforge is **five releases behind** the current upstream stable version, 11.7.0. Is there an issue preventing regu... [08:36:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:41:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:46:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:44:36] (03update) 10mahveotm: jobs: Add timeout support for one-off jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/373 (https://phabricator.wikimedia.org/T401110) [10:08:20] (03update) 10mahveotm: jobs: Add timeout support for one-off jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/373 (https://phabricator.wikimedia.org/T401110) [10:32:20] (03open) 10mahveotm: cli: add timeout support for one-off jobs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/176 (https://phabricator.wikimedia.org/T401110) [10:34:11] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api,jobs-cli] Support `--timeout` for one-off jobs - https://phabricator.wikimedia.org/T401110#12244217 (10Mahveotm) 05Open→03In progress [10:34:16] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api,jobs-cli] Support `--timeout` for one-off jobs - https://phabricator.wikimedia.org/T401110#12244218 (10Mahveotm) a:03Mahveotm [11:38:19] (03open) 10mahveotm: emailer: include pod status message in failure emails [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/58 (https://phabricator.wikimedia.org/T384252) [12:05:44] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-emailer] If the pod is in error status, try to get the status.message field in the email, otherwise just 'error' is not that useful - https://phabricator.wikimedia.org/T384252#12244274 (10Mahveotm) 05Open→03In progress [12:05:57] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/75 (owner: 10l10n-bot) [12:06:01] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/75 (owner: 10l10n-bot) [12:06:45] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-emailer] If the pod is in error status, try to get the status.message field in the email, otherwise just 'error' is not that useful - https://phabricator.wikimedia.org/T384252#12244277 (10Mahveotm) a:03Mahveotm [12:19:44] 10Tool-lexeme-forms, 06translatewiki.net, 07Essential-Work, 10LPL Projects (Ongoing maintenance), 10LPL sprints (S2 2026 August): l10n-bot cannot create GitLab merge requests due to gitlab-ssh migration - https://phabricator.wikimedia.org/T432838#12244291 (10LucasWerkmeister) Something seems to be br... [12:52:44] 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T435704#12244325 (10Xqt) →14Duplicate dup:03T431056 [12:52:47] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [pywikibot-buildservice] New upstream release for Pywikibot - https://phabricator.wikimedia.org/T431056#12244327 (10Xqt) [13:06:05] 10Tool-lexeme-forms, 06translatewiki.net, 07Essential-Work, 10LPL Projects (Ongoing maintenance), 10LPL sprints (S2 2026 August): l10n-bot cannot create GitLab merge requests due to gitlab-ssh migration - https://phabricator.wikimedia.org/T432838#12244328 (10Nikerabbit) At least last export skipped s... [13:20:30] 06cloud-services-team, 10Tools, 07Performance Issue: Stalktoy is often slow or not responding - https://phabricator.wikimedia.org/T427941#12244332 (10Count_Count) Shameless plug: Alternatively use https://allblocksever.toolforge.org/ to show all active and previous global and local blocks as well a global lo... [13:26:06] 06cloud-services-team, 10Tools, 07Performance Issue: Stalktoy is often slow or not responding - https://phabricator.wikimedia.org/T427941#12244333 (10Jules78120) Nice, thank you very much @Count_Count! [14:26:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:31:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:06:31] 10Toolforge, 06tools-platform-team: [lima-kilo] building locally jobs-api when a .tox env exists in the host code dir errors out - https://phabricator.wikimedia.org/T432276#12244422 (10Mahveotm) I am unable to reproduce this. I tried using Lima 2.2.0 with a reverse-SSHFS mount and `followSymlinks` disabled. I... [15:15:55] 10Cloud-VPS (Quota-requests): Quota increase request for project search - https://phabricator.wikimedia.org/T435700#12244425 (10bking) [15:16:11] 10Cloud-VPS (Quota-requests): Quota increase request for project search - https://phabricator.wikimedia.org/T435700#12244426 (10bking) [15:20:43] 10Tool-wikimonitor, 07good first task: [WikiStreamService] Fix silent SSE stream freeze on half-open TCP drops - https://phabricator.wikimedia.org/T435612#12244442 (10Gerges) 05Open→03Resolved Pull request successfully merged. [16:02:33] 10Toolforge, 06tools-platform-team: Allow querying for global edit count - https://phabricator.wikimedia.org/T435728 (10Leaderboard) 03NEW [16:02:44] 10Toolforge, 06tools-platform-team: Allow querying for global edit count - https://phabricator.wikimedia.org/T435728#12244483 (10Leaderboard) [16:04:20] 10Data-Services, 06tools-platform-team: Allow querying for global edit count - https://phabricator.wikimedia.org/T435728#12244484 (10JJMC89) [16:07:05] 10Data-Services, 06tools-platform-team: Allow querying for global edit count - https://phabricator.wikimedia.org/T435728#12244487 (10taavi) →14Duplicate dup:03T344108 [16:07:16] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Engineering, and 4 others: Add global_edit_count to wikireplicas - https://phabricator.wikimedia.org/T344108#12244489 (10taavi) [16:07:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:12:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:09:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:10:39] 10Cloud-VPS (Quota-requests): Quota increase request for project search - https://phabricator.wikimedia.org/T435700#12244553 (10bking) [17:14:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:29:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:34:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:40:14] 10Tool-wikinewsie: Excluded categories not working in Wikinewsie custom portals - https://phabricator.wikimedia.org/T435710#12244561 (10ZhaoFJx) p:05Triage→03High [17:41:29] 10Tool-wikinewsie: Better category building for AI-curated feeds for Wikinewsie - https://phabricator.wikimedia.org/T435709#12244564 (10ZhaoFJx) p:05Triage→03Medium [17:42:01] 10Tool-wikinewsie: Extended time window AI edit summaries for Wikinewsie - https://phabricator.wikimedia.org/T435659#12244567 (10ZhaoFJx) p:05Triage→03Medium [17:42:47] 10Tool-wikinewsie: GDELT feed for Wikinewsie - https://phabricator.wikimedia.org/T435650#12244570 (10ZhaoFJx) p:05Triage→03Medium [17:42:53] 10Tool-wikinewsie: Add tooltip explanations for "Mix", "Editors", "Edits", etc - https://phabricator.wikimedia.org/T435732 (10Habst) 03NEW [17:43:23] 10Tool-wikinewsie: Multimedia focus original reporting on Wikinewsie - https://phabricator.wikimedia.org/T435264#12244583 (10ZhaoFJx) p:05Triage→03Low [17:43:50] 10Tool-wikinewsie: More user-generated metadata for Wikinewsie headlines - https://phabricator.wikimedia.org/T435237#12244586 (10ZhaoFJx) p:05Triage→03Medium [17:44:37] 10Tool-wikinewsie: Wikinewsie doesn't do proper image attribution - https://phabricator.wikimedia.org/T435136#12244589 (10ZhaoFJx) p:05Triage→03High [17:44:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:45:09] 10Tool-wikinewsie: Add tooltip explanations for "Mix", "Editors", "Edits", etc - https://phabricator.wikimedia.org/T435732#12244592 (10ZhaoFJx) p:05Triage→03Medium [17:45:25] 10Tool-wikinewsie: Logging and Metrics Dashboard for Wikinewsie - https://phabricator.wikimedia.org/T435733 (10adi.darkblue) 03NEW [17:45:34] 10Tool-wikinewsie: Wikinewsie footer elements misaligned on mobile view - https://phabricator.wikimedia.org/T435018#12244607 (10ZhaoFJx) p:05Triage→03High [17:46:10] 10Tool-wikinewsie: Logging and Metrics Dashboard for Wikinewsie - https://phabricator.wikimedia.org/T435733#12244610 (10ZhaoFJx) p:05Triage→03Medium [17:46:23] 10Tool-wikinewsie: Logging and Metrics Dashboard for Wikinewsie - https://phabricator.wikimedia.org/T435733#12244614 (10ZhaoFJx) 05Open→03In progress [18:12:22] 10Tool-wikinewsie: Unclickable Username in Wikinewsie Recent Changes - https://phabricator.wikimedia.org/T435734 (10ZhaoFJx) 03NEW [18:12:52] 10Tool-wikinewsie: Wikinewsie Clickable Blue Links - https://phabricator.wikimedia.org/T432859#12244636 (10ZhaoFJx) [18:12:53] 10Tool-wikinewsie: Unclickable Username in Wikinewsie Recent Changes - https://phabricator.wikimedia.org/T435734#12244635 (10ZhaoFJx) [18:12:54] 10Tool-wikinewsie: Recent Changes page for Wikinewsie - https://phabricator.wikimedia.org/T431876#12244637 (10ZhaoFJx) [18:13:24] 10Tool-wikinewsie: Unclickable Username in Wikinewsie Recent Changes - https://phabricator.wikimedia.org/T435734#12244638 (10ZhaoFJx) p:05Triage→03Medium a:03ZhaoFJx [18:13:33] 10Tool-wikinewsie: Unclickable Username in Wikinewsie Recent Changes - https://phabricator.wikimedia.org/T435734#12244642 (10ZhaoFJx) 05Open→03In progress [18:21:05] 10Tool-wikinewsie: Auto-Deploy Wikinewsie Commits - https://phabricator.wikimedia.org/T435735 (10ZhaoFJx) 03NEW [18:21:15] 10Tool-wikinewsie: Auto-Deploy Wikinewsie Commits - https://phabricator.wikimedia.org/T435735#12244656 (10ZhaoFJx) [18:21:16] 10Tool-wikinewsie, 07Spike: Explore Possible Wikinewsie Deploy Pipeline - https://phabricator.wikimedia.org/T434616#12244657 (10ZhaoFJx) [18:21:32] 10Tool-wikinewsie: Auto-Deploy Wikinewsie Commits - https://phabricator.wikimedia.org/T435735#12244658 (10ZhaoFJx) p:05Triage→03High [18:35:41] 10Tool-wikinewsie: Add https://toolcounter.toolforge.org/ counter - https://phabricator.wikimedia.org/T435736 (10Habst) 03NEW [18:47:02] 10Tool-wikinewsie: User Survey - https://phabricator.wikimedia.org/T435737 (10adi.darkblue) 03NEW [18:56:12] 10Tool-wikinewsie: UX Design Walkthrough For AI Features - https://phabricator.wikimedia.org/T435738 (10adi.darkblue) 03NEW [18:56:15] 10Tool-wikinewsie: Unclickable Username in Wikinewsie Recent Changes - https://phabricator.wikimedia.org/T435734#12244701 (10ZhaoFJx) 05In progress→03Resolved [19:01:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:05:53] 10Tool-wikinewsie: Auto-Deploy Wikinewsie Commits - https://phabricator.wikimedia.org/T435735#12244712 (10ZhaoFJx) a:03ZhaoFJx [19:05:54] 10Tool-wikinewsie: Auto-Deploy Wikinewsie Commits - https://phabricator.wikimedia.org/T435735#12244714 (10ZhaoFJx) 05Open→03In progress [19:05:56] 10Tool-wikinewsie: Add https://toolcounter.toolforge.org/ counter - https://phabricator.wikimedia.org/T435736#12244716 (10ZhaoFJx) 05Open→03In progress [19:06:26] 10Tool-wikinewsie: Add https://toolcounter.toolforge.org/ counter - https://phabricator.wikimedia.org/T435736#12244717 (10ZhaoFJx) p:05Triage→03Medium [19:06:27] 10Tool-wikinewsie: Add https://toolcounter.toolforge.org/ counter - https://phabricator.wikimedia.org/T435736#12244719 (10Habst) 05In progress→03Resolved p:05Medium→03Triage [19:06:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:06:44] 10Tool-wikinewsie: Extended time window AI edit summaries for Wikinewsie - https://phabricator.wikimedia.org/T435659#12244723 (10Habst) a:03Habst [19:14:47] 10Cloud-VPS (Quota-requests): Quota increase request for project search - https://phabricator.wikimedia.org/T435700#12244732 (10bking) 05Open→03Invalid Sorry, my request was premature. I think it makes more sense to use normal instances and cinder volumes instead of a single ephemeral disk for Ceph. Apol... [19:23:03] 10Tool-wikinewsie: UX Design Walkthrough For AI Features - https://phabricator.wikimedia.org/T435738#12244740 (10ZhaoFJx) Would it be more closer to an investigation, or a feature design? [19:23:56] 10Tool-wikinewsie: User Survey - https://phabricator.wikimedia.org/T435737#12244742 (10ZhaoFJx) p:05Triage→03Medium [19:31:47] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-platform-team: Reach out to Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T428196#12244751 (10komla) @dschwen please create task for maps-wma2 and tag it with: **Cloud-VPS (Debian Bullseye Deprecation)*... [19:45:47] 10Tool-wikinewsie, 06translatewiki.net: Add Wikinewsie to translatewiki.net - https://phabricator.wikimedia.org/T435739 (10ZhaoFJx) 03NEW [19:45:55] 10Tool-wikinewsie, 06translatewiki.net: Add Wikinewsie to translatewiki.net - https://phabricator.wikimedia.org/T435739#12244766 (10ZhaoFJx) [19:46:03] 10Tool-wikinewsie, 07good first task: Wikinewsie i18n - https://phabricator.wikimedia.org/T432938#12244767 (10ZhaoFJx) [20:52:26] 10Tool-wikinewsie: Auto-Deploy Wikinewsie Commits - https://phabricator.wikimedia.org/T435735#12244797 (10ZhaoFJx) 05In progress→03Stalled It's quite dangerous to use the admins' credentials on GitLab directly... Perhaps there would be a better way? Stalled for now since CI is not configured yet [20:56:53] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: tofu-infra: add cinder volume types - https://phabricator.wikimedia.org/T410148#12244801 (10Mahveotm) [21:23:41] (03PS1) 10Krinkle: build: Add type hints, simplify jsonpCallback logic, add `composer serve` [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328374 [21:25:28] (03PS2) 10Krinkle: build: Add type hints, simplify jsonpCallback logic, add `composer serve` [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328374 [21:25:28] (03PS1) 10Krinkle: api.php: Implement a new `recent=1` feature to return last 30 rows [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328375 [21:36:27] 10Cloud-Services, 06cloud-services-team, 10Castor, 06Collaboration-Services, and 3 others: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12244837 (10Les4353) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://ph... [21:37:14] 06cloud-services-team, 10Castor, 06Collaboration-Services, 10Continuous-Integration-Infrastructure, and 2 others: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12244839 (10Les4353) [21:38:25] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Nairobi 29): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12244841 (10Les4353) p:05Triage→03High Pe... [22:10:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:15:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:25:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:30:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:36:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:41:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:42:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:46:56] (03open) 10bd808: Draft: Poorly factored application of project layout, lint, and CI [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/3 [22:47:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:19:55] 10Tool-wikinewsie: Extended time window AI edit summaries for Wikinewsie - https://phabricator.wikimedia.org/T435659#12244972 (10Habst) 05Open→03Resolved [23:20:36] 10Tool-wikinewsie: Wikinewsie doesn't do proper image attribution - https://phabricator.wikimedia.org/T435136#12244974 (10Habst) a:03Habst [23:26:35] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 10Phabricator, and 2 others: Look for ways to consolidate "we trust this human" access lists - https://phabricator.wikimedia.org/T364516#12244994 (10Nirmos) [23:48:49] 10Tool-wikinewsie: Wikinewsie doesn't do proper image attribution - https://phabricator.wikimedia.org/T435136#12245010 (10Habst) 05Open→03Resolved