[00:07:04] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12254809 (10bd808) `lang=shell-session bd808@de... [00:08:21] 10Tool-wikinewsie: Dynamically adapt to the OS theme preference (prefers-color-scheme) - https://phabricator.wikimedia.org/T436041#12254811 (10waldyrious) 05Open→03Resolved [02:17:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:04:04] 10Tool-Pageviews, 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-Engineering-Wikistats, and 2 others: Pageviews Analysis 3.0 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#12254926 (10MusikAnimal) [03:06:27] 10Tool-Pageviews: Move Pageviews tool to Symfony - https://phabricator.wikimedia.org/T420834#12254929 (10MusikAnimal) 05In progress→03Resolved Done, beta version is now up at: * https://pageviews-test.wmcloud.org * https://pageviews-test.toolforge.org See T378549 for more details and follow-ups [03:07:00] 10Tool-Pageviews, 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-Engineering-Wikistats, and 2 others: Pageviews Analysis 3.0 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#12254936 (10MusikAnimal) WIP: * https://pageviews-test.wmcloud.org * https://pageviews-test.toolforge.org We'll like... [03:09:09] 10Tool-Pageviews: Validate projects on entry in Pageviews instead of bundling the allowlist - https://phabricator.wikimedia.org/T371997#12254938 (10MusikAnimal) 05In progress→03Resolved a:03MusikAnimal I believe this was resolved some time ago. If it hasn't been, it will arrive soon with T378549. [03:29:10] 10Tool-Pageviews: Add import method to request massviews stats from multiple wiki's at once - https://phabricator.wikimedia.org/T204755#12254954 (10MusikAnimal) 05Open→03In progress a:03MusikAnimal [03:32:55] 10Tool-Pageviews: "Category" Massviews error: An unknown error occurred when querying - https://phabricator.wikimedia.org/T419975#12254959 (10MusikAnimal) 05Open→03Declined I'm assuming this was an intermittent issue. It seems to be working fine now. Please re-open if you're still experiencing this. [03:36:48] 10Tool-Pageviews: [Topviews] Allow permalink to search results - https://phabricator.wikimedia.org/T193892#12254963 (10MusikAnimal) 05Open→03In progress a:03MusikAnimal [03:37:11] 10Tool-Pageviews: [Topviews] Unable to view 901-1000 records - https://phabricator.wikimedia.org/T191954#12254966 (10MusikAnimal) 05Open→03Declined [03:38:06] 10Tool-Pageviews, 07I18n, 07Plural-Support: Pageviews Analysis i18n messages in English should use PLURAL and GENDER where those can be used - https://phabricator.wikimedia.org/T185634#12254968 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal [04:01:43] 10Tool-Pageviews: Add namespace filter to Massviews - https://phabricator.wikimedia.org/T166176#12254976 (10MusikAnimal) 05Open→03In progress a:03MusikAnimal [04:03:03] 10Tool-Pageviews: Expand Massviews options to include mainspace and show other source-specific data - https://phabricator.wikimedia.org/T159868#12254980 (10MusikAnimal) →14Duplicate dup:03T166176 [04:03:04] 10Tool-Pageviews: Add namespace filter to Massviews - https://phabricator.wikimedia.org/T166176#12254982 (10MusikAnimal) [04:05:14] 10Tool-Pageviews: Require login for mass-querying tools like Massviews - https://phabricator.wikimedia.org/T420295#12254985 (10MusikAnimal) 05Open→03Declined In the new version (T378549), I'm not sure this is needed, though I do still plan to add OAuth and potentially a job queue system to get around the... [04:05:36] 10Tool-Pageviews: Move Pageviews tool to Symfony - https://phabricator.wikimedia.org/T420834#12254991 (10MusikAnimal) [04:05:37] 10Tool-Pageviews: Require login for mass-querying tools like Massviews - https://phabricator.wikimedia.org/T420295#12254990 (10MusikAnimal) [04:08:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:18:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:23:34] 10Tool-Pageviews, 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-Engineering-Wikistats, and 2 others: Pageviews Analysis 2026 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#12255014 (10MusikAnimal) [04:26:32] 10Tool-Pageviews: Implement a backend API - https://phabricator.wikimedia.org/T157830#12255016 (10MusikAnimal) [04:29:06] 10Tool-Pageviews: Show smaller pie chart in addition to line/bar/radar chart - https://phabricator.wikimedia.org/T149374#12255020 (10MusikAnimal) 05Open→03Declined No one asked for this. [04:29:28] 10Tool-Pageviews: Add option show moving average line over chart - https://phabricator.wikimedia.org/T147515#12255025 (10MusikAnimal) 05Open→03In progress a:03MusikAnimal [04:33:06] 10Tool-Pageviews: Add option show moving average line over chart - https://phabricator.wikimedia.org/T147515#12255030 (10MusikAnimal) You can try this out now in the beta 2026 version, i.e. https://pageviews-test.wmcloud.org/?start=2023-01&end=2026-07&project=en.wikipedia.org&platform=all-access&agent=user&pages... [04:34:36] (03open) 10samwilson: Add `br` Transkribus model [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/16 (https://phabricator.wikimedia.org/T435905) [04:39:50] (03merge) 10samwilson: Add `br` Transkribus model [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/16 (https://phabricator.wikimedia.org/T435905) [04:52:07] 10Tool-Pageviews: Add logscale option - https://phabricator.wikimedia.org/T177369#12255040 (10MusikAnimal) 05Open→03In progress a:03MusikAnimal [04:53:02] 10Tool-Pageviews: Add logarithmic URL query parameter - https://phabricator.wikimedia.org/T177369#12255043 (10MusikAnimal) [04:58:11] 10Tool-Pageviews: Make permalink include chart properties - https://phabricator.wikimedia.org/T204620#12255047 (10MusikAnimal) 05Open→03In progress a:03MusikAnimal [05:06:08] 10Tool-Pageviews: Add URL params for all chart options - https://phabricator.wikimedia.org/T169343#12255052 (10MusikAnimal) 05Open→03In progress a:03MusikAnimal [05:06:48] 10Tool-Pageviews: Add URL params for all chart options - https://phabricator.wikimedia.org/T169343#12255056 (10MusikAnimal) →14Duplicate dup:03T204620 [05:06:49] 10Tool-Pageviews: Make permalink include chart properties - https://phabricator.wikimedia.org/T204620#12255058 (10MusikAnimal) [05:13:09] 10Tool-Pageviews: Ability to process pages that are an instance of a Wikidata item - https://phabricator.wikimedia.org/T164113#12255064 (10MusikAnimal) T164181 sounds like the better solution. I'll look into that instead. [05:13:12] 10Tool-Pageviews: Ability to process pages that are an instance of a Wikidata item - https://phabricator.wikimedia.org/T164113#12255067 (10MusikAnimal) →14Duplicate dup:03T164181 [05:13:13] 10Tool-Pageviews: Add SPARQL as input for Massviews - https://phabricator.wikimedia.org/T164181#12255069 (10MusikAnimal) [05:15:07] 10Tool-Pageviews: Add "weekly" date type to Pageviews apps - https://phabricator.wikimedia.org/T158901#12255075 (10MusikAnimal) 05Open→03Declined [05:37:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:57:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:37:03] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance paws-puppetserver-1 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [06:58:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:08:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:13:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:24:40] 10Cloud-VPS, 06tools-infrastructure-team: Change mount propagation for /mnt/nfs in wikiqlever - https://phabricator.wikimedia.org/T433671#12255298 (10fgiunchedi) Hello all, just wanted to get a notification in re: this work; if dumps are required inside containers then the mounts need to be changed as describe... [07:24:51] 10Cloud-VPS, 06tools-infrastructure-team, 10VPS-Project-wikicommunityhealth: Change mount propagation for /mnt/nfs in backend.wikicommunityhealth - https://phabricator.wikimedia.org/T433667#12255299 (10fgiunchedi) Hello all, just wanted to get a notification in re: this work; if dumps are required inside con... [07:26:35] 10Cloud-VPS, 06tools-infrastructure-team: Migrate to dumps-nfs.w.o in Cloud VPS - https://phabricator.wikimedia.org/T432587#12255300 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This is done, there's two subtasks I'll leave open re: changing mount options for self-managed containers in wikiqlever... [07:28:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:11:07] FIRING: ToolsDBReplicationLagIsTooHigh: ToolsDB replication is lagging - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBReplicationLagIsTooHigh [08:41:38] (03approved) 10tlepage: refactor: made loki a package, and minor refactors [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/35 (https://phabricator.wikimedia.org/T432570) (owner: 10dcaro) [08:41:40] (03update) 10tlepage: refactor: made loki a package, and minor refactors [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/35 (https://phabricator.wikimedia.org/T432570) (owner: 10dcaro) [08:42:41] (03merge) 10tlepage: refactor: made loki a package, and minor refactors [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/35 (https://phabricator.wikimedia.org/T432570) (owner: 10dcaro) [08:42:42] (03update) 10tlepage: write_job_logs: add jobs events [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/34 (https://phabricator.wikimedia.org/T432570) (owner: 10dcaro) [08:45:31] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.44-20260826084251-9b77716c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1390 (https://phabricator.wikimedia.org/T432570) [08:45:35] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.44-20260826084251-9b77716c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1390 (https://phabricator.wikimedia.org/T432570) [08:55:26] 10Tool-inteGraality: Persistence backend for integraality - https://phabricator.wikimedia.org/T436062 (10JeanFred) 03NEW [08:56:20] 10Tool-inteGraality: Dashboard list/status page - https://phabricator.wikimedia.org/T435900#12255590 (10JeanFred) [08:56:21] 10Tool-inteGraality: Persistence backend for integraality - https://phabricator.wikimedia.org/T436062#12255591 (10JeanFred) [08:58:19] 10Tool-inteGraality: Persistence backend for integraality - https://phabricator.wikimedia.org/T436062#12255593 (10JeanFred) p:05Triage→03Medium [08:59:29] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [09:00:44] !log tlepage@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api [09:05:08] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [09:11:08] 10Tool-Pageviews: Show page assessment data in Userviews - https://phabricator.wikimedia.org/T157765#12255659 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal Will be done as part of T378549 [09:14:13] !log tlepage@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [09:17:04] !log tlepage@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [09:25:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:27:14] !log tlepage@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [09:28:15] (03update) 10tlepage: logs-api: bump to 0.0.44-20260826084251-9b77716c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1390 (https://phabricator.wikimedia.org/T432570) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:28:22] (03merge) 10tlepage: logs-api: bump to 0.0.44-20260826084251-9b77716c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1390 (https://phabricator.wikimedia.org/T432570) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:30:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:36:44] (03approved) 10tlepage: write_job_logs: add jobs events [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/34 (https://phabricator.wikimedia.org/T432570) (owner: 10dcaro) [09:36:47] (03update) 10tlepage: write_job_logs: add jobs events [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/34 (https://phabricator.wikimedia.org/T432570) (owner: 10dcaro) [09:37:03] (03merge) 10tlepage: write_job_logs: add jobs events [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/34 (https://phabricator.wikimedia.org/T432570) (owner: 10dcaro) [09:39:16] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.45-20260826093712-2d6a3f95 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1391 (https://phabricator.wikimedia.org/T432570) [09:39:19] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [09:45:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:49:13] !log tlepage@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [09:50:33] !log tlepage@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [10:00:23] !log tlepage@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [10:02:44] (03merge) 10tlepage: logs-api: bump to 0.0.45-20260826093712-2d6a3f95 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1391 (https://phabricator.wikimedia.org/T432570) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:18:03] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance proxy-6 in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [10:56:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:01:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:04:37] 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Arusha 30): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12256117 (10AMarkossyan-WMF) [11:38:59] 10Toolforge, 06tools-infrastructure-team: [disable-tools] Disabled Toolforge tool deletions not occurring - https://phabricator.wikimedia.org/T431241#12256270 (10taavi) 05Open→03Resolved a:03taavi As far as I can tell all currently disabled tools are less than 40 days old so everything is at the mome... [11:48:03] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance proxy-6 in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [12:21:03] 10Toolforge, 06tools-platform-team, 07OKR-Work: Email tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865#12256451 (10fnegri) > In Toolforge, the Prometheus API is not highly available nor clustered, unlike the metricsinfra Alertmanager service. @taavi fair point... [13:09:03] FIRING: TargetDown: Job redis is unreachable in project project-proxy instance proxy-6 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:09:48] FIRING: PuppetDisabled: Puppet disabled on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [13:14:03] FIRING: [2x] TargetDown: Job redis is unreachable in project project-proxy instance proxy-5 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:19:03] RESOLVED: [2x] TargetDown: Job redis is unreachable in project project-proxy instance proxy-5 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:20:30] 10Tool-wikinewsie: Deaths story - https://phabricator.wikimedia.org/T435844#12256693 (10Mollystarkdean) Yes. I talked about obituaries being an important part of any newspaper, including ours. But bundling Wikipedia lists on Wikinewsie seems antithetical to the idea of this automatically ingesting open knowledge... [13:28:16] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12256736 (10bking) Sorry for the delay, I'm starting on this now. [13:31:28] 10Toolforge, 06tools-platform-team, 07OKR-Work: Add scaffolding for the Toolforge Alerting Service - https://phabricator.wikimedia.org/T435971#12256744 (10fnegri) 05Open→03In progress [13:32:24] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12256747 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2003 for hos... [13:41:35] 06cloud-services-team, 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] optionally log deployments to SAL automatically - https://phabricator.wikimedia.org/T393169#12256803 (10aputhin) 05Open→03In progress [13:41:38] 06cloud-services-team, 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] Add source polling build trigger - https://phabricator.wikimedia.org/T362071#12256807 (10aputhin) 05Open→03In progress [13:43:38] 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: Upgrade toolsbeta to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433131#12256824 (10Andrew) 05Open→03Resolved [13:45:02] 06cloud-services-team, 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] optionally log deployments to SAL automatically - https://phabricator.wikimedia.org/T393169#12256831 (10aputhin) 05In progress→03Open [13:46:19] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 07Epic, 06Release-Engineering-Team (Priority Backlog 📥): Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12256837 (10taavi) [13:47:32] 06cloud-services-team, 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] optionally log deployments to SAL automatically - https://phabricator.wikimedia.org/T393169#12256855 (10taavi) This feels like something to implement more generally with the tool audit/activity logging work that's ha... [13:49:01] 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.33 - https://phabricator.wikimedia.org/T408785#12256859 (10Andrew) [13:49:05] 06cloud-services-team, 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] Add source polling build trigger - https://phabricator.wikimedia.org/T362071#12256860 (10aputhin) 05In progress→03Open [13:58:04] (03open) 10samtar: Draft: Add WikipediaLibrary/ to allowlist [toolforge-repos/wmde-github-phab-bot] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-github-phab-bot/-/merge_requests/5 [14:08:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:09:48] RESOLVED: PuppetDisabled: Puppet disabled on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [14:12:08] (03update) 10samtar: Add WikipediaLibrary/ to allowlist [toolforge-repos/wmde-github-phab-bot] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-github-phab-bot/-/merge_requests/5 [14:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:36:38] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12257223 (10bking) I tried reimaging as Trixie, but Puppet is failing because it's still trying to install... [14:36:43] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12257227 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2003 for host cl... [14:37:17] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12257247 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2003 for hos... [14:48:45] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.33.13 (T433132) [14:48:51] T433132: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132 [14:49:06] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster tools upgrade to 1.33.13 (T433132) [14:54:38] (03Abandoned) 10Wandji collins: Remove unwanted debug print statement micorservice [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132120 (https://phabricator.wikimedia.org/T386328) (owner: 10Juniorbesong) [14:56:59] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.33.13 (T433132) [14:57:12] T433132: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132 [14:59:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [15:01:12] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Surprise harbor failure in wmcs.toolforge.k8s.prepare_upgrade - https://phabricator.wikimedia.org/T436126 (10Andrew) 03NEW [15:01:23] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Surprise harbor failure in wmcs.toolforge.k8s.prepare_upgrade - https://phabricator.wikimedia.org/T436126#12257508 (10Andrew) [15:01:26] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132#12257509 (10Andrew) [15:04:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [15:07:03] (03PS1) 10Wandji collins: Matching service to wikidata lexems base on search. Language and commons logic for retreiving files [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1329587 [15:07:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:08:33] (03CR) 10Wandji collins: [V:03+2 C:03+2] "This change has been tested." [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1329587 (owner: 10Wandji collins) [15:12:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:14:31] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/15 [15:14:44] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/27 [15:14:58] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/22 [15:15:58] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/54 [15:16:54] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/39 [15:17:04] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.33.13 (T433132) [15:17:09] T433132: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132 [15:19:03] (03PS1) 10Wandji collins: Create function to convert language codes from ISO 639-3 to ISO 639-1 Bug: T388187 [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1329589 (https://phabricator.wikimedia.org/T388187) [15:19:12] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 (T433132) [15:19:30] (03PS2) 10Wandji collins: Create function to convert language codes from ISO 639-3 to ISO 639-1 [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1329589 (https://phabricator.wikimedia.org/T388187) [15:19:50] (03CR) 10Wandji collins: "recheck" [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1329589 (https://phabricator.wikimedia.org/T388187) (owner: 10Wandji collins) [15:20:21] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 (T433132) [15:21:07] (03CR) 10Wandji collins: [V:03+2 C:03+2] Create function to convert language codes from ISO 639-3 to ISO 639-1 [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1329589 (https://phabricator.wikimedia.org/T388187) (owner: 10Wandji collins) [15:28:04] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 (T433132) [15:28:09] T433132: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132 [15:29:57] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132#12257634 (10Andrew) upgrade plan for control-7: ` [upgrade] Running cluster health checks... [15:30:40] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 (T433132) [15:31:42] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 (T433132) [15:38:09] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 (T433132) [15:38:15] T433132: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132 [15:38:35] 10Toolforge, 06tools-platform-team, 07OKR-Work: Add scaffolding for the Toolforge Alerting Service - https://phabricator.wikimedia.org/T435971#12257698 (10fnegri) > I propose to call it `alerts-api` or `alerting-service`. We discussed this in today's refinement meeting and there was some consensus around `a... [15:39:13] 10Tool-Pageviews, 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-Engineering-Wikistats, and 2 others: Pageviews Analysis 2026 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#12257700 (10SunloungerFrog) A little testing I tried out both WIP versions using massviews against Quarry query 909... [15:42:09] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.restart_static_pods for tools-k8s-control-7 (T433132) [15:43:49] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132#12257733 (10Andrew) After upgrading tools-k8s-control-7, the api server yaml file was out of place so the server couldn't restart. Taavi replaced the file and I r... [15:48:45] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=0) for tools-k8s-control-7 (T433132) [15:48:50] T433132: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132 [15:50:23] 10Toolforge, 06tools-platform-team, 07OKR-Work: [jobs-api] Send webservice start/stop/restart events to the new logs-api event log endpoint - https://phabricator.wikimedia.org/T432570#12257762 (10TLepage-WMF) 05Open→03In progress a:05dcaro→03TLepage-WMF [15:51:25] (03PS1) 10Majavah: k8s: Do not manually restart kube-apiserver [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1329597 (https://phabricator.wikimedia.org/T433132) [15:52:53] 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12257794 (10TLepage-WMF) This is "frozen" as we wait for the logs using the previous label schema to rotate out of Loki’s retention window (14 days). I hav... [15:53:19] (03CR) 10Andrew Bogott: [C:03+1] k8s: Do not manually restart kube-apiserver [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1329597 (https://phabricator.wikimedia.org/T433132) (owner: 10Majavah) [15:54:22] (03CR) 10Majavah: [C:03+2] k8s: Do not manually restart kube-apiserver [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1329597 (https://phabricator.wikimedia.org/T433132) (owner: 10Majavah) [15:55:53] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12257808 (10bd808) >>! In T435393#12254809, @bd... [15:57:36] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: cleanup mcrouter from cloudcontrol hosts - https://phabricator.wikimedia.org/T427189#12257833 (10taavi) 05Open→03Resolved [15:57:54] (03Merged) 10jenkins-bot: k8s: Do not manually restart kube-apiserver [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1329597 (https://phabricator.wikimedia.org/T433132) (owner: 10Majavah) [15:59:32] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132#12257838 (10Andrew) [16:18:29] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/15 [16:18:48] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/27 [16:18:57] FIRING: ProbeDown: Service dumps-https:443 has failed probes (http_dumps-https_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#dumps-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [16:19:07] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/22 [16:19:46] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.32.13 to 1.33.13 (T433132) [16:19:51] T433132: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132 [16:20:08] 10Tool-curator: Integrate Flickr uploads - https://phabricator.wikimedia.org/T414200#12257897 (10Superb_Owl) It looks like this feature is broken for the reasons described above. The error message I keep getting is "There was a problem during the HTTP request: 429 Too Many Requests" [16:20:11] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/54 [16:20:48] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/39 [16:23:57] RESOLVED: ProbeDown: Service dumps-https:443 has failed probes (http_dumps-https_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#dumps-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [16:24:32] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.32.13 to 1.33.13 (T433132) [16:24:59] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.32.13 to 1.33.13 (T433132) [16:25:04] T433132: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132 [16:25:45] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132#12257940 (10Andrew) [16:27:10] 10Tool-wheres-my-code-running, 06Release-Engineering-Team: Handle reverts - https://phabricator.wikimedia.org/T436135 (10hashar) 03NEW [16:30:56] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.32.13 to 1.33.13 (T433132) [16:31:01] T433132: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132 [16:33:04] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 (T433132) [16:43:55] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-1 from 1.32.13 to 1.33.13 (T433132) [16:44:00] T433132: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132 [16:44:50] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-1 from 1.32.13 to 1.33.13 (T433132) [16:46:14] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 (T433132) [16:46:46] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- [16:46:46] k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k [16:46:46] 8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8 [16:46:46] s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worker-nfs-9 (T433132) [16:46:56] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- [16:46:56] k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k [16:46:56] 8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8 [16:46:56] s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worker-nfs-9 (T433132) [16:47:05] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- [16:47:05] k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k [16:47:05] 8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8 [16:47:06] s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worker-nfs-9 (T433132) [16:48:40] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-2 from 1.32.13 to 1.33.13 (T433132) [16:49:39] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-2 from 1.32.13 to 1.33.13 (T433132) [16:49:44] T433132: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132 [16:54:24] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-3 from 1.32.13 to 1.33.13 (T433132) [16:55:17] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-3 from 1.32.13 to 1.33.13 (T433132) [16:55:22] T433132: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132 [16:59:17] FIRING: JobUnavailable: Reduced availability for job pdns in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:59:25] FIRING: [6x] SystemdUnitFailed: hdfs_rsync_clickstream.service on clouddumps1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:59:34] FIRING: DiskSpace: Disk space clouddumps1002:9100:/ 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [17:00:09] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud (T433132) [17:00:31] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud (T433132) [17:00:36] T433132: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132 [17:00:48] FIRING: PuppetFailure: Puppet has failed on clouddumps1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [17:04:17] RESOLVED: JobUnavailable: Reduced availability for job pdns in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:04:34] RESOLVED: DiskSpace: Disk space clouddumps1002:9100:/ 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [17:07:14] FIRING: [2x] ProbeDown: Service clouddumps1002:443 has failed probes (http_dumps_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Dumps/SQL-XML_Dumps#NFS_share_and/or_web_server_issues - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:11:15] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo [17:11:15] rker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-wor [17:11:15] ker-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-work [17:11:15] er-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worker [17:11:15] -nfs-9 (T433132) [17:11:18] T433132: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132 [17:12:04] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-w [17:12:04] orker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-wo [17:12:04] rker-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-wor [17:12:04] ker-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worke [17:12:04] r-nfs-9 (T433132) [17:13:05] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo [17:13:05] rker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-wor [17:13:05] ker-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-work [17:13:05] er-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worker [17:13:05] -nfs-9 (T433132) [17:13:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:14:25] FIRING: [6x] SystemdUnitFailed: hdfs_rsync_clickstream.service on clouddumps1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:15:36] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- [17:15:36] k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k [17:15:36] 8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8 [17:15:36] s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worker-nfs-9 (T433132) [17:17:04] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo [17:17:04] rker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-wor [17:17:04] ker-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-work [17:17:04] er-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worker [17:17:04] -nfs-9 (T433132) [17:17:07] T433132: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132 [17:17:14] RESOLVED: [2x] ProbeDown: Service clouddumps1002:443 has failed probes (http_dumps_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Dumps/SQL-XML_Dumps#NFS_share_and/or_web_server_issues - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:17:14] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- [17:17:14] k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k [17:17:14] 8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8 [17:17:14] s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worker-nfs-9 (T433132) [17:17:46] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo [17:17:46] rker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-wor [17:17:46] ker-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-work [17:17:46] er-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worker [17:17:46] -nfs-9 (T433132) [17:17:55] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- [17:17:55] k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k [17:17:55] 8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8 [17:17:55] s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worker-nfs-9 (T433132) [17:18:27] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo [17:18:27] rker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-wor [17:18:27] ker-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-work [17:18:27] er-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worker [17:18:27] -nfs-9 (T433132) [17:18:38] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132#12258315 (10Andrew) [17:22:21] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12258357 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2003 for host cl... [17:23:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:24:25] FIRING: [6x] SystemdUnitFailed: hdfs_rsync_clickstream.service on clouddumps1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:25:05] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132#12258385 (10Andrew) [17:25:16] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Upgrade tools to Kubernetes version 1.33 - https://phabricator.wikimedia.org/T433132#12258386 (10Andrew) 05Open→03Resolved [17:25:45] 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.33 - https://phabricator.wikimedia.org/T408785#12258389 (10Andrew) 05Open→03Resolved a:03Andrew [17:25:54] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595#12258392 (10Andrew) [17:29:25] FIRING: [6x] SystemdUnitFailed: hdfs_rsync_clickstream.service on clouddumps1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:35:48] RESOLVED: PuppetFailure: Puppet has failed on clouddumps1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [17:38:19] 10Wikibugs, 13Patch-For-Review: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#12258446 (10bd808) test [17:39:25] RESOLVED: SystemdUnitFailed: hdfs_rsync_clickstream.service on clouddumps1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:42:14] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12258489 (10bking) I successfully reimaged the host, but I needed some help from @taavi as the server came... [17:51:20] 10Tool-wheres-my-code-running, 06Release-Engineering-Team: Handle reverts - https://phabricator.wikimedia.org/T436135#12258520 (10bd808) The interesting thing about a revert is that the commit is deployed, but the revert commit is also which then nullifies the original patch. > The reverts were NOT done in Ge... [17:52:27] 10Tool-wheres-my-code-running, 06Release-Engineering-Team: Detect when a commit has been reverted by a later commit - https://phabricator.wikimedia.org/T436135#12258522 (10bd808) [17:53:14] 10Tool-wheres-my-code-running: Setup local and CI linters and tests - https://phabricator.wikimedia.org/T435665#12258523 (10bd808) This is {{done}} except for README.md updates that I forgot to make. [17:53:52] 10Tool-Pageviews, 07I18n: massviews-category-description lego for "category" - https://phabricator.wikimedia.org/T146973#12258524 (10MusikAnimal) a:03MusikAnimal [18:00:07] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12258534 (10bking) There were a few systemd units which I've restarted: `hdfs_rsync_pageview.service` and `... [18:13:06] 10Tool-wikinewsie, 10LPL sprints, 06translatewiki.net, 10LPL Projects (Ongoing maintenance): Add Wikinewsie to translatewiki.net - https://phabricator.wikimedia.org/T435739#12258582 (10ZhaoFJx) >>! In T435739#12246161, @Nikerabbit wrote: > @ZhaoFJx Please give access to push to main to https://gitlab.wikim... [18:14:17] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12258588 (10bking) Update: The above services completed successfully. [18:14:31] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/15 [18:14:46] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/27 [18:14:47] 10Tool-wikinewsie: Placeholder Photos? - https://phabricator.wikimedia.org/T435845#12258589 (10ZhaoFJx) p:05Triage→03Medium [18:15:00] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/22 [18:15:25] 10Tool-wikinewsie: Deaths story - https://phabricator.wikimedia.org/T435844#12258593 (10ZhaoFJx) [18:16:02] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/54 [18:16:12] 10Tool-wikinewsie: Adjustable image alignment on Wikinewsie - https://phabricator.wikimedia.org/T434467#12258597 (10ZhaoFJx) 05Open→03In progress [18:17:01] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/39 [18:17:52] (03update) 10zhaofjx: Fix over-cropping of images [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/8 (https://phabricator.wikimedia.org/T434467) (owner: 10waldir) [18:27:18] 10Toolforge, 06tools-platform-team, 07OKR-Work: Provide a way for users to enable and disable alerts - https://phabricator.wikimedia.org/T432864#12258627 (10komla) [18:35:49] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Doing 😎): Planning for a plan to migrate Beta Cluster again - https://phabricator.wikimedia.org/T433998#12258671 (10ssingh) Traffic will work on upgrading `deployment-cache-(text|upload)08` if that's fine an... [18:43:43] (03open) 10dancy: Include the MediaWiki branch in the results for each place [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/7 (https://phabricator.wikimedia.org/T434726) [18:46:33] (03update) 10bd808: Update README for refactored code [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/8 (https://phabricator.wikimedia.org/T435665) [18:46:36] (03open) 10bd808: Update README for refactored code [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/8 (https://phabricator.wikimedia.org/T435665) [18:57:58] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate discourse away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T436152 (10komla) 03NEW [19:09:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:09:41] (03update) 10dancy: Include the MediaWiki branch in the results for each place [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/7 (https://phabricator.wikimedia.org/T434726) [19:12:25] (03merge) 10dancy: Include the MediaWiki branch in the results for each place [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/7 (https://phabricator.wikimedia.org/T434726) [19:14:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:19:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:21:35] 10Tool-wheres-my-code-running, 13Patch-For-Review: Setup local and CI linters and tests - https://phabricator.wikimedia.org/T435665#12258873 (10bd808) 05Open→03In progress a:03bd808 [19:24:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:14:14] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/15 [20:14:24] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/27 [20:14:32] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/22 [20:15:26] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/54 [20:16:05] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/39 [21:14:19] (03update) 10pacmand: Add optional image pre-processing to suppress show-through [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/14 (https://phabricator.wikimedia.org/T435428) [22:31:08] (03PS1) 10Cwhite: logging: remove ecdsa pki keys [labs/private] - 10https://gerrit.wikimedia.org/r/1329682 (https://phabricator.wikimedia.org/T350516) [22:33:29] (03CR) 10Cwhite: [V:03+2 C:03+2] logging: remove ecdsa pki keys [labs/private] - 10https://gerrit.wikimedia.org/r/1329682 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [22:45:03] (03PS1) 10Cwhite: logging: add new betalogs pki keys [labs/private] - 10https://gerrit.wikimedia.org/r/1329685 (https://phabricator.wikimedia.org/T350516) [22:45:55] (03CR) 10Cwhite: [V:03+2 C:03+2] logging: add new betalogs pki keys [labs/private] - 10https://gerrit.wikimedia.org/r/1329685 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite)