[00:16:22] <wikibugs>	 10Tool-wikinewsie: Add article summaries to clickthrough modals for the articles  below the top article - https://phabricator.wikimedia.org/T436198 (10adi.darkblue) 03NEW
[00:48:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[00:53:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[01:26:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[01:41:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[01:46:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[01:47:23] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate discourse away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T436152#12260084 (10Samwilson) There are two instances:  * discuss-space.discourse.eqiad1.wikimedia.cloud * discuss-space-new.discourse.eqiad1.wikimedia.cloud  The first h...
[04:20:28] <wikibugs>	 10Tool-Pageviews, 07I18n: massviews-category-description lego for "category" - https://phabricator.wikimedia.org/T146973#12260259 (10MusikAnimal) 05Open→03In progress
[04:40:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[04:45:38] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[05:31:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[05:36:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[05:46:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[06:46:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[06:51:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[06:54:46] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate discourse away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T436152#12260404 (10Aklapper) See also {T310031}, {T306070}
[06:56:38] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[06:58:44] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[07:03:44] <jinxer-wm>	 RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[07:56:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[08:01:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[08:07:42] <wikibugs>	 10Data-Services, 06tools-infrastructure-team, 13Patch-For-Review: Clean up clouddumps symlink handling from puppet - https://phabricator.wikimedia.org/T435581#12260572 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This is done
[08:13:07] <wikibugs>	 10Cloud-VPS, 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team, and 4 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12260584 (10fgiunchedi)
[08:33:03] <wmcs-alerts>	 FIRING: PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-puppetserver-1 in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[08:33:03] <wmcs-alerts>	 FIRING: PuppetAgentFailure: Puppet agent failure detected on instance cloudinfra-cloudvps-puppetserver-1 in project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[08:38:03] <wmcs-alerts>	 RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance cloudinfra-cloudvps-puppetserver-1 in project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[08:47:54] <wikibugs>	 (03PS2) 10Raymond Ndibe: webservice-runner: default to 8000 only if PORT and TOOL_WEB_PORT has no value [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1310212 (https://phabricator.wikimedia.org/T432078)
[08:48:03] <wmcs-alerts>	 RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-puppetserver-1 in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[08:48:58] <wikibugs>	 (03CR) 10Raymond Ndibe: "resuming this because we've gotten to the place were we need to merge this" [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1310212 (https://phabricator.wikimedia.org/T432078) (owner: 10Raymond Ndibe)
[08:53:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[09:03:09] <wikibugs>	 10Tool-wikinewsie, 07good first task: Wikinewsie i18n - https://phabricator.wikimedia.org/T432938#12260719 (10Chimaifeanyi29) I am working on this. please assign it to me.
[09:03:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[09:06:51] <wikibugs>	 10Tool-techcontribs: techcontribs needs updating for the new OpenSearch cluster - https://phabricator.wikimedia.org/T436218 (10taavi) 03NEW
[09:14:36] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/15
[09:14:51] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/27
[09:15:08] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/22
[09:16:09] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/54
[09:17:08] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/39
[09:22:18] <wikibugs>	 10Tool-techcontribs: techcontribs needs updating for the new OpenSearch cluster - https://phabricator.wikimedia.org/T436218#12260769 (10Chlod) 05Open→03Resolved p:05Triage→03High a:03Chlod Whoops, shouldn't be reading emails half-asleep. I did not realize this affects Tech Contribs too. Luckily, th...
[09:39:17] <wikibugs>	 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T435704#12260820 (10fnegri) 05Duplicate→03Open This is the separate update process for PAWS, while T435704 is for Toolforge.
[10:03:16] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
[10:03:24] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
[10:13:01] <wikibugs>	 (03approved) 10fnegri: Bring component helmfiles into the modern age. [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1370 (owner: 10andrew)
[10:13:07] <wikibugs>	 (03update) 10fnegri: Bring component helmfiles into the modern age. [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1370 (owner: 10andrew)
[10:13:17] <wikibugs>	 (03update) 10raymond-ndibe: models.py: expose state and webservice_defaults in images endpoint [repos/cloud/toolforge/jobs-api] (add_webservice_endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/347 (https://phabricator.wikimedia.org/T431679)
[10:13:27] <wikibugs>	 (03update) 10fnegri: Bring component helmfiles into the modern age. [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1370 (owner: 10andrew)
[10:14:31] <wikibugs>	 (03update) 10raymond-ndibe: models.py: expose state and webservice_defaults in images endpoint [repos/cloud/toolforge/jobs-api] (add_webservice_endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/347 (https://phabricator.wikimedia.org/T431679)
[10:14:53] <wikibugs>	 (03merge) 10fnegri: Bring component helmfiles into the modern age. [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1370 (owner: 10andrew)
[10:15:00] <wikibugs>	 (03update) 10raymond-ndibe: models.py: expose state and webservice_defaults in images endpoint [repos/cloud/toolforge/jobs-api] (add_webservice_endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/347 (https://phabricator.wikimedia.org/T431679)
[10:15:22] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
[10:15:30] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
[10:17:20] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
[10:17:27] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
[10:19:08] <wikibugs>	 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: toolforge-deploy fails with Helmfile error - https://phabricator.wikimedia.org/T436228 (10fnegri) 03NEW
[10:19:37] <wikibugs>	 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: toolforge-deploy fails with Helmfile error - https://phabricator.wikimedia.org/T436228#12260951 (10fnegri)
[10:20:14] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
[10:22:07] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
[10:35:46] <wikibugs>	 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: toolforge-deploy fails with Helmfile error - https://phabricator.wikimedia.org/T436228#12260985 (10fnegri) Should be fixed by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330345
[10:35:57] <wikibugs>	 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: toolforge-deploy fails with Helmfile error - https://phabricator.wikimedia.org/T436228#12260986 (10fnegri) 05Open→03In progress
[10:40:02] <wikibugs>	 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: toolforge-deploy fails with Helmfile error - https://phabricator.wikimedia.org/T436228#12260996 (10fnegri) a:03fnegri Merged that patch, but it failed to apply in `tools-k8s-control-7`:  ` Notice: /Stage[main]/Kubeadm::Helm/Alternatives::Sele...
[10:41:23] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
[10:41:57] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
[10:43:12] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
[10:43:42] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
[10:44:14] <wikibugs>	 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: toolforge-deploy fails with Helmfile error - https://phabricator.wikimedia.org/T436228#12261014 (10fnegri) 05In progress→03Resolved a:05fnegri→03taavi Ok that was just a race condition, running Puppet a second time fixed it.  The `w...
[10:54:06] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
[10:54:13] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
[11:00:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:05:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:14:18] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/15
[11:14:25] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/27
[11:14:33] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/22
[11:15:28] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/54
[11:15:38] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:16:08] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/39
[11:18:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:28:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:39:44] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[11:43:27] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12261212 (10fgiunchedi) I have repooled clouddumps1002 for dumps-nfs, all good! Feel free to go ahead with...
[11:44:44] <jinxer-wm>	 RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[11:47:41] <wikibugs>	 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, and 3 others: Move WMCS off of Icinga and introduce alertmanager - https://phabricator.wikimedia.org/T328502#12261221 (10fgiunchedi) >>! In T328502#12237329, @Andrew wrote: >>>! In T328502#12231204, @fgiunchedi wrote: >> To be ho...
[12:00:20] <wikibugs>	 10VPS-project-Extdist, 10ExtensionDistributor: extdist servers need to be upgraded - https://phabricator.wikimedia.org/T429659#12261239 (10taavi) The old VM still seems to exist?
[12:14:37] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/15
[12:14:54] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/27
[12:15:09] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/22
[12:16:12] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/54
[12:17:07] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/39
[12:25:59] <wikibugs>	 (03update) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/42
[12:26:06] <wikibugs>	 (03update) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/46
[12:26:08] <wikibugs>	 (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/76
[12:26:25] <wikibugs>	 (03open) 10chimaifeanyi29: T432938: Add interface language switcher with i18n loading mechanism [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/9
[12:28:49] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/1330406 (owner: 10L10n-bot)
[12:28:51] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/massmailer] - 10https://gerrit.wikimedia.org/r/1330407 (owner: 10L10n-bot)
[12:28:52] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1330409 (owner: 10L10n-bot)
[12:36:53] <wikibugs>	 (03update) 10chimaifeanyi29: T432938: Add interface language switcher with i18n loading mechanism [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/9 (https://phabricator.wikimedia.org/T432938)
[12:40:00] <wikibugs>	 (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1330409 (owner: 10L10n-bot)
[12:42:02] <wikibugs>	 (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/1330406 (owner: 10L10n-bot)
[12:42:19] <wikibugs>	 10Cloud-VPS (Quota-requests): Quota increase request for project search - https://phabricator.wikimedia.org/T436242 (10bking) 03NEW
[12:42:38] <wikibugs>	 (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/massmailer] - 10https://gerrit.wikimedia.org/r/1330407 (owner: 10L10n-bot)
[12:46:07] <wikibugs>	 (03update) 10chimaifeanyi29: T432938: Add interface language switcher with i18n loading mechanism [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/9 (https://phabricator.wikimedia.org/T432938)
[12:46:49] <wikibugs>	 (03update) 10dcaro: Silly commit [toolforge-repos/wm-lol] - 10https://gitlab.wikimedia.org/toolforge-repos/wm-lol/-/merge_requests/1
[12:47:01] <wikibugs>	 (03update) 10dcaro: Silly commit [toolforge-repos/wm-lol] - 10https://gitlab.wikimedia.org/toolforge-repos/wm-lol/-/merge_requests/1
[12:48:28] <wikibugs>	 (03update) 10dcaro: Silly commit [toolforge-repos/wm-lol] - 10https://gitlab.wikimedia.org/toolforge-repos/wm-lol/-/merge_requests/1
[12:48:32] <wikibugs>	 (03update) 10dcaro: Silly commit [toolforge-repos/wm-lol] - 10https://gitlab.wikimedia.org/toolforge-repos/wm-lol/-/merge_requests/1
[12:50:43] <wikibugs>	 10Tool-wsindex, 10Wikisource Reader App (Android): Add descriptions for books where Wikipedia articles are available [API] - https://phabricator.wikimedia.org/T430400#12261319 (10Bodhisattwa)
[12:58:40] <wikibugs>	 10Toolforge, 06tools-platform-team: toolforge build clean fails with UNAUTHORIZED error - https://phabricator.wikimedia.org/T436243 (10Don-vip) 03NEW
[12:59:12] <wikibugs>	 10Toolforge, 06tools-platform-team: toolforge build clean fails with UNAUTHORIZED error - https://phabricator.wikimedia.org/T436243#12261350 (10Don-vip)
[13:00:09] <wikibugs>	 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: toolforge-deploy fails with Helmfile error - https://phabricator.wikimedia.org/T436228#12261352 (10fnegri) 05Resolved→03Open Reopening because the patch is not creating the symlink `/usr/bin/helm`:  `lang=shell-session fnegri@tools-k8s-cont...
[13:15:43] <wikibugs>	 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: toolforge-deploy fails with Helmfile error - https://phabricator.wikimedia.org/T436228#12261374 (10taavi) 05Open→03Resolved This was a timing issue; the Puppet code was ordered in a way that `/usr/bin/helm` was still managed by the old...
[13:18:30] <wikibugs>	 10Toolforge, 06tools-platform-team: Toolforge Builds API returns 503 because its Harbor authentication is unauthorized - https://phabricator.wikimedia.org/T436244 (10Schiste) 03NEW
[13:19:59] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[13:20:18] <wikibugs>	 10Toolforge, 06tools-platform-team: maintain-harbor and builds-api should use dedicated Harbor credentials - https://phabricator.wikimedia.org/T436246 (10taavi) 03NEW
[13:20:33] <wikibugs>	 10PAWS, 06tools-platform-team: Access OpenRefine from Jupyter notebook - https://phabricator.wikimedia.org/T436247 (10Reinhard_Kraasch) 03NEW
[13:20:43] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
[13:22:21] <wikibugs>	 (03update) 10dcaro: Silly commit [toolforge-repos/wm-lol] - 10https://gitlab.wikimedia.org/toolforge-repos/wm-lol/-/merge_requests/1
[13:22:23] <wikibugs>	 (03update) 10dcaro: Silly commit [toolforge-repos/wm-lol] - 10https://gitlab.wikimedia.org/toolforge-repos/wm-lol/-/merge_requests/1
[13:24:01] <wikibugs>	 10Toolforge, 06tools-platform-team: maintain-harbor test suite should not output HTTP Authorization header - https://phabricator.wikimedia.org/T436249 (10taavi) 03NEW
[13:25:35] <wikibugs>	 10Toolforge, 06tools-platform-team: Toolforge Builds API returns 503 because its Harbor authentication is unauthorized - https://phabricator.wikimedia.org/T436244#12261481 (10Schiste) →14Duplicate dup:03T436243
[13:25:37] <wikibugs>	 10Toolforge, 06tools-platform-team: toolforge build clean fails with UNAUTHORIZED error - https://phabricator.wikimedia.org/T436243#12261483 (10Schiste)
[13:25:39] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[13:26:00] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
[13:28:39] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[13:28:59] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
[13:29:14] <wikibugs>	 10Toolforge, 06tools-platform-team: Investigate alerting in case all builds fail to start - https://phabricator.wikimedia.org/T436251 (10taavi) 03NEW
[13:29:52] <wikibugs>	 10Cloud-VPS, 10Data-Services, 06tools-infrastructure-team, 06Data-Platform-SRE: Export additional nfsd stats from /proc/fs/nfsd - https://phabricator.wikimedia.org/T436252 (10fgiunchedi) 03NEW
[13:34:51] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[13:39:42] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
[13:49:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[13:54:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[14:02:29] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12261664 (10bking) Per discussion with @brouberol , this reimage caused some Airflow DAGs to fail, as the S...
[14:03:32] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12261665 (10taavi) Is there a specific task for that Airflow issue?
[14:03:33] <wikibugs>	 10Tool-wikinewsie: Add tooltip explanations for "Mix", "Editors", "Edits", etc - https://phabricator.wikimedia.org/T435732#12261667 (10Chimaifeanyi29) Would you be willing to clarify what you want here, I want to work on it. Thank you.
[14:13:05] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12261706 (10bking) I don't think it's been created yet, but we will link as soon as it's ready. In the mean...
[14:14:22] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/15
[14:14:29] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/27
[14:14:36] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/22
[14:15:30] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/54
[14:15:52] <wikibugs>	 10Toolforge, 06tools-platform-team: maintain-harbor test suite should not output HTTP Authorization header - https://phabricator.wikimedia.org/T436249#12261719 (10Raymond_Ndibe) p:05Triage→03High a:03Raymond_Ndibe
[14:16:10] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/39
[14:17:03] <wikibugs>	 10Toolforge, 06tools-platform-team: maintain-harbor and builds-api should use dedicated Harbor credentials - https://phabricator.wikimedia.org/T436246#12261735 (10Raymond_Ndibe) a:03Raymond_Ndibe
[14:17:49] <wikibugs>	 14cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Push-to-Deploy), 13Patch-For-Review: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version - https://phabricator.wikimedia.org/T359649#12261747 (10saper) I have added the information from this ticket to https://wikitech....
[14:25:39] <wikibugs>	 10Toolforge, 06tools-platform-team: toolforge build clean fails with UNAUTHORIZED error - https://phabricator.wikimedia.org/T436243#12261788 (10aputhin) 05Open→03Resolved a:03fnegri `builds-api` issues have been fixed and builds are working normally again. Thanks for the report and do let us know if...
[14:33:06] <wikibugs>	 10Toolforge, 06tools-platform-team: toolforge build clean fails with UNAUTHORIZED error - https://phabricator.wikimedia.org/T436243#12261832 (10Don-vip) Thanks a lot! It works now :)
[14:51:14] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12261906 (10fgiunchedi) >>! In T403154#12261664, @bking wrote: > Per discussion with @brouberol , this reim...
[14:51:36] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12261907 (10bd808) Nearly there. `lang=shell-se...
[14:58:48] <wikibugs>	 (03open) 10raymond-ndibe: functional-tests: stop using verbose in curl [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1392 (https://phabricator.wikimedia.org/T436249)
[15:03:55] <wikibugs>	 (03update) 10raymond-ndibe: models: support webservice job-type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898)
[15:17:22] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Tool-bub2: Migrate bub2 away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435761#12262051 (10komla) @theprotonade @wassan.anmol117 this is a reminder that this project will be shut down on Tuesday, September 1st.  Reach out if you can...
[15:18:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[15:20:12] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Capacity Exchange: Migrate capacity-exchange away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435763#12262059 (10komla) @Albertoleoncio @awight @Ederporto this is a reminder that the `capacity-exchange` project will be shut down on T...
[15:21:32] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12262061 (10bking) Cool, we should be ready next week. Per taavi's request, the ticket for the Airflow work...
[15:23:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[15:24:44] <wikibugs>	 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, and 3 others: Move WMCS off of Icinga and introduce alertmanager - https://phabricator.wikimedia.org/T328502#12262064 (10Andrew) >>! In T328502#12261221, @fgiunchedi wrote: >>>! In T328502#12237329, @Andrew wrote: >>>>! In T32850...
[15:25:42] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate discourse away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T436152#12262066 (10komla) @Samwilson it seems this project can be deleted?  Please confirm, otherwise, we can add it to the list of projects requiring extension past the...
[15:28:27] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate fastcci away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435771#12262069 (10komla) @dschwen this is a reminder that the `fastcci` project will be shut down on Tuesday, September 1st.  Reach out if you cannot meet the deadline.
[15:29:38] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10WMF-Memory-Bank: Migrate foundationmemory away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435772#12262070 (10komla) @Varnent this is a reminder that this project will be shut down on Tuesday, September 1st.  Reach out if you cannot...
[15:33:35] <wikibugs>	 (03merge) 10samtar: Add WikipediaLibrary/ to allowlist [toolforge-repos/wmde-github-phab-bot] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-github-phab-bot/-/merge_requests/5
[15:51:52] <wikibugs>	 (03update) 10raymond-ndibe: functional-tests: stop using verbose in curl [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1392 (https://phabricator.wikimedia.org/T436249)
[16:00:03] <wikibugs>	 (03update) 10raymond-ndibe: functional-tests: stop using verbose in curl [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1392 (https://phabricator.wikimedia.org/T436249)
[16:02:21] <wikibugs>	 (03open) 10chimaifeanyi29: T435732: Add tooltip explanations for Mix, Editors, Edits [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/10 (https://phabricator.wikimedia.org/T435732)
[16:04:15] <wikibugs>	 (03update) 10chimaifeanyi29: T432938: Add interface language switcher with i18n loading mechanism [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/9 (https://phabricator.wikimedia.org/T432938)
[16:06:01] <wikibugs>	 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T435704#12262273 (10aputhin) p:05Triage→03High
[16:06:03] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-platform-team: [pywikibot-buildservice] New upstream release for Pywikibot - https://phabricator.wikimedia.org/T431056#12262272 (10TLepage-WMF) a:03TLepage-WMF
[16:06:16] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-platform-team: [pywikibot-buildservice] New upstream release for Pywikibot - https://phabricator.wikimedia.org/T431056#12262274 (10aputhin) p:05Triage→03High
[16:07:01] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-platform-team: [pywikibot-buildservice] New upstream release for Pywikibot - https://phabricator.wikimedia.org/T431056#12262280 (10aputhin) >>! In T431056#12244172, @Xqt wrote: > Pywikibot on Toolforge is **five releases behind** the current upstream stable version...
[16:10:01] <wikibugs>	 (03update) 10fnegri: functional-tests: stop using verbose in curl [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1392 (https://phabricator.wikimedia.org/T436249) (owner: 10raymond-ndibe)
[16:10:09] <wikibugs>	 (03approved) 10fnegri: functional-tests: stop using verbose in curl [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1392 (https://phabricator.wikimedia.org/T436249) (owner: 10raymond-ndibe)
[16:10:12] <wikibugs>	 (03update) 10fnegri: functional-tests: stop using verbose in curl [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1392 (https://phabricator.wikimedia.org/T436249) (owner: 10raymond-ndibe)
[16:10:19] <wikibugs>	 10Quarry, 06tools-platform-team, 07good first task, 13Patch-For-Review: Add nofollow attribute in some links to prevent bots from following unnecessary ones - https://phabricator.wikimedia.org/T197488#12262318 (10aputhin)
[16:10:41] <wikibugs>	 10VPS-project-Extdist, 10ExtensionDistributor: extdist servers need to be upgraded - https://phabricator.wikimedia.org/T429659#12262319 (10Bawolff) Whoops, i was going to delete after a week once i was sure the new server was working good but i forgot. Deleted now
[16:11:11] <wikibugs>	 10Quarry, 06tools-platform-team, 07good first task, 13Patch-For-Review: Add nofollow attribute in some links to prevent bots from following unnecessary ones - https://phabricator.wikimedia.org/T197488#12262321 (10aputhin) a:03Osps7
[16:11:19] <wikibugs>	 06cloud-services-team, 10Openstack-Magnum, 06tools-infrastructure-team, 13Patch-For-Review: Create two or three shared magnum templates - https://phabricator.wikimedia.org/T429164#12262322 (10Andrew) 05Open→03Resolved
[16:11:21] <wikibugs>	 06cloud-services-team, 10Openstack-Magnum, 06tools-infrastructure-team, 13Patch-For-Review: Investigate new Magnum drivers - https://phabricator.wikimedia.org/T393782#12262324 (10Andrew)
[16:18:57] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 10Ceph, and 3 others: cloudcephosd1044 boot issues - https://phabricator.wikimedia.org/T429267#12262353 (10VRiley-WMF) Will be looking at this today and hopefully we can resolve that issue.
[16:20:18] <wikibugs>	 (03approved) 10raymond-ndibe: functional-tests: stop using verbose in curl [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1392 (https://phabricator.wikimedia.org/T436249)
[16:22:37] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[16:26:09] <wikibugs>	 06tools-infrastructure-team: Track toolforge virtualization footprint: ram/cores/instance &c - https://phabricator.wikimedia.org/T436275 (10Andrew) 03NEW
[16:26:39] <wikibugs>	 06tools-infrastructure-team: Review capacity projection setup for toolforge and cloud-vps - https://phabricator.wikimedia.org/T434128#12262384 (10Andrew) >>! In T434128#12193999, @BLiviero-WMF wrote: > initial thoughts... > - I am not sure how to internalize "subscription" (or over-subscription) and the risks as...
[16:26:46] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
[16:27:50] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
[16:28:32] <icinga-wm>	 PROBLEM - Host cloudcephosd1044 is DOWN: PING CRITICAL - Packet loss = 100%
[16:29:39] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
[16:30:23] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
[16:33:47] <jinxer-wm>	 FIRING: NodeDown: Node cloudcephosd1044 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1044 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown
[16:35:02] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
[16:35:24] <wikibugs>	 (03update) 10raymond-ndibe: functional-tests: stop using verbose in curl [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1392 (https://phabricator.wikimedia.org/T436249)
[16:35:33] <wikibugs>	 (03merge) 10raymond-ndibe: functional-tests: stop using verbose in curl [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1392 (https://phabricator.wikimedia.org/T436249)
[16:38:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[16:39:51] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 10Ceph, and 3 others: cloudcephosd1044 boot issues - https://phabricator.wikimedia.org/T429267#12262463 (10VRiley-WMF) 05In progress→03Resolved reseated backplan and cables. After that I was able to boot it up and from iDRAC, everythi...
[16:41:58] <icinga-wm>	 RECOVERY - Host cloudcephosd1044 is UP: PING OK - Packet loss = 0%, RTA = 0.18 ms
[16:43:47] <jinxer-wm>	 RESOLVED: NodeDown: Node cloudcephosd1044 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1044 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown
[17:18:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[17:20:12] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12262649 (10bd808) I seem to have fixed the opc...
[17:39:19] <wikibugs>	 10Tool-wikinewsie, 07good first task: Wikinewsie i18n - https://phabricator.wikimedia.org/T432938#12262769 (10ZhaoFJx) a:03Chimaifeanyi29 >>! In T432938#12260719, @Chimaifeanyi29 wrote: > I am working on this. please assign it to me.  Feel free to assign the task on your own next time! Kindly refer to [[ htt...
[17:39:36] <wikibugs>	 10Tool-wikinewsie, 07good first task: Wikinewsie i18n - https://phabricator.wikimedia.org/T432938#12262772 (10ZhaoFJx) 05Open→03In progress
[17:40:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[17:43:39] <wikibugs>	 06tools-infrastructure-team: Track toolforge virtualization footprint: ram/cores/instance &c - https://phabricator.wikimedia.org/T436275#12262779 (10Andrew) a:03Andrew
[17:45:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[17:46:27] <wikibugs>	 (03open) 10andrew: Add a new 'legacy' flavor to replace one that was accidentally removed [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/346 (https://phabricator.wikimedia.org/T436275)
[17:47:00] <wikibugs>	 (03update) 10andrew: Add a new 'legacy' flavor to replace one that was accidentally removed [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/346 (https://phabricator.wikimedia.org/T436275)
[17:48:43] <wikibugs>	 (03update) 10andrew: Add a new 'legacy' flavor to replace one that was accidentally removed [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/346 (https://phabricator.wikimedia.org/T436275)
[17:50:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[17:55:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:22:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:27:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:57:06] <wikibugs>	 (03update) 10mahveotm: jobs: Add timeout support for one-off jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/373 (https://phabricator.wikimedia.org/T401110)
[19:21:46] <wikibugs>	 (03PS1) 10Cwhite: add curator example passwords for pcc [labs/private] - 10https://gerrit.wikimedia.org/r/1330608 (https://phabricator.wikimedia.org/T350516)
[19:26:44] <wikibugs>	 10Tool-clip2commons: Add multilingual MediaInfo captions during upload - https://phabricator.wikimedia.org/T434095#12262997 (10Dev) Implemented in [[ https://gitlab.wikimedia.org/toolforge-repos/clip2commons/-/commit/1e8dd58a5f7ab107ad7d5408d2384420980e9ea5 | 1e8dd58  ]] on main: a Caption field on the upload fo...
[19:27:10] <wikibugs>	 10Tool-clip2commons: Add multilingual MediaInfo captions during upload - https://phabricator.wikimedia.org/T434095#12262999 (10Dev) 05Open→03Resolved
[19:29:15] <wikibugs>	 (03CR) 10Cwhite: [V:03+2 C:03+2] add curator example passwords for pcc [labs/private] - 10https://gerrit.wikimedia.org/r/1330608 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite)
[19:32:22] <wikibugs>	 (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/42 (owner: 10l10n-bot)
[19:32:25] <wikibugs>	 (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/42 (owner: 10l10n-bot)
[19:33:05] <wikibugs>	 (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/46 (owner: 10l10n-bot)
[19:33:10] <wikibugs>	 (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/46 (owner: 10l10n-bot)
[19:34:35] <wikibugs>	 (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/76 (owner: 10l10n-bot)
[19:34:42] <wikibugs>	 (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/76 (owner: 10l10n-bot)
[19:44:47] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate imagebulk away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435773#12263036 (10komla) @Jayprakash12345 this is a reminder that this project will be shut down on Tuesday, September 1st.  Reach out if you cannot meet the deadline.
[19:46:48] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate incubator away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435774#12263037 (10komla) @Hydriz this is a reminder that the `incubator` project will be shut down on Tuesday, September 1st.  Reach out if you cannot meet the deadline.
[19:48:16] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10ISA: Migrate isa away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435776#12263039 (10komla) @Eugene233 @NavinoEvans @Sebastian_Berlin-WMSE this is a reminder that the `isa` project will be shut down on Tuesday, September 1st.  Reach...
[19:49:37] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Matrix: Migrate matrix away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435777#12263044 (10komla) @Chicocvenancio @Gergo29 @MarkAHershberger @Tobias1984 this is a reminder that this project will be shut down on Tuesday, September 1st...
[19:50:57] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10a Wikimedia CLI: Migrate mwcli away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435778#12263045 (10komla) @Addshore @jeena  @Ollie.Shotton_WMDE this is a reminder that this project will be shut down on Tuesday, September 1st.  Reach...
[19:54:11] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate ocrtoy away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435779#12263054 (10komla) @Xover this is a reminder that this project will be shut down on Tuesday, September 1st.  Reach out if you cannot meet the deadline.
[19:55:01] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate signwriting away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435781#12263065 (10komla) @Ryan_Lane @Slevinski this is a reminder that this project will be shut down on Tuesday, September 1st.  Reach out if you cannot meet the dead...
[19:56:11] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate spi-tools away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435782#12263080 (10komla) @RoySmith this is a reminder that this project will be shut down on Tuesday, September 1st.  Reach out if you cannot meet the deadline.
[19:57:31] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate text-to-speech away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435785#12263097 (10komla) @TheresNoTime this is a reminder that `text-to-speech` project will be shut down on Tuesday, September 1st.  Reach out if you cannot meet t...
[19:59:22] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate teyora away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435786#12263103 (10komla) @Ed6767 this is a reminder that this project will be shut down on Tuesday, September 1st.  Reach out if you cannot meet the deadline.
[20:00:42] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate utrs away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435787#12263131 (10komla) @TParis this is a reminder that the `utrs` project will be shut down on Tuesday, September 1st.  Reach out if you cannot meet the deadline.
[20:03:32] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate webperf away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435788#12263158 (10komla) This is a reminder that this project will be shut down on Tuesday, September 1st.  Reach out if you cannot meet the deadline.
[20:06:28] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wikidocumentaries away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435789#12263169 (10komla) @NatHillard @Nikerabbit  @Orienteerix @Susannaanas @TuukkaH @Zache this is a reminder that the `wikidocumentaries` project will be shut...
[20:07:06] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wildcat away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435790#12263182 (10komla) @Danny_B  this is a reminder that this project will be shut down on Tuesday, September 1st.  Reach out if you cannot meet the deadline.
[20:08:09] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wmcs-uptime away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435759#12263184 (10komla) @TheresNoTime this is a reminder that `wmcs-uptime` project will be shut down on Tuesday, September 1st.  Reach out if you cannot meet the dea...
[20:09:40] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate utrs away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435787#12263191 (10JJMC89) >>! In T435787#12263131, @komla wrote: > @TParis this is a reminder that the `utrs` project will be shut down on Tuesday, September 1st. >  > Reach...
[20:13:47] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate spi-tools away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435782#12263210 (10RoySmith) Thanks for the ping.  I've been working my way through this slowly, but I'll get back to it.
[20:25:44] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[20:29:23] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate ocrtoy away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435779#12263308 (10Xover) The plan is to complete the migration this weekend.
[20:30:30] <icinga-wm>	 PROBLEM - Host cloudcephosd1044 is DOWN: PING CRITICAL - Packet loss = 100%
[20:30:44] <jinxer-wm>	 RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[20:32:30] <icinga-wm>	 RECOVERY - Host cloudcephosd1044 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms
[20:32:43] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node
[20:33:40] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
[20:35:19] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 10Ceph, and 3 others: cloudcephosd1044 boot issues - https://phabricator.wikimedia.org/T429267#12263348 (10Andrew) Looks good -- I've put it back in service. Thank you!
[20:50:37] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate spi-tools away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435782#12263422 (10RoySmith) Done.  My apologies for dragging my feet on this.
[20:51:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[20:56:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[20:57:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[21:03:30] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 03Readers Essential Work (Sunsetting and cleanup): Migrate reading-web-staging away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435780#12263492 (10Jdlrobson-WMF)
[21:04:47] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12263498 (10bd808) To prove to myself that the...
[21:06:58] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 03Readers Essential Work (Sunsetting and cleanup): Migrate reading-web-staging away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435780#12263526 (10Jdlrobson-WMF) p:05Triage→03Low
[21:12:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[21:22:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[21:32:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[21:38:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[21:47:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:06:44] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[22:18:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[22:21:44] <jinxer-wm>	 RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[22:51:19] <wikibugs>	 (03update) 10bd808: Update README for refactored code [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/8 (https://phabricator.wikimedia.org/T435665)
[23:07:48] <wikibugs>	 (03approved) 10dancy: Update README for refactored code [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/8 (https://phabricator.wikimedia.org/T435665) (owner: 10bd808)
[23:08:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[23:23:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[23:27:05] <wikibugs>	 (03merge) 10bd808: Update README for refactored code [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/8 (https://phabricator.wikimedia.org/T435665)
[23:27:55] <wikibugs>	 10Tool-wheres-my-code-running, 13Patch-For-Review: Setup local and CI linters and tests - https://phabricator.wikimedia.org/T435665#12263837 (10bd808) 05In progress→03Resolved
[23:29:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[23:34:38] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[23:47:57] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Doing 😎): Planning for a plan to migrate Beta Cluster again - https://phabricator.wikimedia.org/T433998#12263848 (10bd808) >>! In T433998#12258671, @ssingh wrote: > Traffic will work on upgrading `deployment...