[00:14:08] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [00:14:13] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [00:14:21] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [00:14:25] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [00:14:37] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [00:14:41] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [00:16:50] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [00:16:55] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [00:17:16] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [00:17:20] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [00:18:13] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [00:18:18] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [00:19:12] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [00:19:19] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [00:19:42] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [00:19:46] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [00:19:59] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [00:20:05] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [00:20:11] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [00:20:15] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [00:20:21] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [00:46:42] 10Tool-wikinewsie, 06WikiNYC Tech Committee: Set up Wikinewsie tool as alpha release on Toolforge - https://phabricator.wikimedia.org/T427575#12269453 (10Pharos) [00:49:32] 10Tool-wikinewsie: Create new Wikidata items from Wikinewsie - https://phabricator.wikimedia.org/T432411#12269455 (10Pharos) Another approach to adapt might be that of https://www.wikidata.org/wiki/Wikidata:Cradle - see for example https://cradle.toolforge.org/#/subject/demonstration_or_protest which I started b... [00:59:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:04:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:37:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:40:15] (03update) 10raymond-ndibe: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) (owner: 10mahveotm) [01:42:12] 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance: [components-api] do not expose rebuild_storage at the api level - https://phabricator.wikimedia.org/T434780#12269478 (10Raymond_Ndibe) 05In progress→03Resolved [01:55:14] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 07Documentation: [tofu-cloudvps] Document importing legacy cloudvps_puppet_prefix objects - https://phabricator.wikimedia.org/T423970#12269479 (10bd808) [01:57:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:19:09] (03approved) 10samwilson: Add optional image pre-processing to suppress show-through [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/14 (owner: 10pacmand) [02:20:25] (03update) 10samwilson: Add optional image pre-processing to suppress show-through [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/14 (https://phabricator.wikimedia.org/T435428) (owner: 10pacmand) [02:21:10] (03merge) 10samwilson: Add optional image pre-processing to suppress show-through [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/14 (https://phabricator.wikimedia.org/T435428) (owner: 10pacmand) [02:29:22] (03update) 10samwilson: Deprecate available_langs API, and use attributes for Open API details [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/17 (https://phabricator.wikimedia.org/T330061) [02:35:53] (03merge) 10samwilson: Deprecate available_langs API, and use attributes for Open API details [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/17 (https://phabricator.wikimedia.org/T330061) [03:13:05] (03PS28) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [03:13:31] (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [03:15:40] FIRING: SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:41:47] (03open) 10bd808: puppet_prefix: Handle empty arrays in hiera data [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/14 (https://phabricator.wikimedia.org/T436412) [04:20:39] (03PS29) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [04:20:56] (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [04:24:24] (03update) 10bd808: puppet_prefix: Handle empty arrays in hiera data [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/14 (https://phabricator.wikimedia.org/T436412) [04:26:38] (03update) 10bd808: puppet_prefix: Handle empty arrays in hiera data [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/14 (https://phabricator.wikimedia.org/T436412) [04:35:28] (03update) 10bd808: puppet_prefix: Handle empty arrays in hiera data [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/14 (https://phabricator.wikimedia.org/T436412) [04:37:46] (03update) 10bd808: puppet_prefix: Handle empty arrays in hiera data [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/14 (https://phabricator.wikimedia.org/T436412) [04:37:52] (03update) 10bd808: puppet_prefix: Handle empty arrays in hiera data [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/14 (https://phabricator.wikimedia.org/T436412) [04:38:57] (03PS30) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [04:55:41] (03PS31) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [05:00:25] (03PS1) 10Krinkle: Switch CVNBot2,3,4,5,13,15,20,21 to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332534 (https://phabricator.wikimedia.org/T327136) [05:04:34] (03CR) 10Krinkle: [C:03+2] Switch CVNBot2,3,4,5,13,15,20,21 to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332534 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [05:05:08] (03Merged) 10jenkins-bot: Switch CVNBot2,3,4,5,13,15,20,21 to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332534 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [05:22:22] (03PS1) 10Krinkle: tasks: Update cvnlog command for pycvnbot [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1332535 (https://phabricator.wikimedia.org/T327136) [05:34:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:39:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:52:49] (03PS1) 10Krinkle: Switch CVNBot1,CVNBot14,CVNBot18 to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332537 (https://phabricator.wikimedia.org/T327136) [05:52:57] (03CR) 10Krinkle: [C:03+2] Switch CVNBot1,CVNBot14,CVNBot18 to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332537 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [05:53:30] (03Merged) 10jenkins-bot: Switch CVNBot1,CVNBot14,CVNBot18 to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332537 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [05:53:36] (03CR) 10Krinkle: [V:03+2 C:03+2] tasks: Update cvnlog command for pycvnbot [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1332535 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [06:00:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:15:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:20:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:25:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:32:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:37:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:14:05] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [07:14:17] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [07:14:31] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [07:15:40] FIRING: SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:16:34] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [07:16:58] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [07:17:44] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [07:18:45] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [07:36:35] 10Cloud-VPS, 06tools-infrastructure-team, 10Cumin, 06Infrastructure-Foundations: Enable cumin hostfile backend on cloudcumin hosts - https://phabricator.wikimedia.org/T433916#12269851 (10elukey) @Andrew I think that the two tasks are duplicates, Riccardo's reply is the reason to not deploy this everywhere.... [07:48:38] (03PS32) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [07:49:45] (03PS33) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [07:49:48] (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [07:50:39] (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [08:49:48] (03CR) 10Hashar: "I haven't looked at the code but here are some comments related to tox/flake/pyproject etc." [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [09:13:57] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [09:14:02] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [09:14:09] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [09:15:03] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [09:15:23] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [09:15:43] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [09:16:29] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [09:19:43] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wikidocumentaries away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435789#12270177 (10Zache) @komla , as now one is not chipped in I can update virtual machine to to Bookworm/Trixie. However, I think that I do not have working s... [09:32:59] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [pywikibot-buildservice] New upstream release for Pywikibot - https://phabricator.wikimedia.org/T431056#12270202 (10TLepage-WMF) 05Open→03Resolved Updated pywikibot in `tool-pywikibot/pywikibot-scripts-stable:latest` to v11.7.0: ` 2026-08-31... [10:02:55] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate utrs away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435787#12270287 (10AmandaNP) I intended to do it this past weekend, but I was sick. I leave for a trip by the end of the week. Can I have until the 4th? [10:08:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:15:40] FIRING: SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:18:10] 06tools-infrastructure-team, 13Patch-For-Review: Track toolforge virtualization footprint: ram/cores/instance &c - https://phabricator.wikimedia.org/T436275#12270449 (10fgiunchedi) >>! In T436275#12266302, @Andrew wrote: > Will prometheus get crabby if I only write the metrics once per day? The queries will be... [11:31:13] PROBLEM - SSH on cloudcephosd1035 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [11:34:02] PROBLEM - Host cloudcephosd1035 is DOWN: PING CRITICAL - Packet loss = 100% [11:34:47] FIRING: NodeDown: Node cloudcephosd1035 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1035 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [11:36:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [11:48:59] 10Data-Services, 06tools-platform-team: WikiReplicas replication stalled for s2, s4, s6 and s7 for three hours now - https://phabricator.wikimedia.org/T436514 (10Count_Count) 03NEW [12:06:04] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate discourse away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T436152#12270576 (10Tgr) For the other Discourse we did {T262275}. I don't recall why that didn't happen for the larger one - maybe because it had private spaces? Or maybe... [12:10:09] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate discourse away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T436152#12270584 (10Tgr) It looks like I actually [[https://space-archive.toolforge.org/|did it]], but the redirect (T370529) wasn't set up. [12:26:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [12:27:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:28:06] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/43 [12:28:07] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/77 [12:28:08] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/47 [12:31:16] (03update) 10raymond-ndibe: global: add public tool search with config endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/186 (https://phabricator.wikimedia.org/T433690) (owner: 10dcaro) [12:42:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:44:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:44:40] (03update) 10raymond-ndibe: global: add public tool search with config endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/186 (https://phabricator.wikimedia.org/T433690) (owner: 10dcaro) [12:49:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:23:43] 10Cloud-VPS, 06tools-infrastructure-team: Redirect discuss-space.wmflabs.org to wmds-archive.toolforge.org - https://phabricator.wikimedia.org/T370529#12270801 (10taavi) 05Open→03Resolved a:03taavi Sorry for the delay here as this was missed previously. Done. [13:29:47] FIRING: NodeDown: Node cloudcephosd1035 has been down for long. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1035 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [13:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:59:17] 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, 10ops-eqiad: cloudcephosd1035 offline - https://phabricator.wikimedia.org/T436520 (10Andrew) 03NEW [14:02:18] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate discourse away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T436152#12270991 (10Tgr) https://space-archive.toolforge.org/ redirects now (thx @taavi) so I think there's nothing else to do here, the VMs can be deleted. The discourse... [14:06:46] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate discourse away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T436152#12271141 (10taavi) Thanks! Please delete the VMs once you are ready, I can then take care of deleting the project (which is not needed for the redirect, since it t... [14:17:07] 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, 10ops-eqiad: cloudcephosd1035 offline - https://phabricator.wikimedia.org/T436520#12271173 (10VRiley-WMF) a:03VRiley-WMF [14:22:06] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate utrs away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435787#12271193 (10komla) @AmandaNP, yes. I will add this to the list of projects asking for extension! [14:25:06] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wikidocumentaries away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435789#12271208 (10komla) @Zache noted. I will add this to the list of projects asking for an extension [14:27:39] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate ocrtoy away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435779#12271227 (10komla) Thank you! [14:51:53] (03PS34) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [14:52:42] (03PS35) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [14:52:47] (03PS36) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [15:02:42] 10Data-Services, 06tools-platform-team: WikiReplicas replication stalled for s2, s4, s6 and s7 for three hours now - https://phabricator.wikimedia.org/T436514#12271414 (10Count_Count) 05Open→03Resolved a:03Count_Count Fixed now. [15:03:00] (03PS37) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [15:07:33] (03CR) 10Krinkle: "Thanks. RE: EventStreams, not in the short term as I have other more immediate things to deal with, including upstream QUnit and other too" [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [15:10:38] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfdw.wmnet' [15:10:38] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2005-dev.codfdw.wmnet' [15:10:44] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfdw.wmnet' [15:10:44] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2006-dev.codfdw.wmnet' [15:11:12] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet' [15:15:40] FIRING: SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:21:46] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:23:12] (03update) 10raymond-ndibe: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) (owner: 10mahveotm) [15:23:51] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet' [15:26:46] RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:28:08] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet' (T436113) [15:42:06] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet' (T436113) [15:42:25] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2010-dev.codfw.wmnet' (T436113) [15:46:07] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33410122686 (https://github.com/cluebotng/component-configs/commits/f04e19b5856fdc410758a31113106c5d61074e04) [15:46:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [15:48:39] (03PS38) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [15:48:55] (03update) 10raymond-ndibe: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) (owner: 10mahveotm) [15:48:59] 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, 10ops-eqiad, 06SRE: cloudcephosd1035 offline - https://phabricator.wikimedia.org/T436520#12271704 (10VRiley-WMF) 05Open→03Resolved This unit is back up. I updated the BIOS and that seemed to correct the issue. [15:50:34] (03update) 10raymond-ndibe: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) (owner: 10mahveotm) [15:50:34] (03update) 10raymond-ndibe: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) (owner: 10mahveotm) [15:51:04] RECOVERY - SSH on cloudcephosd1035 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u10 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [15:51:06] RECOVERY - Host cloudcephosd1035 is UP: PING OK - Packet loss = 0%, RTA = 0.50 ms [15:54:47] RESOLVED: NodeDown: Node cloudcephosd1035 has been down for long. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1035 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [15:54:47] RESOLVED: NodeDown: Node cloudcephosd1035 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1035 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [15:56:35] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2010-dev.codfw.wmnet' (T436113) [15:56:46] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12271764 (10bd808) I made a [[https://gitlab.wikimedia.org/cloudvps-r... [16:02:05] 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, 10ops-eqiad, 06SRE: cloudcephosd1035 offline - https://phabricator.wikimedia.org/T436520#12271784 (10Andrew) I do not see anything obvious in the logs explaining the crash. I will put this back in service and we'll see how things go. [16:02:49] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33411677163 (https://github.com/cluebotng/component-configs/commits/4ce131b4635a642a351fd11a9bfe616b5cf675a4) [16:02:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [16:03:12] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' (T436113) [16:12:05] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet' (T436113) [16:13:36] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' (T436113) [16:22:17] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet' (T436113) [16:25:52] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet' (T436113) [16:31:18] (03open) 10taavi: envvars-admission: Drop old ElasticSearch-related variables [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1394 (https://phabricator.wikimedia.org/T401818) [16:32:13] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet' (T436113) [16:33:32] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet' (T436113) [16:39:32] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33415131548 (https://github.com/cluebotng/component-configs/commits/cd0c47ddd7ab13987792b87be0155ebff12a87bb) [16:39:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [16:39:36] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet' (T436113) [16:39:56] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet' (T436113) [16:44:10] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33415531754 (https://github.com/cluebotng/component-configs/commits/fa3e6f3691a39e901537a1debc031329ff5c3eeb) [16:44:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [16:45:52] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet' (T436113) [16:46:58] 10Cloud-VPS (Quota-requests), 10Wikidata: Quota increase request for project pm20database - https://phabricator.wikimedia.org/T433839#12271978 (10fnegri) 05Open→03Declined Declining this task, @bking please proceed with one of the two options outlined by @Andrew above. [16:53:13] 10Cloud-VPS (Quota-requests): Quota increase request for project search - https://phabricator.wikimedia.org/T436242#12271994 (10fnegri) > testing Ceph integrations with 3 different VMMs Is this somehow related to "search" functionality? If not, please request a dedicated project instead: https://phabricator.wik... [17:09:55] !log tools.cluebotng-staging Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/33418019170 (https://github.com/cluebotng/component-configs/commits/c689d315eb65ad2cb2f8d0b28293864f7948ce53) [17:09:55] !log tools.cluebotng-monitoring Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/33418019250 (https://github.com/cluebotng/component-configs/commits/c689d315eb65ad2cb2f8d0b28293864f7948ce53) [17:12:19] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33418019192 (https://github.com/cluebotng/component-configs/commits/c689d315eb65ad2cb2f8d0b28293864f7948ce53) [17:12:22] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33418019342 (https://github.com/cluebotng/component-configs/commits/c689d315eb65ad2cb2f8d0b28293864f7948ce53) [17:12:27] !log tools.cluebot-syncer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33418019235 (https://github.com/cluebotng/component-configs/commits/c689d315eb65ad2cb2f8d0b28293864f7948ce53) [17:12:32] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33418019188 (https://github.com/cluebotng/component-configs/commits/c689d315eb65ad2cb2f8d0b28293864f7948ce53) [17:13:40] (03PS39) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [17:13:45] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33418019223 (https://github.com/cluebotng/component-configs/commits/c689d315eb65ad2cb2f8d0b28293864f7948ce53) [17:16:04] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wikidocumentaries away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435789#12272069 (10Zache) @komla Thank you! [17:45:30] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33421096964 (https://github.com/cluebotng/component-configs/commits/50d17b37689aacfab96d6a5683d74be969ee0bc3) [17:45:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [17:48:28] 10Tool-wikinewsie: Wikinewsie aggregation of stories - https://phabricator.wikimedia.org/T436544 (10Pharos) 03NEW [17:49:21] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/43 (owner: 10l10n-bot) [17:49:24] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/43 (owner: 10l10n-bot) [17:50:13] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/77 (owner: 10l10n-bot) [17:50:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:53:21] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/77 (owner: 10l10n-bot) [17:54:34] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/47 (owner: 10l10n-bot) [17:54:38] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/47 (owner: 10l10n-bot) [17:55:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:07:14] 10Tool-wikinewsie: Wikinewsie aggregation of stories - https://phabricator.wikimedia.org/T436544#12272233 (10Pharos) [18:09:45] (03open) 10taavi: Rebrand Elasticsearch alerts as OpenSearch alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/71 (https://phabricator.wikimedia.org/T401818) [18:32:17] 10Cloud-VPS (Debian Bullseye Deprecation): Extension request for account-creation-assistance - https://phabricator.wikimedia.org/T433749#12272320 (10stwalkerster) 05Open→03Resolved a:03stwalkerster All our instances are now trixie, all bullseye/bookworm instances are gone. [18:44:14] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace deployment-wikifeeds02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436462#12272360 (10Southparkfan) The VM has been decommissioned, but [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+... [18:47:43] (03PS40) 10Krinkle: Rewrite CVNBot in Python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [18:49:21] (03PS41) 10Krinkle: Rewrite CVNBot in Python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [18:55:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:00:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:01:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:06:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:08:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:09:55] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [19:10:30] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [19:10:41] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [19:11:06] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [19:11:50] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [19:12:16] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [19:15:40] FIRING: SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:23:50] (03PS42) 10Krinkle: Rewrite CVNBot in Python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [19:23:50] (03PS1) 10Krinkle: ListManager: Optimize classify_editor from 3 queries to 1 query [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332795 (https://phabricator.wikimedia.org/T327136) [19:25:20] (03PS2) 10Krinkle: ListManager: Optimize classify_editor from 3 queries to 1 query [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332795 (https://phabricator.wikimedia.org/T327136) [19:31:59] 10Cloud-VPS (Debian Bullseye Deprecation), 10Countervandalism-Network: Upgrade cvn fleet from Debian 11 Bullseye to Debian 12 Bookworm (PHP 7.4 > PHP 8.2) - https://phabricator.wikimedia.org/T395164#12272502 (10AntiCompositeNumber) [19:47:05] (03PS3) 10Krinkle: ListManager: Optimize classify_editor from 3 queries to 1 query [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332795 (https://phabricator.wikimedia.org/T327136) [19:48:31] (03CR) 10Krinkle: [C:03+2] Rewrite CVNBot in Python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [19:49:00] (03CR) 10Krinkle: [C:03+2] ListManager: Optimize classify_editor from 3 queries to 1 query [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332795 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [19:49:39] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [19:49:58] (03Merged) 10jenkins-bot: Rewrite CVNBot in Python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [19:50:18] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [19:50:23] (03Merged) 10jenkins-bot: ListManager: Optimize classify_editor from 3 queries to 1 query [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332795 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [19:51:27] (03open) 10ladsgroup: Allow thumb.wikimedia.org as host for images [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/18 (https://phabricator.wikimedia.org/T436551) [19:52:54] (03PS1) 10Krinkle: Switch CVNBot_v5 template to /srv/cvn/git/CVNBot directory [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332800 [19:55:23] (03CR) 10Krinkle: [C:03+2] Switch CVNBot_v5 template to /srv/cvn/git/CVNBot directory [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332800 (owner: 10Krinkle) [19:55:59] (03Merged) 10jenkins-bot: Switch CVNBot_v5 template to /srv/cvn/git/CVNBot directory [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332800 (owner: 10Krinkle) [19:56:19] (03PS1) 10Krinkle: Swigtch one bot on cvn-app12 to CVNBot 5 (pycvnbot) [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332801 (https://phabricator.wikimedia.org/T327136) [19:58:12] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12272658 (10Southparkfan) >>! In T435393#12269425, @bd808 wrote: > Th... [19:59:36] (03PS2) 10Krinkle: Switch one bot on cvn-app12 to CVNBot 5 (pycvnbot) [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332801 (https://phabricator.wikimedia.org/T327136) [19:59:40] (03CR) 10Krinkle: [C:03+2] Switch one bot on cvn-app12 to CVNBot 5 (pycvnbot) [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332801 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [20:00:15] (03Merged) 10jenkins-bot: Switch one bot on cvn-app12 to CVNBot 5 (pycvnbot) [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332801 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [20:01:52] 10Cloud-VPS, 06tools-infrastructure-team: Update ENC Hiera value column type to MEDIUMTEXT - https://phabricator.wikimedia.org/T436561 (10taavi) 03NEW [20:02:30] 10Cloud-VPS, 06tools-infrastructure-team: Update ENC Hiera value column type to MEDIUMTEXT - https://phabricator.wikimedia.org/T436561#12272683 (10taavi) a:03taavi [20:04:14] 10Cloud-VPS, 06tools-infrastructure-team: Update ENC Hiera value column type to MEDIUMTEXT - https://phabricator.wikimedia.org/T436561#12272684 (10taavi) In `codfw1dev`, seems to work just fine: ` MariaDB [labspuppet]> ALTER TABLE hieraassignment CHANGE COLUMN hiera_data hiera_data MEDIUMTEXT NOT NULL; Query O... [20:07:00] (03PS1) 10Krinkle: Revert CVNBot6 back [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332803 (https://phabricator.wikimedia.org/T327136) [20:07:08] (03PS2) 10Krinkle: Revert CVNBot6 back [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332803 (https://phabricator.wikimedia.org/T327136) [20:07:20] (03CR) 10Krinkle: [C:03+2] Revert CVNBot6 back [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332803 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [20:07:54] (03Merged) 10jenkins-bot: Revert CVNBot6 back [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332803 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [20:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:21:53] (03PS1) 10Krinkle: Program: Start global catcher earlier [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332806 (https://phabricator.wikimedia.org/T327136) [20:21:56] (03PS1) 10Krinkle: IrcClient: Increase send_delay from 0.3s to 0.4s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332807 [20:24:37] (03CR) 10Krinkle: [C:03+2] Program: Start global catcher earlier [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332806 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [20:25:20] (03CR) 10Krinkle: [C:03+2] IrcClient: Increase send_delay from 0.3s to 0.4s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332807 (owner: 10Krinkle) [20:26:15] (03Merged) 10jenkins-bot: Program: Start global catcher earlier [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332806 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [20:27:09] (03Merged) 10jenkins-bot: IrcClient: Increase send_delay from 0.3s to 0.4s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332807 (owner: 10Krinkle) [20:27:47] (03approved) 10samwilson: Allow thumb.wikimedia.org as host for images [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/18 (https://phabricator.wikimedia.org/T436551) (owner: 10ladsgroup) [20:27:52] (03merge) 10samwilson: Allow thumb.wikimedia.org as host for images [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/18 (https://phabricator.wikimedia.org/T436551) (owner: 10ladsgroup) [20:32:28] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-wikifeeds02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436462#12272771 (10Southparkfan) 05In progress→03Resolved [20:35:19] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [20:35:19] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan+apply for main branch [20:35:22] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [20:35:58] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [20:49:30] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' (T436113) [21:01:20] PROBLEM - SSH on cloudcontrol1006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [21:01:22] FIRING: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [21:02:10] FIRING: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [21:02:18] RECOVERY - SSH on cloudcontrol1006 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u4 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [21:02:24] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' (T436113) [21:02:42] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' (T436113) [21:02:43] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1005.eqiad.wmnet' (T436113) [21:02:48] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' (T436113) [21:06:22] RESOLVED: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [21:07:10] RESOLVED: GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [21:14:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [21:14:29] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate discourse away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T436152#12272901 (10Tgr) Deleted. [21:15:57] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' (T436113) [21:16:01] !log taavi@cloudcumin1001 discourse START - Cookbook wmcs.vps.delete_project for project discourse in eqiad1 [21:16:28] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: delete project discourse [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/347 [21:17:10] (03merge) 10taavi: projects: delete project discourse [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/347 (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [21:18:02] !log taavi@cloudcumin1001 discourse END (PASS) - Cookbook wmcs.vps.delete_project (exit_code=0) for project discourse in eqiad1 [21:18:28] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate discourse away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T436152#12272909 (10taavi) 05Open→03Resolved a:03Tgr Removed the project as well, thanks! [21:20:25] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet' (T436113) [21:20:55] RESOLVED: SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:26:20] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12272969 (10Southparkfan) As mentioned @IRC: I have performed a smoke... [21:32:47] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [21:33:10] FIRING: GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [21:33:52] FIRING: [30x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [21:34:12] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1011.eqiad.wmnet' (T436113) [21:35:07] FIRING: [30x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [21:38:10] RESOLVED: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [21:38:52] RESOLVED: [18x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1011.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [21:44:09] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' (T436113) [21:52:16] PROBLEM - Host cloudnet1005 is DOWN: PING CRITICAL - Packet loss = 100% [21:54:31] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' (T436113) [21:54:46] RECOVERY - Host cloudnet1005 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [21:55:17] (03PS1) 10Krinkle: Switch one bot on cvn-app12 to CVNBot 5, again [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332815 (https://phabricator.wikimedia.org/T327136) [21:55:42] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12273057 (10Southparkfan) @Krinkle Hi! You have migrated the Beta Clu... [21:55:45] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' (T436113) [21:56:21] (03CR) 10Krinkle: [C:03+2] Switch one bot on cvn-app12 to CVNBot 5, again [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332815 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [21:56:55] (03Merged) 10jenkins-bot: Switch one bot on cvn-app12 to CVNBot 5, again [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332815 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [22:06:46] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/33415504817 (https://github.com/cluebotng/component-configs/commits/c8b5e187308387092b1a090b7a94a51d8952328d) [22:09:26] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33418019331 (https://github.com/cluebotng/component-configs/commits/c689d315eb65ad2cb2f8d0b28293864f7948ce53) [22:21:16] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' (T436113) [22:21:17] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' (T436113) [22:26:51] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' (T436113) [22:26:52] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' (T436113) [22:32:34] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' (T436113) [22:32:34] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' (T436113) [22:36:51] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate dumps away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435767#12273217 (10komla) @Hydriz @Nemo_bis this is a reminder that the `dumps` project will be shut down tomorrow, September 1st. Please reach out if you cannot meet the de... [22:38:13] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' (T436113) [22:38:14] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' (T436113) [22:43:41] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' (T436113) [22:43:42] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' (T436113) [22:48:57] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' (T436113) [22:48:58] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1046.eqiad.wmnet' (T436113) [22:54:46] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' (T436113) [22:54:47] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' (T436113) [22:59:48] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' (T436113) [22:59:50] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' (T436113) [23:04:15] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' (T436113) [23:04:16] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' (T436113) [23:09:13] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' (T436113) [23:09:14] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1050.eqiad.wmnet' (T436113) [23:14:35] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' (T436113) [23:14:36] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' (T436113) [23:19:26] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' (T436113) [23:19:27] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' (T436113) [23:24:38] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' (T436113) [23:24:39] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1053.eqiad.wmnet' (T436113) [23:30:18] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' (T436113) [23:30:19] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' (T436113) [23:34:59] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' (T436113) [23:35:00] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' (T436113) [23:39:53] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' (T436113) [23:39:54] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' (T436113) [23:44:19] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' (T436113) [23:44:20] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' (T436113) [23:48:58] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' (T436113) [23:49:00] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' (T436113) [23:52:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:54:11] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' (T436113) [23:54:12] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' (T436113) [23:57:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown