[00:14:02] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/13 [00:14:08] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/13 [00:14:16] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/25 [00:14:20] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/25 [00:14:30] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/20 [00:14:34] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/20 [00:16:35] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/52 [00:16:39] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/52 [00:16:55] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: tofu-infra: add cinder volume types - https://phabricator.wikimedia.org/T410148#12245035 (10Mahveotm) Hi, I found this exciting, and would like to work on this task. I've done some digging in, and started a WIP implementation in `tofu-infra`.... [00:17:01] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/78 [00:17:04] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/78 [00:17:41] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/37 [00:17:44] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/37 [00:18:31] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/41 [00:18:36] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/41 [00:18:57] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/26 [00:18:57] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/26 [00:19:01] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/26 [00:19:11] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/29 [00:19:16] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/29 [00:19:19] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/32 [00:19:22] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/32 [00:19:25] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/32 [00:47:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:57:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:02:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:18:10] 10Tool-wikinewsie: Adjustable image alignment on Wikinewsie - https://phabricator.wikimedia.org/T434467#12245073 (10Habst) 05Open→03Resolved a:03Habst [01:18:31] 10Tool-wikinewsie: Wikinewsie uses low-resolution thumbnails when higher-resolution exists - https://phabricator.wikimedia.org/T433758#12245076 (10Habst) 05Open→03Resolved a:03Habst [01:18:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:23:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:23:58] 10Tool-wikinewsie: Adjustable image alignment on Wikinewsie - https://phabricator.wikimedia.org/T434467#12245078 (10Habst) 05Resolved→03Open [02:44:59] 10Tool-wikinewsie: Adjustable image alignment on Wikinewsie - https://phabricator.wikimedia.org/T434467#12245099 (10Pharos) If it's too slow to do universally, maybe this could be set on a case-by-case basis by a user (via an 'adjust crop' button on the article card), and then the result is saved for other users. [02:46:03] FIRING: ProjectProxyMainProxyDown: Proxy service address is unreachable - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyDown [02:51:03] RESOLVED: ProjectProxyMainProxyDown: Proxy service address is unreachable - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyDown [03:15:58] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Nairobi 29): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12245106 (10cscott) Is it possible to just sk... [03:23:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:28:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:33:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:04:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:07:33] (03open) 10bd808: Draft: Setup local and CI linters and tests [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/4 (https://phabricator.wikimedia.org/T435665) [04:09:00] (03close) 10bd808: Draft: Poorly factored application of project layout, lint, and CI [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/3 [04:09:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:34:14] (03PS19) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [05:11:53] 10Toolforge, 06tools-platform-team: Link for https://wikilinkchecker.toolforge.org/ is broken on toolhub.wikimedia.org - https://phabricator.wikimedia.org/T435752 (10Betateschter) 03NEW [05:12:23] 10Toolforge, 06tools-platform-team: Link for https://wikilinkchecker.toolforge.org/ is broken on toolhub.wikimedia.org - https://phabricator.wikimedia.org/T435752#12245133 (10Betateschter) [05:20:02] 10Tool-wikilinkchecker: Link for https://wikilinkchecker.toolforge.org/ is broken on toolhub.wikimedia.org - https://phabricator.wikimedia.org/T435752#12245134 (10JJMC89) [05:28:04] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Nairobi 29): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12245143 (10Peter) p:05High→03Low > Per T... [05:56:23] 10Tool-techcontribs: Undefined Gerrit emails causes whole page load failure - https://phabricator.wikimedia.org/T435755 (10Chlod) 03NEW [05:56:45] 10Tool-techcontribs: Undefined Gerrit emails causes whole page load failure - https://phabricator.wikimedia.org/T435755#12245214 (10Chlod) p:05Triage→03High [07:22:12] 10Cloud-VPS (Debian Bullseye Deprecation): Debian Bullseye deprecation extension for statanalyser - https://phabricator.wikimedia.org/T433570#12245282 (10komla) @Leaderboard do you consider this resolved or will you be working on an OS upgrade? I have marked this as a project on Trixie. [07:34:35] 06cloud-services-team, 10Castor, 06Collaboration-Services, 10Continuous-Integration-Infrastructure, and 2 others: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12245304 (10hashar) I worked around the issue by having the Castor agent to be connected via the ins... [08:00:12] 10Toolforge, 06tools-platform-team: [lima-kilo] building locally jobs-api when a .tox env exists in the host code dir errors out - https://phabricator.wikimedia.org/T432276#12245347 (10dcaro) This happens when running lima 2.2.0, on fedora 44, with the VM running debian 13.1 (reinstalled today), docker 29.7.2,... [08:27:11] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wmcs-uptime away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435759 (10komla) 03NEW [08:31:07] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wmcs-uptime away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435759#12245444 (10komla) [08:33:19] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate bub2 away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435761 (10komla) 03NEW [08:33:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:36:21] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate capacity-exchange away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435763 (10komla) 03NEW [08:36:27] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate cvn away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435764 (10komla) 03NEW [08:38:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:40:44] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate cvn away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435764#12245534 (10komla) →14Duplicate dup:03T395164 [08:40:45] 10Cloud-VPS (Debian Bullseye Deprecation), 10Countervandalism-Network: Upgrade cvn fleet from Debian 11 Bullseye to Debian 12 Bookworm (PHP 7.4 > PHP 8.2) - https://phabricator.wikimedia.org/T395164#12245536 (10komla) [08:43:39] 10Cloud-VPS (Debian Bullseye Deprecation), 10Countervandalism-Network: Upgrade cvn fleet from Debian 11 Bullseye to Debian 12 Bookworm (PHP 7.4 > PHP 8.2) - https://phabricator.wikimedia.org/T395164#12245565 (10Krinkle) Rather than invest in backporting old Mono to new Debian, or invest in migrating CVNBot to... [08:44:09] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate dumps away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435767 (10komla) 03NEW [08:47:29] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate etytree away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435770 (10komla) 03NEW [08:47:31] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate fastcci away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435771 (10komla) 03NEW [08:47:39] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate foundationmemory away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435772 (10komla) 03NEW [08:47:42] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate imagebulk away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435773 (10komla) 03NEW [08:47:45] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate incubator away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435774 (10komla) 03NEW [08:49:43] (03CR) 10CI reject: [V:04-1] build: Add type hints, simplify jsonpCallback logic, add `composer serve` [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328374 (owner: 10Krinkle) [08:49:48] (03CR) 10CI reject: [V:04-1] api.php: Implement a new `recent=1` feature to return last 30 rows [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328375 (owner: 10Krinkle) [08:51:49] (03PS3) 10Krinkle: build: Add type hints, simplify jsonpCallback logic, add `composer serve` [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328374 [08:51:53] (03PS2) 10Krinkle: api.php: Implement a new `recent=1` feature to return last 30 rows [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328375 [08:52:47] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate isa away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435776 (10komla) 03NEW [08:52:53] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate matrix away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435777 (10komla) 03NEW [08:52:57] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate mwcli away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435778 (10komla) 03NEW [08:53:01] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate ocrtoy away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435779 (10komla) 03NEW [08:53:07] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate reading-web-staging away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435780 (10komla) 03NEW [08:53:12] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate signwriting away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435781 (10komla) 03NEW [08:53:17] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate spi-tools away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435782 (10komla) 03NEW [08:53:22] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate swift away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435783 (10komla) 03NEW [08:53:29] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate text-to-speech away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435785 (10komla) 03NEW [08:53:31] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate teyora away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435786 (10komla) 03NEW [08:53:39] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate utrs away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435787 (10komla) 03NEW [08:53:41] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate webperf away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435788 (10komla) 03NEW [08:53:48] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wikidocumentaries away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435789 (10komla) 03NEW [08:53:52] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wildcat away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435790 (10komla) 03NEW [08:57:51] (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [08:58:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:05:03] 10Toolforge, 06tools-platform-team, 07OKR-Work: Email tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865#12245889 (10dcaro) > one is storing configuration in the Metricsinfra db, the other is discovering later that using webhooks has some limitation that requires... [09:09:32] (03update) 10dcaro: Remove commit as an option in help message for `toolforge build start --ref` [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/149 (https://phabricator.wikimedia.org/T429227) (owner: 10sebastian-berlin-wmse) [09:09:39] (03update) 10dcaro: Remove commit as an option in help message for `toolforge build start --ref` [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/149 (https://phabricator.wikimedia.org/T429227) (owner: 10sebastian-berlin-wmse) [09:09:59] (03update) 10dcaro: emailer: include pod status message in failure emails [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/58 (https://phabricator.wikimedia.org/T384252) (owner: 10mahveotm) [09:10:12] (03update) 10dcaro: cli: add timeout support for one-off jobs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/176 (https://phabricator.wikimedia.org/T401110) (owner: 10mahveotm) [09:10:14] (03CR) 10CI reject: [V:04-1] build: Add type hints, simplify jsonpCallback logic, add `composer serve` [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328374 (owner: 10Krinkle) [09:10:14] (03CR) 10CI reject: [V:04-1] api.php: Implement a new `recent=1` feature to return last 30 rows [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328375 (owner: 10Krinkle) [09:10:24] (03update) 10dcaro: jobs: Add timeout support for one-off jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/373 (https://phabricator.wikimedia.org/T401110) (owner: 10mahveotm) [09:10:36] (03update) 10dcaro: cli: preserve option separators for subcommands [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/74 (https://phabricator.wikimedia.org/T370184) (owner: 10mahveotm) [09:10:47] (03PS20) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [09:12:14] (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [09:13:33] 10Toolforge, 06tools-platform-team: [builds-cli] remove wrong help message for `--ref` - https://phabricator.wikimedia.org/T435793 (10dcaro) 03NEW [09:13:34] (03update) 10dcaro: Remove commit as an option in help message for `toolforge build start --ref` [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/149 (https://phabricator.wikimedia.org/T435793) (owner: 10sebastian-berlin-wmse) [09:13:59] 10Toolforge, 06tools-platform-team: [builds-cli] remove wrong help message for `--ref` - https://phabricator.wikimedia.org/T435793#12245925 (10dcaro) 05Open→03In progress p:05Triage→03Low [09:30:43] 10Tool-bash, 10Tool-sal, 10Stashbot: Migrate storage to new OpenSearch v2 Toolforge cluster - https://phabricator.wikimedia.org/T435223#12245957 (10taavi) 05Open→03Resolved a:03taavi [09:36:06] (03update) 10dcaro: Silly commit [toolforge-repos/wm-lol] - 10https://gitlab.wikimedia.org/toolforge-repos/wm-lol/-/merge_requests/1 [09:45:58] (03update) 10dcaro: Silly commit [toolforge-repos/wm-lol] - 10https://gitlab.wikimedia.org/toolforge-repos/wm-lol/-/merge_requests/1 [09:50:14] (03update) 10dcaro: Silly commit [toolforge-repos/wm-lol] - 10https://gitlab.wikimedia.org/toolforge-repos/wm-lol/-/merge_requests/1 [09:51:59] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/32713575575 (https://github.com/cluebotng/component-configs/commits/15461b10d7f124d327ec6065e11b3fce0465cc68) [09:52:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [09:59:50] 10Cloud-VPS (Debian Bullseye Deprecation), 10SRE-swift-storage: Migrate swift away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435783#12246060 (10Peachey88) [10:03:50] 10Cloud-VPS (Debian Bullseye Deprecation), 06Reader Experience Team, 06Reader Growth Team: Migrate reading-web-staging away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435780#12246073 (10Peachey88) [10:04:31] 10Cloud-VPS (Debian Bullseye Deprecation), 10Tool-mwcli, 10a Wikimedia CLI: Migrate mwcli away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435778#12246078 (10Peachey88) [10:05:01] 10Toolforge, 06tools-platform-team: [logs-api] rename jobs-api source to jobs source for event logs - https://phabricator.wikimedia.org/T435798 (10dcaro) 03NEW [10:05:31] 10Cloud-VPS (Debian Bullseye Deprecation), 10Matrix: Migrate matrix away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435777#12246091 (10Peachey88) [10:06:02] 10Cloud-VPS (Debian Bullseye Deprecation), 10ISA: Migrate isa away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435776#12246106 (10Peachey88) [10:06:58] 10Cloud-VPS (Debian Bullseye Deprecation), 10SRE-swift-storage: Migrate swift away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435783#12246113 (10MatthewVernon) a:03Eevans [the remaining older systems in this project are cassandra-related] [10:07:15] 10Cloud-VPS (Debian Bullseye Deprecation), 10WMF-Memory-Bank: Migrate foundationmemory away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435772#12246116 (10Peachey88) [10:09:45] 10Cloud-VPS (Debian Bullseye Deprecation), 10Capacity Exchange: Migrate capacity-exchange away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435763#12246120 (10Peachey88) [10:10:07] 10Cloud-VPS (Debian Bullseye Deprecation), 10Tool-bub2: Migrate bub2 away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435761#12246121 (10Peachey88) [10:11:49] (03open) 10dcaro: write_jobs_logs: rename from `jobs-api` to `jobs` source [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/33 (https://phabricator.wikimedia.org/T435798) [10:11:52] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [logs-api] rename jobs-api source to jobs source for event logs - https://phabricator.wikimedia.org/T435798#12246125 (10dcaro) p:05Triage→03High [10:12:44] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-api] rename jobs-api source to jobs source for event logs - https://phabricator.wikimedia.org/T435798#12246141 (10dcaro) [10:17:50] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate webperf away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435788#12246143 (10Peachey88) [10:23:41] (03update) 10raymond-ndibe: Remove commit as an option in help message for `toolforge build start --ref` [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/149 (https://phabricator.wikimedia.org/T435793) (owner: 10sebastian-berlin-wmse) [10:23:43] (03approved) 10raymond-ndibe: Remove commit as an option in help message for `toolforge build start --ref` [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/149 (https://phabricator.wikimedia.org/T435793) (owner: 10sebastian-berlin-wmse) [10:31:27] 10Tool-wikinewsie, 10LPL sprints, 06translatewiki.net, 10LPL Projects (Ongoing maintenance): Add Wikinewsie to translatewiki.net - https://phabricator.wikimedia.org/T435739#12246150 (10Nikerabbit) [10:33:05] 10Tool-wikinewsie, 10LPL sprints, 06translatewiki.net, 10LPL Projects (Ongoing maintenance): Add Wikinewsie to translatewiki.net - https://phabricator.wikimedia.org/T435739#12246154 (10Nikerabbit) [10:37:37] 10Tool-wikinewsie, 10LPL sprints, 06translatewiki.net, 10LPL Projects (Ongoing maintenance): Add Wikinewsie to translatewiki.net - https://phabricator.wikimedia.org/T435739#12246161 (10Nikerabbit) p:05Triage→03Medium @ZhaoFJx Please give access to push to main to https://gitlab.wikimedia.org/l10n-bot i... [10:49:03] 10Tool-wikinewsie, 10LPL sprints, 06translatewiki.net, 10LPL Projects (Ongoing maintenance): Add Wikinewsie to translatewiki.net - https://phabricator.wikimedia.org/T435739#12246200 (10Nikerabbit) [11:06:27] (03update) 10raymond-ndibe: cli: preserve option separators for subcommands [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/74 (https://phabricator.wikimedia.org/T370184) (owner: 10mahveotm) [11:07:32] (03update) 10raymond-ndibe: cli: add timeout support for one-off jobs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/176 (https://phabricator.wikimedia.org/T401110) (owner: 10mahveotm) [11:09:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:14:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:19:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:27:24] (03update) 10dcaro: write_jobs_logs: rename from `jobs-api` to `jobs` source [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/33 (https://phabricator.wikimedia.org/T435798) [11:34:23] (03update) 10raymond-ndibe: api: add envvars to source build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/184 (https://phabricator.wikimedia.org/T434496) (owner: 10dcaro) [11:39:10] 10Tool-clip2commons: Add a Vitest unit test suite - https://phabricator.wikimedia.org/T434090#12246276 (10Dev) Test cases added - [[ https://gitlab.wikimedia.org/toolforge-repos/clip2commons/-/commit/9f2e3a0ed41858c26bc340bce7f9cd902251be85 | test: add Vitest unit suite, CI pipeline, and test documentation ]]... [11:39:22] 10Tool-clip2commons: Add a Vitest unit test suite - https://phabricator.wikimedia.org/T434090#12246278 (10Dev) 05Open→03Resolved [12:13:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:15:24] (03update) 10raymond-ndibe: tests: add some tests for runtime.build_start [repos/cloud/toolforge/components-api] (add_build_envvars) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/185 (https://phabricator.wikimedia.org/T434496) (owner: 10dcaro) [12:16:31] 10Toolforge, 06tools-platform-team: [lima-kilo] building locally jobs-api when a .tox env exists in the host code dir errors out - https://phabricator.wikimedia.org/T432276#12246433 (10Mahveotm) hi David, i've been able to reproduce this. `text $ docker buildx build --progress=plain --target image --file .pi... [12:18:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:26:35] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/42 [12:26:37] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/46 [12:28:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:30:45] 10Toolforge, 06tools-platform-team: [lima-kilo] building locally jobs-api when a .tox env exists in the host code dir errors out - https://phabricator.wikimedia.org/T432276#12246499 (10dcaro) > i think using a .dockerignore will be a better way to fix this safely. say, something like this, .tox/ withinn the .d... [12:31:38] 10Toolforge, 06tools-platform-team: [lima-kilo] building locally jobs-api when a .tox env exists in the host code dir errors out - https://phabricator.wikimedia.org/T432276#12246505 (10dcaro) p:05Triage→03Low [12:32:04] (03update) 10dcaro: global: move `image` inside tests [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/96 (https://phabricator.wikimedia.org/T435182) [12:38:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:41:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:45:46] (03open) 10dcaro: logs-api: the url changes `jobs-api` to `jobs` [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/104 (https://phabricator.wikimedia.org/T435798) [12:45:46] (03open) 10dcaro: api-gateway: change the logs-api url [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1387 (https://phabricator.wikimedia.org/T435798) [12:46:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:55:45] (03update) 10dcaro: write_jobs_logs: rename from `jobs-api` to `jobs` source [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/33 (https://phabricator.wikimedia.org/T435798) [12:56:17] (03update) 10dcaro: api-gateway: change the logs-api url [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1387 (https://phabricator.wikimedia.org/T435798) [12:56:34] (03update) 10dcaro: logs-api: the url changes `jobs-api` to `jobs` [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/104 (https://phabricator.wikimedia.org/T435798) [13:08:40] (03PS1) 10Atsuko: idp: provision the airflow_experiment_platform placeholder secret [labs/private] - 10https://gerrit.wikimedia.org/r/1328598 (https://phabricator.wikimedia.org/T416709) [13:09:26] 10Tool-wikimedia-attribution, 06MediaWiki-API-Platform-Team, 10MediaWiki-REST-API, 06MediaWiki-Core-Platform-Team (Kanban), and 2 others: Download CTA should be present only on Wikipedia projects - https://phabricator.wikimedia.org/T434214#12246659 (10pmiazga) 05Open→03Resolved [13:09:56] 10Tool-wikimedia-attribution, 06MediaWiki-API-Platform-Team, 10MediaWiki-REST-API, 06MediaWiki-Core-Platform-Team (Kanban), and 2 others: Download CTA should be present only on Wikipedia projects - https://phabricator.wikimedia.org/T434214#12246661 (10pmiazga) @Zaidusyy, thank you for submitting the pa... [13:09:58] (03CR) 10Atsuko: [V:03+2 C:03+2] idp: provision the airflow_experiment_platform placeholder secret [labs/private] - 10https://gerrit.wikimedia.org/r/1328598 (https://phabricator.wikimedia.org/T416709) (owner: 10Atsuko) [13:12:19] (03update) 10raymond-ndibe: tests: add some tests for runtime.build_start [repos/cloud/toolforge/components-api] (add_build_envvars) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/185 (https://phabricator.wikimedia.org/T434496) (owner: 10dcaro) [13:13:01] (03update) 10raymond-ndibe: api: add envvars to source build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/184 (https://phabricator.wikimedia.org/T434496) (owner: 10dcaro) [13:14:07] (03update) 10raymond-ndibe: support --webservice option [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T428898) [13:14:29] (03PS1) 10Brouberol: Add the postgresql-superset-metrics fixture password [labs/private] - 10https://gerrit.wikimedia.org/r/1328601 [13:14:39] (03CR) 10Brouberol: [C:03+2] Add the postgresql-superset-metrics fixture password [labs/private] - 10https://gerrit.wikimedia.org/r/1328601 (owner: 10Brouberol) [13:14:41] (03CR) 10Brouberol: [V:03+2 C:03+2] Add the postgresql-superset-metrics fixture password [labs/private] - 10https://gerrit.wikimedia.org/r/1328601 (owner: 10Brouberol) [13:20:55] (03update) 10dcaro: tests: add some tests for runtime.build_start [repos/cloud/toolforge/components-api] (add_build_envvars) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/185 (https://phabricator.wikimedia.org/T434496) [13:31:21] 10VPS-project-Wikistats: check current stable version automatically - https://phabricator.wikimedia.org/T111569#12246744 (10Meno25) [13:43:20] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [13:43:23] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [13:44:28] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [13:44:32] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [13:50:10] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [13:50:13] (03update) 10dcaro: Silly commit [toolforge-repos/wm-lol] - 10https://gitlab.wikimedia.org/toolforge-repos/wm-lol/-/merge_requests/1 [13:53:09] 10Cloud-VPS, 06tools-infrastructure-team: Improve VM backup performance - https://phabricator.wikimedia.org/T434014#12246840 (10Andrew) 05Open→03Resolved Short of adding a third backup server I don't have good ideas about how to speed this up further. [13:53:20] 10Cloud-VPS, 06tools-infrastructure-team: Rebalance WMCS instance backups between the two hosts - https://phabricator.wikimedia.org/T430018#12246843 (10Andrew) 05Open→03Resolved [13:54:39] (03PS1) 10Atsuko: Revert "idp: provision the airflow_experiment_platform placeholder secret" [labs/private] - 10https://gerrit.wikimedia.org/r/1328611 [13:54:48] (03CR) 10Atsuko: [V:03+2 C:03+2] Revert "idp: provision the airflow_experiment_platform placeholder secret" [labs/private] - 10https://gerrit.wikimedia.org/r/1328611 (owner: 10Atsuko) [13:55:19] (03PS1) 10Atsuko: Re-do "idp: provision the airflow_experiment_platform placeholder secret" [labs/private] - 10https://gerrit.wikimedia.org/r/1328612 [13:56:08] (03PS2) 10Atsuko: Re-do "idp: provision the airflow_experiment_platform placeholder secret" [labs/private] - 10https://gerrit.wikimedia.org/r/1328612 [14:08:10] (03CR) 10Atsuko: [V:03+2 C:03+2] Re-do "idp: provision the airflow_experiment_platform placeholder secret" [labs/private] - 10https://gerrit.wikimedia.org/r/1328612 (owner: 10Atsuko) [14:25:28] 10Tool-bash, 10Tool-sal, 10Stashbot: Migrate storage to new OpenSearch v2 Toolforge cluster - https://phabricator.wikimedia.org/T435223#12246970 (10bd808) Thank you for taking care of this for me @taavi. I'm sorry that I was so unresponsive that you needed to do the work for me, and I am grateful that yo... [14:40:31] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Nairobi 29): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12247078 (10TLepage-WMF) Hello! > Is any thr... [14:40:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:41:34] 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Nairobi 29): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12247079 (10taavi) [14:42:17] (03update) 10raymond-ndibe: jobs: Add timeout support for one-off jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/373 (https://phabricator.wikimedia.org/T401110) (owner: 10mahveotm) [14:43:35] (03update) 10raymond-ndibe: logs-api: the url changes `jobs-api` to `jobs` [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/104 (https://phabricator.wikimedia.org/T435798) (owner: 10dcaro) [14:43:36] (03approved) 10raymond-ndibe: logs-api: the url changes `jobs-api` to `jobs` [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/104 (https://phabricator.wikimedia.org/T435798) (owner: 10dcaro) [14:43:55] 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Nairobi 29): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12247099 (10taavi) p:05Low→03Triage [14:44:00] (03update) 10raymond-ndibe: write_jobs_logs: rename from `jobs-api` to `jobs` source [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/33 (https://phabricator.wikimedia.org/T435798) (owner: 10dcaro) [14:44:02] (03approved) 10raymond-ndibe: write_jobs_logs: rename from `jobs-api` to `jobs` source [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/33 (https://phabricator.wikimedia.org/T435798) (owner: 10dcaro) [14:44:24] (03update) 10raymond-ndibe: tests: add some tests for runtime.build_start [repos/cloud/toolforge/components-api] (add_build_envvars) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/185 (https://phabricator.wikimedia.org/T434496) (owner: 10dcaro) [14:44:27] (03update) 10raymond-ndibe: tests: add some tests for runtime.build_start [repos/cloud/toolforge/components-api] (add_build_envvars) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/185 (https://phabricator.wikimedia.org/T434496) (owner: 10dcaro) [14:44:32] (03approved) 10raymond-ndibe: tests: add some tests for runtime.build_start [repos/cloud/toolforge/components-api] (add_build_envvars) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/185 (https://phabricator.wikimedia.org/T434496) (owner: 10dcaro) [14:45:13] (03update) 10raymond-ndibe: api-gateway: change the logs-api url [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1387 (https://phabricator.wikimedia.org/T435798) (owner: 10dcaro) [14:45:16] (03approved) 10raymond-ndibe: api-gateway: change the logs-api url [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1387 (https://phabricator.wikimedia.org/T435798) (owner: 10dcaro) [14:45:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:46:12] (03update) 10raymond-ndibe: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) (owner: 10mahveotm) [14:47:57] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Doing 😎): Planning for a plan to migrate Beta Cluster again - https://phabricator.wikimedia.org/T433998#12247125 (10bd808) @Southparkfan contacted me yesterday with some questions and a rough outline of how... [14:48:53] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10Puppet-Core: Normalise hiera default values - https://phabricator.wikimedia.org/T289665#12247148 (10LSobanski) p:05Triage→03Low [14:50:58] (03update) 10raymond-ndibe: cli: add timeout support for one-off jobs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/176 (https://phabricator.wikimedia.org/T401110) (owner: 10mahveotm) [14:54:11] 10Cloud Services Proposals, 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, and 3 others: Easing pain points caused by divergence between cloudservices and production puppet usecases - https://phabricator.wikimedia.org/T285539#12247225 (10LSobanski) p:05Triage→03Low [15:00:36] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [15:00:40] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [15:01:30] 10Toolforge, 06tools-platform-team: [lima-kilo] building locally jobs-api when a .tox env exists in the host code dir errors out - https://phabricator.wikimedia.org/T432276#12247307 (10dcaro) > i don’t have a strong opinion on this. my preference was to avoid changing the mount behavior unless necessary. .tox... [15:04:11] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [15:06:03] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance cloudinfra-cloudvps-puppetserver-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [15:07:57] PROBLEM - Host cloudcephosd1041 is DOWN: PING CRITICAL - Packet loss = 100% [15:10:00] 10Toolforge, 06tools-platform-team: [toolsdb] 2026-08-18 Transaction History Length growing too much - https://phabricator.wikimedia.org/T435220#12247418 (10fnegri) @magnusmanske thank you! Yes things are a bit better, the history length is still growing, but it seems to be growing slower, let's see after a fe... [15:10:27] RECOVERY - Host cloudcephosd1041 is UP: PING OK - Packet loss = 0%, RTA = 0.44 ms [15:14:33] (03update) 10renovatebot: Update pnpm to v11.23.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/14 [15:14:37] (03open) 10renovatebot: Update pnpm to v11.23.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/14 [15:14:51] (03update) 10renovatebot: Update pnpm to v11.23.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/26 [15:14:55] (03open) 10renovatebot: Update pnpm to v11.23.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/26 [15:15:11] (03update) 10renovatebot: Update pnpm to v11.23.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/21 [15:15:17] (03open) 10renovatebot: Update pnpm to v11.23.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/21 [15:16:18] (03update) 10renovatebot: Update pnpm to v11.23.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/53 [15:16:23] (03open) 10renovatebot: Update pnpm to v11.23.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/53 [15:17:26] (03update) 10renovatebot: Update pnpm to v11.23.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/38 [15:17:31] (03open) 10renovatebot: Update pnpm to v11.23.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/38 [15:17:35] 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Nairobi 29): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12247469 (10AMarkossyan-WMF) a:03Mhurd [15:18:48] FIRING: PuppetDisabled: Puppet disabled on cloudcephosd2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [15:25:39] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [15:28:10] 10Tool-refill: Failed verification template incorrectly added to URL - https://phabricator.wikimedia.org/T435842 (10GoingBatty) 03NEW [15:33:51] 10Toolforge, 06tools-platform-team, 07OKR-Work: [jobs-api] Send webservice start/stop/restart events to the new logs-api event log endpoint - https://phabricator.wikimedia.org/T432570#12247594 (10dcaro) [15:35:07] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1080.eqiad.wmnet' (T435537) [15:35:13] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [15:41:32] (03PS4) 10Krinkle: build: Add type hints, simplify jsonpCallback logic, add `composer serve` [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328374 [15:41:39] (03PS3) 10Krinkle: api.php: Implement a new `recent=1` feature to return last 30 rows [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328375 [15:45:11] 10Toolforge, 06tools-platform-team, 07OKR-Work: [jobs-api] Send webservice start/stop/restart events to the new logs-api event log endpoint - https://phabricator.wikimedia.org/T432570#12247647 (10dcaro) [15:51:12] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 10Ceph, and 3 others: cloudcephosd1044 boot issues - https://phabricator.wikimedia.org/T429267#12247674 (10Andrew) Just checking -- is this ticket in my court or yours? [15:56:47] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] add job publish support - https://phabricator.wikimedia.org/T433355#12247714 (10Raymond_Ndibe) 05In progress→03Resolved [15:57:02] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-api] refactor models in preparation for webservice job-type - https://phabricator.wikimedia.org/T434166#12247719 (10Raymond_Ndibe) 05Open→03Resolved [15:57:48] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [toollabs-images] stop hardcoding port 8000 in the webservice-runner scripts - https://phabricator.wikimedia.org/T432078#12247723 (10Raymond_Ndibe) 05Open→03In progress [15:59:14] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1080.eqiad.wmnet' (T435537) [15:59:21] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [16:00:40] 10Tool-wikinewsie: Deaths story - https://phabricator.wikimedia.org/T435844 (10Mollystarkdean) 03NEW p:05Triage→03Low [16:01:01] 10Tool-wikinewsie: Placeholder Photos? - https://phabricator.wikimedia.org/T435845 (10Mollystarkdean) 03NEW [16:01:10] 10Tool-wikinewsie: Placeholder Photos? - https://phabricator.wikimedia.org/T435845#12247772 (10Mollystarkdean) [16:14:48] 10Toolforge, 06tools-platform-team: [builds-cli] remove wrong help message for `--ref` - https://phabricator.wikimedia.org/T435793#12247879 (10dcaro) a:03Sebastian_Berlin-WMSE [16:15:57] 10Toolforge, 06tools-platform-team, 07OKR-Work: Email tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865#12247901 (10fnegri) 05Open→03In progress [16:20:09] 10Tool-wikilinkchecker: Link for https://wikilinkchecker.toolforge.org/ is broken on toolhub.wikimedia.org - https://phabricator.wikimedia.org/T435752#12247935 (10bd808) The 404 is legitimate. There is no webservice running for the https://toolsadmin.wikimedia.org/tools/id/linkchecker tool. There is no sign in... [16:25:01] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T435537) [16:25:04] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T435537) [16:25:07] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [16:25:44] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate False, for hosts list: ['cloudvirt1040', 'cloudvirt1044', 'cloudvirt1042', 'cloudvirt1045', 'cloudvirt1043', 'cloudvirt1046', 'cloudvirt1041', 'cloudvirt1047', 'cloudvirt1048', 'cloudvirt1049', 'cloudvirt1052', 'cloudvirt1053', 'cloudvirt1050', 'cloudvirt1051', 'cloudvirt1057', 'cloudvirt1056', [16:25:45] 'cloudvirt1061', 'cloudvirt1058', 'cloudvirt1060', 'cloudvirt1059', 'cloudvirt1055', 'cloudvirt1054', 'cloudvirtlocal1001', 'cloudvirtlocal1002', 'cloudvirtlocal1003', 'cloudvirt1062', 'cloudvirt1066', 'cloudvirt1065', 'cloudvirt1067', 'cloudvirt1063', 'cloudvirt1064', 'cloudvirt1071', 'cloudvirt1068', 'cloudvirt1069', 'cloudvirt1070', 'cloudvirt1076', 'cloudvirt1075', 'cloudvirt1072', 'cloudvirt1073', 'cloudvirt1074', 'c [16:25:45] loudvirt1077', 'cloudvirt1078', 'cloudvirt1079', 'cloudvirt1080'] [16:25:56] !log andrew@cloudcumin1001 cloudvirt-canary END (ERROR) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=97) on eqiad1, with recreate False, for hosts list: ['cloudvirt1040', 'cloudvirt1044', 'cloudvirt1042', 'cloudvirt1045', 'cloudvirt1043', 'cloudvirt1046', 'cloudvirt1041', 'cloudvirt1047', 'cloudvirt1048', 'cloudvirt1049', 'cloudvirt1052', 'cloudvirt1053', 'cloudvirt1050', 'cloudvirt1051', 'cloudvirt105 [16:25:56] 7', 'cloudvirt1056', 'cloudvirt1061', 'cloudvirt1058', 'cloudvirt1060', 'cloudvirt1059', 'cloudvirt1055', 'cloudvirt1054', 'cloudvirtlocal1001', 'cloudvirtlocal1002', 'cloudvirtlocal1003', 'cloudvirt1062', 'cloudvirt1066', 'cloudvirt1065', 'cloudvirt1067', 'cloudvirt1063', 'cloudvirt1064', 'cloudvirt1071', 'cloudvirt1068', 'cloudvirt1069', 'cloudvirt1070', 'cloudvirt1076', 'cloudvirt1075', 'cloudvirt1072', 'cloudvirt1073' [16:25:56] , 'cloudvirt1074', 'cloudvirt1077', 'cloudvirt1078', 'cloudvirt1079', 'cloudvirt1080'] [16:26:23] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate False, for hosts list: ['cloudvirt1040', 'cloudvirt1044', 'cloudvirt1042', 'cloudvirt1045', 'cloudvirt1043', 'cloudvirt1046', 'cloudvirt1041', 'cloudvirt1047', 'cloudvirt1048', 'cloudvirt1049', 'cloudvirt1052', 'cloudvirt1053', 'cloudvirt1050', 'cloudvirt1051', 'cloudvirt1057', 'cloudvirt1056', [16:26:23] 'cloudvirt1061', 'cloudvirt1058', 'cloudvirt1060', 'cloudvirt1059', 'cloudvirt1055', 'cloudvirt1054', 'cloudvirtlocal1001', 'cloudvirtlocal1002', 'cloudvirtlocal1003', 'cloudvirt1062', 'cloudvirt1066', 'cloudvirt1065', 'cloudvirt1067', 'cloudvirt1063', 'cloudvirt1064', 'cloudvirt1071', 'cloudvirt1068', 'cloudvirt1069', 'cloudvirt1070', 'cloudvirt1076', 'cloudvirt1075', 'cloudvirt1072', 'cloudvirt1073', 'cloudvirt1074', 'c [16:26:23] loudvirt1077', 'cloudvirt1078', 'cloudvirt1079', 'cloudvirt1080'] [16:28:37] 10Cloud-Services, 06Abstract Wikipedia team, 10Wikifunctions, 07Essential-Work: Please close WMCS project "wikifunctions" - https://phabricator.wikimedia.org/T435848 (10Jdforrester-WMF) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.w... [16:29:08] 10Cloud-VPS (Project-requests), 06Abstract Wikipedia team, 10Wikifunctions, 07Essential-Work: Please close WMCS project "wikifunctions" - https://phabricator.wikimedia.org/T435848#12247999 (10Jdforrester-WMF) [16:31:35] (03open) 10dcaro: write_job_logs: add jobs events [repos/cloud/toolforge/logs-api] (rename_jobs_source) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/34 (https://phabricator.wikimedia.org/T432570) [16:31:50] !log andrew@cloudcumin1001 cloudvirt-canary END (FAIL) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=99) on eqiad1, with recreate False, for hosts list: ['cloudvirt1040', 'cloudvirt1044', 'cloudvirt1042', 'cloudvirt1045', 'cloudvirt1043', 'cloudvirt1046', 'cloudvirt1041', 'cloudvirt1047', 'cloudvirt1048', 'cloudvirt1049', 'cloudvirt1052', 'cloudvirt1053', 'cloudvirt1050', 'cloudvirt1051', 'cloudvirt1057 [16:31:50] ', 'cloudvirt1056', 'cloudvirt1061', 'cloudvirt1058', 'cloudvirt1060', 'cloudvirt1059', 'cloudvirt1055', 'cloudvirt1054', 'cloudvirtlocal1001', 'cloudvirtlocal1002', 'cloudvirtlocal1003', 'cloudvirt1062', 'cloudvirt1066', 'cloudvirt1065', 'cloudvirt1067', 'cloudvirt1063', 'cloudvirt1064', 'cloudvirt1071', 'cloudvirt1068', 'cloudvirt1069', 'cloudvirt1070', 'cloudvirt1076', 'cloudvirt1075', 'cloudvirt1072', 'cloudvirt1073', [16:31:50] 'cloudvirt1074', 'cloudvirt1077', 'cloudvirt1078', 'cloudvirt1079', 'cloudvirt1080'] [16:32:33] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate False, for hosts list: ['cloudvirt1080'] [16:32:56] !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate False, for hosts list: ['cloudvirt1080'] [16:33:13] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1079.eqiad.wmnet' (T435537) [16:33:17] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [16:54:04] 10Tool-wikinewsie: Wikinewsie footer elements misaligned on mobile view - https://phabricator.wikimedia.org/T435018#12248147 (10Akash_A) Hello there, I tried reproducing this issue and wasn´t successful and I think I am missing something here. By referring to Wikinewsie, we are referring to the currently closed... [17:00:18] RESOLVED: PuppetDisabled: Puppet disabled on cloudcephosd2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [17:00:39] 10Cloud-VPS (Project-requests), 06Abstract Wikipedia team, 10Wikifunctions, 07Essential-Work: Please close WMCS project "wikifunctions" - https://phabricator.wikimedia.org/T435848#12248166 (10taavi) a:03taavi [17:01:21] !log taavi@cloudcumin1001 wikifunctions START - Cookbook wmcs.vps.delete_project for project wikifunctions in eqiad1 [17:01:25] !log taavi@cloudcumin1001 wikifunctions END (FAIL) - Cookbook wmcs.vps.delete_project (exit_code=99) for project wikifunctions in eqiad1 [17:03:50] !log taavi@cloudcumin1001 wikifunctions START - Cookbook wmcs.vps.delete_project for project wikifunctions in eqiad1 [17:04:10] (03update) 10dcaro: write_job_logs: add jobs events [repos/cloud/toolforge/logs-api] (rename_jobs_source) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/34 (https://phabricator.wikimedia.org/T432570) [17:04:15] (03update) 10dcaro: write_job_logs: add jobs events [repos/cloud/toolforge/logs-api] (rename_jobs_source) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/34 (https://phabricator.wikimedia.org/T432570) [17:04:21] (03update) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: delete project wikifunctions [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/345 [17:04:23] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: delete project wikifunctions [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/345 [17:05:09] (03merge) 10taavi: projects: delete project wikifunctions [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/345 (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [17:05:38] !log taavi@cloudcumin1001 wikifunctions END (PASS) - Cookbook wmcs.vps.delete_project (exit_code=0) for project wikifunctions in eqiad1 [17:06:50] 10Cloud-VPS (Project-requests), 06Abstract Wikipedia team, 10Wikifunctions, 07Essential-Work: Please close WMCS project "wikifunctions" - https://phabricator.wikimedia.org/T435848#12248182 (10taavi) 05Open→03Resolved [17:07:39] (03merge) 10bd808: Update template for the 2024.04 builder stack and Python 3.14 runtime [toolforge-repos/containers-cookiecutter] - 10https://gitlab.wikimedia.org/toolforge-repos/containers-cookiecutter/-/merge_requests/1 [17:10:53] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1079.eqiad.wmnet' (T435537) [17:11:02] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [17:16:07] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate False, for hosts list: ['cloudvirt1079'] [17:16:19] !log andrew@cloudcumin1001 cloudvirt-canary END (FAIL) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=99) on eqiad1, with recreate False, for hosts list: ['cloudvirt1079'] [17:16:37] (03open) 10dcaro: refactor: made loki a package, and minor refactors [repos/cloud/toolforge/logs-api] (rename_jobs_source) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/35 (https://phabricator.wikimedia.org/T432570) [17:17:09] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate False, for hosts list: ['cloudvirt1040', 'cloudvirt1044', 'cloudvirt1042', 'cloudvirt1045', 'cloudvirt1043', 'cloudvirt1046', 'cloudvirt1041', 'cloudvirt1047', 'cloudvirt1048', 'cloudvirt1049', 'cloudvirt1052', 'cloudvirt1053', 'cloudvirt1050', 'cloudvirt1051', 'cloudvirt1057', 'cloudvirt1056', [17:17:09] 'cloudvirt1061', 'cloudvirt1058', 'cloudvirt1060', 'cloudvirt1059', 'cloudvirt1055', 'cloudvirt1054', 'cloudvirtlocal1001', 'cloudvirtlocal1002', 'cloudvirtlocal1003', 'cloudvirt1062', 'cloudvirt1066', 'cloudvirt1065', 'cloudvirt1067', 'cloudvirt1063', 'cloudvirt1064', 'cloudvirt1071', 'cloudvirt1068', 'cloudvirt1069', 'cloudvirt1070', 'cloudvirt1076', 'cloudvirt1075', 'cloudvirt1072', 'cloudvirt1073', 'cloudvirt1074', 'c [17:17:09] loudvirt1077', 'cloudvirt1078', 'cloudvirt1079', 'cloudvirt1080'] [17:17:21] !log andrew@cloudcumin1001 cloudvirt-canary END (FAIL) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=99) on eqiad1, with recreate False, for hosts list: ['cloudvirt1040', 'cloudvirt1044', 'cloudvirt1042', 'cloudvirt1045', 'cloudvirt1043', 'cloudvirt1046', 'cloudvirt1041', 'cloudvirt1047', 'cloudvirt1048', 'cloudvirt1049', 'cloudvirt1052', 'cloudvirt1053', 'cloudvirt1050', 'cloudvirt1051', 'cloudvirt1057 [17:17:21] ', 'cloudvirt1056', 'cloudvirt1061', 'cloudvirt1058', 'cloudvirt1060', 'cloudvirt1059', 'cloudvirt1055', 'cloudvirt1054', 'cloudvirtlocal1001', 'cloudvirtlocal1002', 'cloudvirtlocal1003', 'cloudvirt1062', 'cloudvirt1066', 'cloudvirt1065', 'cloudvirt1067', 'cloudvirt1063', 'cloudvirt1064', 'cloudvirt1071', 'cloudvirt1068', 'cloudvirt1069', 'cloudvirt1070', 'cloudvirt1076', 'cloudvirt1075', 'cloudvirt1072', 'cloudvirt1073', [17:17:21] 'cloudvirt1074', 'cloudvirt1077', 'cloudvirt1078', 'cloudvirt1079', 'cloudvirt1080'] [17:17:44] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T435537) [17:17:47] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T435537) [17:17:48] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [17:18:04] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate False, for hosts list: ['cloudvirt1079'] [17:18:32] !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate False, for hosts list: ['cloudvirt1079'] [17:18:41] (03update) 10dcaro: refactor: made loki a package, and minor refactors [repos/cloud/toolforge/logs-api] (rename_jobs_source) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/35 (https://phabricator.wikimedia.org/T432570) [17:18:58] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1078.eqiad.wmnet' (T435537) [17:22:25] (03update) 10dcaro: write_job_logs: add jobs events [repos/cloud/toolforge/logs-api] (rename_jobs_source) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/34 (https://phabricator.wikimedia.org/T432570) [17:22:47] (03update) 10dcaro: write_job_logs: add jobs events [repos/cloud/toolforge/logs-api] (minor_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/34 (https://phabricator.wikimedia.org/T432570) [17:26:13] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate reading-web-staging away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435780#12248272 (10Jdlrobson-WMF) a:03Jdlrobson-WMF I am handling this migration. I think the essential parts are done. I just haven't worked out whether I n... [17:26:20] 10Cloud-VPS (Debian Bullseye Deprecation), 03Readers Essential Work: Migrate reading-web-staging away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435780#12248276 (10Jdlrobson-WMF) [17:26:27] 10Cloud-VPS (Debian Bullseye Deprecation), 03Readers Essential Work: Migrate reading-web-staging away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435780#12248277 (10Jdlrobson-WMF) [17:27:43] 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Nairobi 29): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12248281 (10taavi) @AMarkossyan-WMF This task is tagged with #cloud-vp... [17:28:10] (03close) 10dcaro: deployment: drop PSP [repos/cloud/toolforge/builds-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-admission/-/merge_requests/8 (https://phabricator.wikimedia.org/T368142) (owner: 10aborrero) [17:29:04] (03close) 10dcaro: [dev] fix CONTRIBUTING.md [repos/cloud/toolforge/buildpacks/go-toml] (v2) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/go-toml/-/merge_requests/1 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [17:31:55] 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Nairobi 29): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12248305 (10AMarkossyan-WMF) @taavi Sorry, I assigned this to Monte to... [17:32:04] FIRING: TargetDown: Job manager is unreachable in project metricsinfra instance metricsinfra-controller-3 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [17:32:05] 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Nairobi 29): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12248306 (10AMarkossyan-WMF) a:05Mhurd→03None [17:37:04] RESOLVED: TargetDown: Job manager is unreachable in project metricsinfra instance metricsinfra-controller-3 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [17:51:18] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Add IPv6 PTR record for mail.wikimedia.az - https://phabricator.wikimedia.org/T432761#12248378 (10Andrew) I replaced the existing record (pointing to mail.azwikimedia.eqiad1.wikimedia.cloud.) to point to mail.wikimedia.az.... [17:52:22] 10Tool-wikinewsie: Wikinewsie footer elements misaligned on mobile view - https://phabricator.wikimedia.org/T435018#12248383 (10Pharos) No, we are referring to the new tool hosted at https://wikinewsie.toolforge.org/ [17:53:46] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1078.eqiad.wmnet' (T435537) [17:53:52] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [17:55:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:00:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:04:10] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T435537) [18:04:13] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T435537) [18:04:16] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [18:04:24] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate False, for hosts list: ['cloudvirt1078'] [18:04:52] !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate False, for hosts list: ['cloudvirt1078'] [18:05:03] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1077.eqiad.wmnet' (T435537) [18:24:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:29:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:32:32] 10Tools: phabulous: Priority field unusable - https://phabricator.wikimedia.org/T435825#12248557 (10Aklapper) Indeed, Phabulous creates a link to the default form `https://phabricator.wikimedia.org/maniphest/task/edit/form/1` which has had the `Priority` field hidden for the last years. The Priority field was `l... [18:32:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:33:11] 10Tools: phabulous: Priority field not taken into account as Phabricator Form 1 hides the Priority field - https://phabricator.wikimedia.org/T435825#12248559 (10Aklapper) p:05Triage→03Low [18:34:06] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1077.eqiad.wmnet' (T435537) [18:34:12] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [18:37:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:43:00] (03PS5) 10Krinkle: build: Add type hints, simplify jsonpCallback logic, add `composer serve` [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328374 [18:43:00] (03PS4) 10Krinkle: api.php: Implement a new `recent=1` feature to return last 30 rows [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328375 [18:43:21] (03CR) 10CI reject: [V:04-1] api.php: Implement a new `recent=1` feature to return last 30 rows [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328375 (owner: 10Krinkle) [18:43:30] (03CR) 10CI reject: [V:04-1] build: Add type hints, simplify jsonpCallback logic, add `composer serve` [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328374 (owner: 10Krinkle) [18:44:26] (03PS6) 10Krinkle: build: Add type hints, simplify jsonpCallback logic, add `composer serve` [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328374 [18:44:27] (03PS5) 10Krinkle: api.php: Implement a new `recent=1` feature to return last 30 rows [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328375 [18:46:06] (03CR) 10Krinkle: [C:03+2] build: Add type hints, simplify jsonpCallback logic, add `composer serve` [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328374 (owner: 10Krinkle) [18:46:40] (03CR) 10Krinkle: [C:03+2] api.php: Implement a new `recent=1` feature to return last 30 rows [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328375 (owner: 10Krinkle) [18:46:41] (03Merged) 10jenkins-bot: build: Add type hints, simplify jsonpCallback logic, add `composer serve` [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328374 (owner: 10Krinkle) [18:47:13] (03Merged) 10jenkins-bot: api.php: Implement a new `recent=1` feature to return last 30 rows [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1328375 (owner: 10Krinkle) [18:52:28] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T435537) [18:52:31] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T435537) [18:52:36] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [18:52:41] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate False, for hosts list: ['cloudvirt1077'] [18:53:04] !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate False, for hosts list: ['cloudvirt1077'] [19:00:45] (03close) 10andrew: Add v6 PTR for mail.wikimedia.az [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/343 (https://phabricator.wikimedia.org/T432761) [19:00:49] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Add IPv6 PTR record for mail.wikimedia.az - https://phabricator.wikimedia.org/T432761#12248627 (10Andrew) 05Open→03Resolved [19:11:56] 10Tool-wikinewsie: Wikinewsie footer elements misaligned on mobile view - https://phabricator.wikimedia.org/T435018#12248658 (10Akash_A) Got it, thank you. I am interested in working on this, could you provide the repo link please? [19:14:54] 10Tool-wikinewsie: Wikinewsie footer elements misaligned on mobile view - https://phabricator.wikimedia.org/T435018#12248665 (10Pharos) >>! In T435018#12248658, @Akash_A wrote: > Got it, thank you. I am interested in working on this, could you provide the repo link please? Thanks, the repo is at https://gitlab.... [19:25:36] 10Tool-wikinewsie: Wikinewsie footer elements misaligned on mobile view - https://phabricator.wikimedia.org/T435018#12248689 (10Akash_A) a:03Akash_A Thanks, I shall work on this particular bug then. Claiming this. [19:43:59] (03PS21) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [19:44:26] (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [19:56:00] (03open) 10ecoholic: T435018: Wikinewsie footer elements misaligned on mobile view [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/7 [19:57:42] 10Tool-wikinewsie: Wikinewsie footer elements misaligned on mobile view - https://phabricator.wikimedia.org/T435018#12248893 (10Akash_A) Have opened a PR, please do review and let me know if that does it. [20:06:06] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 07Epic, 06Release-Engineering-Team (Priority Backlog 📥): Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12248928 (10bd808) [20:47:33] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance cloudinfra-cloudvps-puppetserver-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [21:50:39] 10Toolforge, 06tools-infrastructure-team: [disable-tools] Disabled Toolforge tool deletions not occurring - https://phabricator.wikimedia.org/T431241#12249229 (10bd808) https://toolsadmin.wikimedia.org/tools/id/vote-tracker is still stuck on something. `lang=shell-session bd808@tools-nfs-3:~$ sudo journalctl -... [22:01:19] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [22:01:22] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [22:01:41] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [22:01:46] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [22:02:27] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node [22:08:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [22:34:53] 10Tools, 06Commons, 07Community-Wishlist: Image Editor for Commons - https://phabricator.wikimedia.org/T426106#12249475 (10MusikAnimal) 05Duplicate→03Resolved [22:41:03] 10Tools, 06Commons, 07Community-Wishlist: Image Editor for Commons - https://phabricator.wikimedia.org/T426106#12249476 (10JJMC89) →14Duplicate dup:03T428062 [22:41:04] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12249477 (10JJMC89) [22:43:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [23:00:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:05:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:42:57] 10Toolforge, 06tools-platform-team: [builds-api] TOOL_DATA_DIR not available during `toolforge build start` execution - https://phabricator.wikimedia.org/T434369#12249631 (10bd808) >>! In T434369#12219331, @Gouvernathor wrote: > I'll note however that considering that only the run time, and not the build time,... [23:57:31] 10Striker, 06tools-platform-team: Migrate existing Striker created Diffusion repos to GitLab - https://phabricator.wikimedia.org/T315706#12249646 (10bd808) >>! In T315706#12218842, @PacmanD wrote: > Hi, I was trying to locate the source code for https://toolsadmin.wikimedia.org/tools/id/ws-google-ocr -- it...