[00:09:40] (03update) 10swfrench: kubernetes.py: Rename rollout_is_progressing to rollout_was_abandoned [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1267 (https://phabricator.wikimedia.org/T375514) (owner: 10dancy) [00:12:30] (03approved) 10swfrench: kubernetes.py: Rename rollout_is_progressing to rollout_was_abandoned [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1267 (https://phabricator.wikimedia.org/T375514) (owner: 10dancy) [02:24:46] (03approved) 10ejegg: fundraising: Drop libc-client-dev from the bookworm PHP 8.2 image [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/119 (owner: 10jforrester) [04:57:25] (03PS1) 10Pwangai: docker: Base the dev image on bookworm [integration/quibble] - 10https://gerrit.wikimedia.org/r/1326974 (https://phabricator.wikimedia.org/T435275) [05:45:21] 10GitLab (Project and group requests), 06Release-Engineering-Team, 07Essential-Work: GitLab Private Repository Request for: CSP Research - https://phabricator.wikimedia.org/T420485#12229554 (10brennen) 05Stalledβ†’03Declined Please reopen if this is still needed. [05:48:33] 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for gaiarajan - https://phabricator.wikimedia.org/T434626#12229561 (10brennen) 05Openβ†’03Resolved a:03brennen [05:50:03] 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for gaiarajan - https://phabricator.wikimedia.org/T434626#12229568 (10brennen) User appears human and has an interest in a project hosted on GitLab. Not adding to Trusted-Contributors since I d... [06:12:10] (03open) 10phedenskog: Make the retry dashboard actionable [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/24 [06:12:19] (03merge) 10phedenskog: Make the retry dashboard actionable [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/24 [06:47:53] (03open) 10phedenskog: Record which repos and branches flaky tests flake on [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/25 [06:48:23] (03approved) 10phedenskog: Record which repos and branches flaky tests flake on [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/25 [06:48:26] (03merge) 10phedenskog: Record which repos and branches flaky tests flake on [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/25 [07:02:55] (03open) 10phedenskog: Turn the browser tests page into a flakiness dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/26 [07:54:36] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12229790 (10ABran-WMF) [08:11:39] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12229818 (10ABran-WMF) [09:01:34] 10Gerrit: ?format=TEXT does no longer give raw files from gitiles - https://phabricator.wikimedia.org/T435289 (10Physikerwelt) 03NEW [09:05:13] 10GitLab, 06Collaboration-Services: GitLab fails to download artifacts between jobs - https://phabricator.wikimedia.org/T435290 (10Don-vip) 03NEW [09:05:57] 10GitLab, 06Collaboration-Services: GitLab fails to download artifacts between jobs - https://phabricator.wikimedia.org/T435290#12229962 (10ABran-WMF) [09:06:04] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12229963 (10ABran-WMF) [09:08:54] 10GitLab, 06Collaboration-Services: GitLab fails to download artifacts between jobs - https://phabricator.wikimedia.org/T435290#12229981 (10ABran-WMF) thanks for mentioning it @Don-vip, we're currently working on debugging the issue. If it lasts for too long, we'll revert the move behind CDN. [09:20:21] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12230010 (10BTullis) [09:20:35] 10GitLab, 06Collaboration-Services: GitLab fails to download artifacts between jobs - https://phabricator.wikimedia.org/T435290#12230011 (10Don-vip) OK, thank you! [09:23:31] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12230018 (10BTullis) Hi, just to let you know, we're experiencing some instability with our gitsync pods on our [[https://wikitech.wikimedia.org/wiki/Dat... [09:31:24] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12230054 (10ABran-WMF) >>! In T425441#12230018, @BTullis wrote: > Hi, just to let you know, we're experiencing some instability with our gitsync pods on... [09:32:32] 10GitLab, 06Collaboration-Services: GitLab fails to download artifacts between jobs - https://phabricator.wikimedia.org/T435290#12230060 (10ABran-WMF) Highlighting my answer in the CDN ticket: >>! In T425441#12230054, @ABran-WMF wrote: > > We reverted the change moving Gitlab behind CDN because of similar is... [09:47:54] 10GitLab, 06Collaboration-Services: GitLab fails to download artifacts between jobs - https://phabricator.wikimedia.org/T435290#12230101 (10Don-vip) It's up and running again, thanks a lot! https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/pipelines/219518 [10:17:41] 10Gerrit, 06Collaboration-Services, 10MathSearch: MathSearch can no longer download individual files from gitiles (403) - https://phabricator.wikimedia.org/T435289#12230277 (10hashar) 05Openβ†’03Resolved a:03hashar https://portal.mardi4nfdi.de/wiki/Special:MathDebug?action=visualDiff&base=refs/change... [10:21:25] (03CR) 10Hashar: [C:03+1] "`--break-system-packages` is due to a change in the Python package provided by Debian Bookworm. We do the same for the quibble images." [integration/quibble] - 10https://gerrit.wikimedia.org/r/1326974 (https://phabricator.wikimedia.org/T435275) (owner: 10Pwangai) [10:21:40] 10Gerrit, 06Collaboration-Services, 10MathSearch: MathSearch can no longer download individual files from gitiles (403) - https://phabricator.wikimedia.org/T435289#12230296 (10Aklapper) 05Resolvedβ†’03Invalid Marking as invalid as this is likely expected behavior. [10:21:45] (03CR) 10Hashar: [C:03+2] Zuul: [BlueSpiceWikiFarm] Add StandardDialogs dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1326889 (owner: 10Hslater) [10:23:35] (03Merged) 10jenkins-bot: Zuul: [BlueSpiceWikiFarm] Add StandardDialogs dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1326889 (owner: 10Hslater) [10:28:15] 06Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T430835#12230307 (10Aklapper) Status after deploying to group1: T435167 is noisy so asked for a backport in that ticket; filed an Echo DB issue about... [10:43:52] 10GitLab, 06Collaboration-Services: GitLab fails to download artifacts between jobs - https://phabricator.wikimedia.org/T435290#12230401 (10LSobanski) p:05Triageβ†’03Medium a:03ABran-WMF [10:59:04] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [11:04:00] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#12230527 (10mfossati) Thank you @Aklapper πŸ™‚ . [11:36:10] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Improve situation with K3s cluster's cred/passwd error - https://phabricator.wikimedia.org/T434781#12230679 (10jnuche) After understanding the issue better I went back to https://github.com/k3s-io/k3s/issues and fou... [11:41:03] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:07:13] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Add new nodes to Catalyst cluster for code-server - https://phabricator.wikimedia.org/T432751#12230802 (10jnuche) a:03jnuche [12:07:45] 10GitLab (CI & Job Runners), 06Collaboration-Services: Puppet CA certificate for WMCS puppet server gitlab-runners-puppetserver-01 is about to expire - https://phabricator.wikimedia.org/T433581#12230807 (10LSobanski) @Andrew now that I'm looking at the alerts, they're for //integration// and //cloudinfra// pro... [12:16:26] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Add new nodes to Catalyst cluster for code-server - https://phabricator.wikimedia.org/T432751#12230837 (10jnuche) [12:23:59] (03update) 10jforrester: fundraising: Drop libc-client-dev from the bookworm PHP 8.2 image [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/119 [12:24:21] (03update) 10jforrester: fundraising: Drop libc-client-dev from the bookworm PHP 8.2 image [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/119 [12:29:13] 10Continuous-Integration-Infrastructure: Integration Puppetserver CA about to expire - https://phabricator.wikimedia.org/T435319 (10taavi) 03NEW [12:30:03] (03merge) 10jforrester: fundraising: Drop libc-client-dev from the bookworm PHP 8.2 image [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/119 [12:30:28] !log Updating development images on contint primary for β€œfundraising: Drop libc-client-dev from the bookworm PHP 8.2 image” [12:30:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:31:16] 10GitLab (CI & Job Runners), 06Collaboration-Services: Puppet CA certificate for WMCS puppet server gitlab-runners-puppetserver-01 is about to expire - https://phabricator.wikimedia.org/T433581#12230938 (10taavi) 05Openβ†’03Resolved The `gitlab-runners` CA was renewed just as fine. The alert was not for... [12:56:03] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:00:35] (03CR) 10Jforrester: [C:03+2] docker: Base the dev image on bookworm [integration/quibble] - 10https://gerrit.wikimedia.org/r/1326974 (https://phabricator.wikimedia.org/T435275) (owner: 10Pwangai) [13:19:26] (03Merged) 10jenkins-bot: docker: Base the dev image on bookworm [integration/quibble] - 10https://gerrit.wikimedia.org/r/1326974 (https://phabricator.wikimedia.org/T435275) (owner: 10Pwangai) [13:23:30] (03PS1) 10Hashar: dockerfiles: add python-all image based on Trixie [integration/config] - 10https://gerrit.wikimedia.org/r/1323856 (https://phabricator.wikimedia.org/T421348) [13:30:21] (03PS2) 10Hashar: dockerfiles: add python-all image based on Trixie [integration/config] - 10https://gerrit.wikimedia.org/r/1323856 (https://phabricator.wikimedia.org/T421348) [13:37:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:47:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:48:06] hashar: You nerd-sniped me into venv-ing the quibble install. :-) [13:48:44] (03CR) 10Jforrester: "Should we call this python-311-314 instead so it's clear what it contains?" [integration/config] - 10https://gerrit.wikimedia.org/r/1323856 (https://phabricator.wikimedia.org/T421348) (owner: 10Hashar) [13:59:05] (03PS1) 10Jforrester: Docker: [quibble-bookworm] Switch quibble to venv for easier installs [integration/config] - 10https://gerrit.wikimedia.org/r/1327120 [14:03:39] hashar: Your +1 on ^ would be great. [14:10:01] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Add new nodes to Catalyst cluster for code-server - https://phabricator.wikimedia.org/T432751#12231462 (10jnuche) 05Openβ†’03Resolved [14:21:59] (03merge) 10kamila: make-container-image: add mediawiki publish-85 flavour [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/262 (https://phabricator.wikimedia.org/T432988) [14:23:48] (03open) 10arnaudb: Add a dedicated debug-ci project runner for the CDN repro (T425441) [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/626 (https://phabricator.wikimedia.org/T425441) [14:27:09] 06Project-Admins: Create User-Les4353 project. - https://phabricator.wikimedia.org/T435338 (10Les4353) 03NEW [14:28:39] 06Project-Admins: Create User-Les4353 project - https://phabricator.wikimedia.org/T435338#12231582 (10Les4353) [14:31:03] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [14:31:50] (03update) 10arnaudb: Draft: Add a dedicated debug-ci project runner for the CDN repro (T425441) [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/626 (https://phabricator.wikimedia.org/T425441) [14:36:44] (03update) 10arnaudb: Draft: Add a dedicated debug-ci project runner for the CDN repro (T425441) [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/626 (https://phabricator.wikimedia.org/T425441) [14:37:49] (03PS1) 10Jforrester: docker: Use venv for Quibble installation [integration/quibble] - 10https://gerrit.wikimedia.org/r/1327130 [14:43:12] (03update) 10arnaudb: Draft: Add a dedicated debug-ci project runner for the CDN repro (T425441) [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/626 (https://phabricator.wikimedia.org/T425441) [14:44:52] (03PS4) 10Pwangai: castor: measure cache-load rsync [integration/config] - 10https://gerrit.wikimedia.org/r/1313076 (https://phabricator.wikimedia.org/T432685) (owner: 10Phedenskog) [14:45:21] (03update) 10arnaudb: Add a dedicated debug-ci project runner for the CDN repro (T425441) [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/626 (https://phabricator.wikimedia.org/T425441) [14:45:44] 10Gerrit, 06Collaboration-Services, 10MathSearch: MathSearch can no longer download individual files from gitiles (403) - https://phabricator.wikimedia.org/T435289#12231670 (10Physikerwelt) 05Invalidβ†’03Open ο»Ώο»Ώο»ΏThe parameters vary. For example, my last attempt was url: https://gerrit.wikimedia.org/r/plug... [14:48:06] !log integration: updated castor save job to have rsync emit statistics | https://gerrit.wikimedia.org/r/c/integration/config/+/1321049 | T432685 [14:48:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:48:10] T432685: Measure the castor sync time - https://phabricator.wikimedia.org/T432685 [14:51:04] (03CR) 10Hashar: [C:04-1] "I have deployed the job, example https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/6913970/console" [integration/config] - 10https://gerrit.wikimedia.org/r/1321049 (https://phabricator.wikimedia.org/T432685) (owner: 10Pwangai) [14:55:07] 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for gaiarajan - https://phabricator.wikimedia.org/T434626#12231738 (10gaiarajan) Thank you so much! [14:56:09] 10Gerrit, 06Collaboration-Services, 10MathSearch, 13Patch-For-Review: MathSearch can no longer download individual files from gitiles (403) - https://phabricator.wikimedia.org/T435289#12231744 (10Physikerwelt) It worked with the user agent. I reopened the ticket from a MathSearch perspective, as this appea... [14:57:42] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12231758 (10ABran-WMF) After analyzing what happened this morning, it appears we had 2 distinct issues: the one triggering {T435292}, reported by @BTulli... [15:03:12] (03open) 10phedenskog: Also count errors as flaky tests for browser tests [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/27 [15:03:32] (03merge) 10phedenskog: Also count errors as flaky tests for browser tests [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/27 [15:03:43] (03merge) 10phedenskog: Turn the browser tests page into a flakiness dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/26 [15:03:44] (03CR) 10Hashar: [C:03+2] "I have changed the regex to make the `INFO:quibble.commands:` parts to be optional:" [integration/config] - 10https://gerrit.wikimedia.org/r/1321049 (https://phabricator.wikimedia.org/T432685) (owner: 10Pwangai) [15:07:01] (03Merged) 10jenkins-bot: castor: Add rsync stats and timing to the cache save [integration/config] - 10https://gerrit.wikimedia.org/r/1321049 (https://phabricator.wikimedia.org/T432685) (owner: 10Pwangai) [15:09:55] (03PS5) 10Pwangai: castor: measure cache-load rsync [integration/config] - 10https://gerrit.wikimedia.org/r/1313076 (https://phabricator.wikimedia.org/T432685) (owner: 10Phedenskog) [15:12:07] 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work: Phabricator Switchover - https://phabricator.wikimedia.org/T435087#12231997 (10Arnoldokoth) [15:12:17] (03merge) 10dancy: main.py: Clarify what review_diffs() is returning [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1265 [15:12:53] (03PS1) 10Hashar: castor: update jobs for releng/castor:0.4.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1327136 (https://phabricator.wikimedia.org/T432685) [15:12:56] (03update) 10dancy: kubernetes.py: Rename rollout_is_progressing to rollout_was_abandoned [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1267 (https://phabricator.wikimedia.org/T375514) [15:14:10] (03CR) 10Hashar: [C:03+2] "The jobs will be switched up with Id5c8a07b4f9bbb79fe0002dd24f8161739e9efbc once I have built the image." [integration/config] - 10https://gerrit.wikimedia.org/r/1313076 (https://phabricator.wikimedia.org/T432685) (owner: 10Phedenskog) [15:16:01] (03merge) 10dancy: kubernetes.py: Rename rollout_is_progressing to rollout_was_abandoned [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1267 (https://phabricator.wikimedia.org/T375514) [15:18:27] (03Merged) 10jenkins-bot: castor: measure cache-load rsync [integration/config] - 10https://gerrit.wikimedia.org/r/1313076 (https://phabricator.wikimedia.org/T432685) (owner: 10Phedenskog) [15:21:06] (03open) 10dancy: deployment-info.json: Report the checkouts of the master branch [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1268 (https://phabricator.wikimedia.org/T434726) [15:23:43] 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work: Phabricator Switchover - https://phabricator.wikimedia.org/T435087#12232197 (10Arnoldokoth) Thanks @brennen @Aklapper This will be happening tomorrow at: https://zonestamp.toolforge.org/1787205600 [15:27:51] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T435153#12232230 (10bd808) 05Openβ†’03Invalid Data collector failure. `lang=shell-session bd808@deployment-cache-text08.deployment-prep.eqiad1:~$ w 15:26:27 up 33 days, 22:19... [15:29:14] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://phabricator.wikimedia.org/T435193#12232236 (10bd808) 05Openβ†’03In progress a:03bd808 `lang=shell-session,counterexample bd808@deployment-cache-text08.deployment-prep.eqiad1:~ $ s... [15:32:20] (03merge) 10dancy: deployment-info.json: Report the checkouts of the master branch [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1268 (https://phabricator.wikimedia.org/T434726) [15:32:26] (03CR) 10Hashar: [C:03+2] "I am updating all jobs 😊" [integration/config] - 10https://gerrit.wikimedia.org/r/1327136 (https://phabricator.wikimedia.org/T432685) (owner: 10Hashar) [15:32:38] 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work: Phabricator Switchover - https://phabricator.wikimedia.org/T435087#12232260 (10Arnoldokoth) [15:34:00] (03update) 10dancy: Release 4.283.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1269 [15:34:00] (03open) 10dancy: Release 4.283.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1269 [15:34:27] 10Gerrit, 06Collaboration-Services, 10MathSearch, 13Patch-For-Review: MathSearch can no longer download individual files from gitiles (403) - https://phabricator.wikimedia.org/T435289#12232269 (10Krinkle) See also {T422686} [15:35:21] (03Merged) 10jenkins-bot: castor: update jobs for releng/castor:0.4.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1327136 (https://phabricator.wikimedia.org/T432685) (owner: 10Hashar) [15:37:02] (03merge) 10dancy: Release 4.283.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1269 [15:39:22] 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Phabricator Switchover - https://phabricator.wikimedia.org/T435087#12232338 (10Arnoldokoth) [15:44:54] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://phabricator.wikimedia.org/T435193#12232384 (10bd808) `varnish-frontend-slowlog` seems to have gone mad with syslog logging. [15:52:42] 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Phabricator Switchover - https://phabricator.wikimedia.org/T435087#12232440 (10Arnoldokoth) [15:56:17] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://phabricator.wikimedia.org/T435193#12232453 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/2005c88736e880620f552501db032db6e950cb18%5E%21/#F0 ` diff -... [15:56:24] 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Phabricator Switchover - https://phabricator.wikimedia.org/T435087#12232454 (10Arnoldokoth) [15:56:38] 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Phabricator Switchover - https://phabricator.wikimedia.org/T435087#12232461 (10Arnoldokoth) [16:04:09] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://phabricator.wikimedia.org/T435193#12232528 (10bd808) `thumb_domain` added in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1306896 / {c98b9043cafe306fc195bc87fcd76745a6ac8b1d}... [16:04:48] (03PS1) 10Vaughn Walters: jjb: [catalyst-daily-*] Alert the same people as the beta Selenium jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1327147 (https://phabricator.wikimedia.org/T435221) [16:09:29] (03CR) 10Vaughn Walters: [C:03+2] jjb: [catalyst-daily-*] Alert the same people as the beta Selenium jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1327147 (https://phabricator.wikimedia.org/T435221) (owner: 10Vaughn Walters) [16:10:07] (03CR) 10Vaughn Walters: [C:03+2] "ran jjb-update succesfully and now merging" [integration/config] - 10https://gerrit.wikimedia.org/r/1327147 (https://phabricator.wikimedia.org/T435221) (owner: 10Vaughn Walters) [16:11:13] (03Merged) 10jenkins-bot: jjb: [catalyst-daily-*] Alert the same people as the beta Selenium jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1327147 (https://phabricator.wikimedia.org/T435221) (owner: 10Vaughn Walters) [16:11:40] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://phabricator.wikimedia.org/T435193#12232571 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/6dc48006f47b9c742e9ab0ec859fb8f7ca0dca9f%5E%21/#F0 ` diff -... [16:15:03] 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Phabricator Switchover - https://phabricator.wikimedia.org/T435087#12232583 (10Arnoldokoth) [16:26:04] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:29:57] 06Release-Engineering-Team, 10Catalyst: Migrate Patchdemo & Catalyst to PHP 8.5 - https://phabricator.wikimedia.org/T435368 (10brennen) 03NEW [16:33:23] 06Release-Engineering-Team, 10Catalyst: Migrate Patchdemo & Catalyst to PHP 8.5 - https://phabricator.wikimedia.org/T435368#12232696 (10brennen) Capturing from @jnuche in RelEng meeting chat: > Catalyst base would also need a bump I'm seeing: https://gitlab.wikimedia.org/repos/releng/dev-images/-/blob/main/do... [16:36:07] 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Phabricator Switchover - https://phabricator.wikimedia.org/T435087#12232736 (10brennen) > This will be happening tomorrow at: https://zonestamp.toolforge.org/1787205600 I will try to be available during that window... [16:41:22] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Integration Puppetserver CA about to expire - https://phabricator.wikimedia.org/T435319#12232773 (10dancy) [16:42:28] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Integration Puppetserver CA about to expire - https://phabricator.wikimedia.org/T435319#12232779 (10hashar) a:03hashar [16:43:39] 10Beta-Cluster-Infrastructure, 06Traffic: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://phabricator.wikimedia.org/T435193#12232802 (10bd808) p:05Triageβ†’03High The VCL compiler is not happy yet: `counterexample Error: '/usr/local/sbin/reload-vcl -n fronte... [16:46:12] 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Phabricator Switchover - https://phabricator.wikimedia.org/T435087#12232855 (10Aklapper) I also plan to be around if my alarm clock doesn't die [16:49:03] 10Beta-Cluster-Infrastructure, 06Traffic: Puppet broken on deployment-cache-text08 by disk space & Puppet VCL changes - https://phabricator.wikimedia.org/T435193#12232879 (10taavi) >>! In T435193#12232804, @bd808 wrote: > @BCornwall is this a magically secret prod-only file that we should be feature flagging t... [16:49:07] 10Beta-Cluster-Infrastructure, 06Traffic: Puppet broken on deployment-cache-text08 by disk space & Puppet VCL changes - https://phabricator.wikimedia.org/T435193#12232881 (10bd808) [16:51:39] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Integration Puppetserver CA about to expire - https://phabricator.wikimedia.org/T435319#12232943 (10hashar) [16:52:22] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Integration Puppetserver CA about to expire - https://phabricator.wikimedia.org/T435319#12232950 (10hashar) We briefly talked about it on last Friday, the alert showing up has: ` Puppet CA certificate Puppet CA: integration-puppetmaster-02.in... [16:52:56] dancy: I looked at the certificate revocation list instead of the authority one :] [16:53:03] aha! [16:53:05] crl.pem [16:53:23] which I think is several MBytes large cause I imagine it has a collection of every single cert that got revoked over X years [16:53:53] /var/lib/puppet/ssl/crl.pem is 12MBytes [16:54:33] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Integration Puppetserver CA about to expire - https://phabricator.wikimedia.org/T435319#12232960 (10hashar) We briefly talked about it on last Friday, the alert showing up has: ` Puppet CA certificate Puppet CA: integration-puppetmaster-02.in... [16:56:34] 06Release-Engineering-Team, 10Catalyst: Migrate Patchdemo & Catalyst to PHP 8.5 - https://phabricator.wikimedia.org/T435368#12232981 (10brennen) Supersedes {T423359}. [16:57:32] 06Release-Engineering-Team, 10dev-images, 10Catalyst (Luka Ijo Pimeja Jan): Upgrade Patchdemo/Catalyst PHP images to use PHP 8.3, ensure the node matches what's used in CI - https://phabricator.wikimedia.org/T423359#12232989 (10brennen) 05Openβ†’03Resolved The remaining item here is now {T435368}. [17:02:57] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Integration Puppetserver CA about to expire - https://phabricator.wikimedia.org/T435319#12233034 (10hashar) 05Openβ†’03Resolved I ran the runbook at https://wikitech.wikimedia.org/wiki/Help:Project_puppetserver#Renewing_puppetserver_CA_... [17:04:41] dancy: well the alert is gone https://alerts.wikimedia.org/?q=alertname%3DPuppetCertificateAboutToExpire [17:04:50] and the CA expiry got pushed by nearly 10 years [17:04:53] success [17:05:07] Interesting. [17:05:14] And clients are happy? [17:05:24] the local one yeah [17:06:17] abnd the agent on integration-agent-docker-1093 works fine :) [17:06:37] and as a follow up [17:06:57] the integration-puppetserver was used heavily when we were still managing lot of things via puppet [17:07:12] but nowadays that is rather stable andbarely changing short of a few hot fix once every six months or so [17:07:31] so certainly we no more the puppetserver as a workaround for lack of access to operations/puppet +2 (which was my rant) [17:07:38] but we do have a secret stored there [17:07:39] for cumin [17:07:56] and cumin is quite helpful to act on the fleet of agents [17:08:09] so yeah I am tempted to just leave it as is [17:08:18] Agreed [17:11:44] I was worried that we'd have to do something to all the clients. But if that's not the case, this was an easy fix. [17:12:31] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Integration Puppetserver CA about to expire - https://phabricator.wikimedia.org/T435319#12233120 (10dancy) Thank you @hashar! [17:14:44] dancy: yes you do? at the bottom of https://wikitech.wikimedia.org/wiki/Help:Project_puppetserver#Renewing_puppetserver_CA_certificate is a step to delete the cached CA cert from clients so that it will fetch the updated one [17:15:32] 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Phabricator Switchover - https://phabricator.wikimedia.org/T435087#12233139 (10Arnoldokoth) [17:15:51] taavi: Any idea why run-puppet-agent succeeds as is? [17:16:05] because the old certificate still has validity left [17:16:16] that operating is not replacing the CA certificate entirely, it's just extending its validity [17:19:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:19:09] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T435375 (10wmcs-alerts) 03NEW [17:21:18] taavi: hmm. I don't understand. The puppetserver is responding with the new cert, so where does the cached CA cert get involved? [17:21:23] 10Continuous-Integration-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Re-build integration-cumin.integration.eqiad1.wikimedia.cloud on something newer than bullseye - https://phabricator.wikimedia.org/T433592#12233165 (10Jdforrester-WMF) I created the instance, but I can't shell into the new box... [17:22:19] dancy: when the the agent on the clients is validating the certificate that the server presents during the TLS handshake [17:23:36] similar to the trusted CA certificate bundle in your browser or operating system or whatever, the puppet agent has its own copy of the puppet CA certificate that it uses to validate any other puppet-related certificates it sees [17:24:29] if you do not update that copy of the CA certificate, the agent is going to think that the CA it's given is expired in a couple of weeks and refuse to trust any certificates it sees [17:27:49] 10Continuous-Integration-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Re-build integration-cumin.integration.eqiad1.wikimedia.cloud on something newer than bullseye - https://phabricator.wikimedia.org/T433592#12233216 (10taavi) The initial Puppet run on the host is failing to compile, which means... [17:29:39] 10Continuous-Integration-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Re-build integration-cumin.integration.eqiad1.wikimedia.cloud on something newer than bullseye - https://phabricator.wikimedia.org/T433592#12233220 (10Jdforrester-WMF) I applied them but it didn't let me in, so I removed them a... [17:34:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:36:36] 10Continuous-Integration-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Re-build integration-cumin.integration.eqiad1.wikimedia.cloud on something newer than bullseye - https://phabricator.wikimedia.org/T433592#12233263 (10taavi) Yeah, if the initial Puppet run fails then the instance usually gets... [17:37:30] 10Continuous-Integration-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Re-build integration-cumin.integration.eqiad1.wikimedia.cloud on something newer than bullseye - https://phabricator.wikimedia.org/T433592#12233266 (10Jdforrester-WMF) Aha! OK, that's fine. [17:43:39] 10Beta-Cluster-Infrastructure, 06Traffic: Puppet broken on deployment-cache-text08 by disk space & Puppet VCL changes - https://phabricator.wikimedia.org/T435193#12233303 (10bd808) 05In progressβ†’03Resolved Thanks for spotting that for me @taavi. Puppet run is clean now: `lang=shell-session root@deploy... [17:52:37] 06Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Configure additional httpbb checks to perform during a deployment - https://phabricator.wikimedia.org/T428972#12233334 (10Scott_French) The new tests are live and appear to work as expected - i.e., wired into both the `appserver` and `pretrain` suites.... [17:53:40] 10Continuous-Integration-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Re-build integration-cumin.integration.eqiad1.wikimedia.cloud on something newer than bullseye - https://phabricator.wikimedia.org/T433592#12233336 (10Jdforrester-WMF) OK, newly-re-created integration-cumin-01 is now set up, wi... [17:56:28] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform: Investigate why the job median time regressed in May for quibble-with-gated-extensions-vendor-mysql-php83 - https://phabricator.wikimedia.org/T427752#12233370 (10hashar) [17:57:55] (03open) 10phedenskog: Show castor restore times next to the save times [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/28 (https://phabricator.wikimedia.org/T432688) [18:15:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:17:34] 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work, 13Patch-For-Review: Phabricator Switchover - https://phabricator.wikimedia.org/T435087#12233441 (10Arnoldokoth) [18:20:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:20:54] Amir1: Warning: Undefined variable $wmgThumbUrl in /srv/mediawiki-staging/wmf-config/filebackend.php on line 301 during l10n build on beta. [18:21:07] on it [18:21:12] I hope to have a fix soon [18:21:23] Cool. thx [18:27:24] dancy: it was soo dumb https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1327167 [18:27:57] Gotta love a one-character fix [18:28:06] Yuuup. [18:28:12] my apologies, it's such a weird undocumented behavior [18:30:36] of course there is a wikipedia aritcle for it https://en.wikipedia.org/wiki/Elvis_operator [18:47:16] 06Project-Admins: Create User-Les4353 project - https://phabricator.wikimedia.org/T435338#12233599 (10Les4353) [18:56:08] 10Phabricator-Bot-Requests, 06Discovery-Search, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Requesting Phab bot for Machine Learning/Data Engineering team - https://phabricator.wikimedia.org/T433543#12233651 (10bking) 05Openβ†’03Declined As stated in T431498, the primary purpose of this bot would be... [19:00:25] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create User-Les4353 project - https://phabricator.wikimedia.org/T435338#12233665 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Requested public project tag #user-les4353 has been created: https://phabricator.wikimedia.org/project/manage/8975/ Int... [19:10:34] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cirrussearch12 on project deployment-prep - https://phabricator.wikimedia.org/T424104#12233705 (10bking) 05Openβ†’03Resolved This instance no longer exists. Resolving... [19:10:41] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T435375#12233708 (10bd808) 05Openβ†’03Invalid [19:15:57] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): No Puppet resources found on instance deployment-cirrussearch14 on project deployment-prep - https://phabricator.wikimedia.org/T424100#12233761 (10bking) 05In progressβ†’03Resolved This instance no longer exists. Resolving... [19:30:21] 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 10Cloud-VPS (Debian Bullseye Deprecation): Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393 (10bd808) 03NEW [19:30:57] 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 10Cloud-VPS (Debian Bullseye Deprecation): Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12233843 (10bd808) [19:30:59] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 10Cloud-VPS (Debian Bullseye Deprecation), 07Epic: Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12233844 (10bd808) [20:38:22] (03open) 10dancy: kubernetes.py: Revise wording in docstrings [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1270 [20:40:56] (03merge) 10dancy: kubernetes.py: Revise wording in docstrings [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1270 [20:55:57] (03update) 10addshore: Keep existing linux/arm fallback behavior, but fix detection so it actually [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/662 (https://phabricator.wikimedia.org/T435037) [20:58:01] (03update) 10dancy: kubernetes.py: Roll back to the revision from before the deployment [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1271 (https://phabricator.wikimedia.org/T412941) [20:58:04] (03open) 10dancy: kubernetes.py: Roll back to the revision from before the deployment [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1271 (https://phabricator.wikimedia.org/T412941) [21:06:24] (03update) 10dancy: kubernetes.py: Roll back to the revision from before the deployment [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1271 (https://phabricator.wikimedia.org/T412941) [21:09:27] (03update) 10dancy: kubernetes.py: Roll back to the revision from before the deployment [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1271 (https://phabricator.wikimedia.org/T412941) [21:17:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [21:17:08] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T435408 (10wmcs-alerts) 03NEW [21:18:51] (03open) 10dancy: kubernetes.py: Make the train-dev replica count a setting [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1272 [21:21:13] (03merge) 10dancy: kubernetes.py: Make the train-dev replica count a setting [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1272 [21:37:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [21:41:03] (03update) 10dancy: kubernetes.py: Roll back to the revision from before the deployment [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1271 (https://phabricator.wikimedia.org/T412941) [21:41:04] (03update) 10dancy: deploy-service: Run helmfile commands, add --confirm-diffs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1273 (https://phabricator.wikimedia.org/T412941) [21:41:15] (03update) 10dancy: deploy-service: Run helmfile commands, add --confirm-diffs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1273 (https://phabricator.wikimedia.org/T412941) [21:41:15] (03update) 10dancy: kubernetes.py: Roll back to the revision from before the deployment [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1271 (https://phabricator.wikimedia.org/T412941) [21:41:17] (03open) 10dancy: deploy-service: Run helmfile commands, add --confirm-diffs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1273 (https://phabricator.wikimedia.org/T412941) [22:26:15] 06Release-Engineering-Team: Configure logstash error checks for automated supervision of Pretrain deployments - https://phabricator.wikimedia.org/T435419 (10Scott_French) 03NEW [22:26:45] 06Release-Engineering-Team: Configure logstash error checks for automated supervision of Pretrain deployments - https://phabricator.wikimedia.org/T435419#12234620 (10Scott_French) p:05Triageβ†’03Medium [22:27:04] 06Release-Engineering-Team (Doing 😎): [FY26-27 ST5.1.2] Automated supervision - https://phabricator.wikimedia.org/T427968#12234629 (10Scott_French) [22:27:05] 06Release-Engineering-Team: Configure logstash error checks for automated supervision of Pretrain deployments - https://phabricator.wikimedia.org/T435419#12234628 (10Scott_French) [22:32:54] 06Release-Engineering-Team: Configure logstash error checks for automated supervision of Pretrain deployments - https://phabricator.wikimedia.org/T435419#12234642 (10Scott_French) [22:40:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [22:45:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [22:55:04] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T435408#12234663 (10bd808) 05Openβ†’03Invalid I think the crawler pressure is so high that the CDN edge is getting swamped out and missing metrics reporting periods. {F9917405... [22:56:16] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12234668 (10bd808) "Fun" new developments. :/ >>! In T435408#12234663, @bd808 wrote: > I think the crawler pressure is so high that the CDN edge is get... [23:27:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [23:27:11] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T435421 (10wmcs-alerts) 03NEW [23:37:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [23:45:15] 06Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Configure additional httpbb checks to perform during a deployment - https://phabricator.wikimedia.org/T428972#12234814 (10bd808) >>! In T428972#12233334, @Scott_French wrote: > 2. Updating https://wikitech.wikimedia.org/wiki/Httpbb#Adding_tests in some... [23:59:39] (03PS1) 10MusikAnimal: Zuul: [mediawiki/extensions/PersonalDashboard] Publish doxygen docs [integration/config] - 10https://gerrit.wikimedia.org/r/1327222 (https://phabricator.wikimedia.org/T435392)