[00:07:04] 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12254809 (10bd808) `lang=shell-session bd808@de... [00:14:03] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-deploy06 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [00:57:03] FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [01:16:47] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12254869 (10Hakimi97) @tstarling Since it was decided that the current solution is to block all public internet traffic except for those within the IP w... [01:51:29] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12254894 (10tstarling) >>! In T420833#12254869, @Hakimi97 wrote: > @tstarling Since it was decided that the current solution is to block all public inte... [02:17:03] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12254909 (10Andrew) {F100150875} [03:04:50] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12254928 (10AlexisJazz) (adding some Commons app contributors to inform them about possible changes) [07:10:09] (03PS2) 10Hashar: Phase out quibble --commands [integration/config] - 10https://gerrit.wikimedia.org/r/1293711 (https://phabricator.wikimedia.org/T321617) [07:10:58] (03CR) 10Hashar: [C:03+2] "Jobs updated!" [integration/config] - 10https://gerrit.wikimedia.org/r/1293711 (https://phabricator.wikimedia.org/T321617) (owner: 10Hashar) [07:11:42] !log integration: updated Quibble jobs to replace deprecated `--commands` option by 1..n `--command` option(s) | https://gerrit.wikimedia.org/r/c/integration/config/+/1293711 | T321617 [07:11:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:11:45] T321617: Fully remove deprecated --commands - https://phabricator.wikimedia.org/T321617 [07:12:41] (03Merged) 10jenkins-bot: Phase out quibble --commands [integration/config] - 10https://gerrit.wikimedia.org/r/1293711 (https://phabricator.wikimedia.org/T321617) (owner: 10Hashar) [07:14:42] (03PS2) 10Hashar: Remove deprecated --commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/1293712 (https://phabricator.wikimedia.org/T321617) [07:51:03] (03open) 10phedenskog: Add Prometheus exporter for CI feedback time [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/43 (https://phabricator.wikimedia.org/T435814) [07:57:37] brennen: o/ copied releng/composer-test-php84:8.4.7-s3 over to the new releng s3 bucket! [07:57:40] (03update) 10phedenskog: Add Prometheus exporter for CI feedback time [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/43 (https://phabricator.wikimedia.org/T435814) [08:03:17] (03merge) 10phedenskog: Add Prometheus exporter for CI feedback time [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/43 (https://phabricator.wikimedia.org/T435814) [08:04:43] brennen: note `releng/composer-test-php84:8.4.7-s3` is obsolete. PHP 8.4 has been upgraded and those -test image got deleted in favor of using `releng/composer-phpXX:yyy` with `--entrypoint=/run-test`. See https://gerrit.wikimedia.org/r/c/integration/config/+/1170585 which was done in order to reduce the number of images we maintain ( https://phabricator.wikimedia.org/T397429 ) [08:17:11] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.47.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T430836#12255488 (10hashar) [10:01:58] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.47.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T430836#12255865 (10hashar) The Wikibase clients issue T436060 got prompty resolved. I have promoted group 1 wikis... [10:43:04] (03CR) 10Hashar: backend: Use venv for Quibble on Postgres (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/1329356 (owner: 10Jforrester) [11:00:22] (03open) 10phedenskog: Add agent performance chart to the CI load dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/44 [11:01:28] (03CR) 10Phedenskog: [C:03+2] Remove deprecated --commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/1293712 (https://phabricator.wikimedia.org/T321617) (owner: 10Hashar) [11:04:20] (03merge) 10phedenskog: Add agent performance chart to the CI load dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/44 [11:04:37] 10Continuous-Integration-Infrastructure, 10Castor, 10Cloud-VPS (Quota-requests), 06Test Platform (Arusha 30): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12256117 (10AMarkossyan-WMF) [11:05:09] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor, 07Spike, 06Test Platform (Arusha 30): Create a plan on how to make Castor save jobs async - https://phabricator.wikimedia.org/T425411#12256123 (10AMarkossyan-WMF) [11:12:31] 10Phabricator, 07Regression: (DateInvalidTimeZoneException) DateTimeZone::__construct(): Unknown or bad timezone (Europe/Kiev) - https://phabricator.wikimedia.org/T435441#12256150 (10Aklapper) 05Open→03Declined I currently don't plan to edit database entries for those >100 users who have set their time... [11:17:00] (03Merged) 10jenkins-bot: Remove deprecated --commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/1293712 (https://phabricator.wikimedia.org/T321617) (owner: 10Hashar) [11:44:04] (03PS1) 10Michael Große: zuul: add TestKitchen as GrowthExperiments browser test dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1329537 (https://phabricator.wikimedia.org/T432920) [11:44:44] friendly nudge on T433592 [11:44:45] T433592: Re-build integration-cumin.integration.eqiad1.wikimedia.cloud on something newer than bullseye - https://phabricator.wikimedia.org/T433592 [11:45:19] (03CR) 10CI reject: [V:04-1] zuul: add TestKitchen as GrowthExperiments browser test dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1329537 (https://phabricator.wikimedia.org/T432920) (owner: 10Michael Große) [12:16:19] 10GitLab, 06Release-Engineering-Team (Priority Backlog 📥), 06Collaboration-Services, 10Development-Metrics: development-metrics crawler bot runs into 404 errors collection GitLab data - https://phabricator.wikimedia.org/T409903#12256424 (10Aklapper) [12:17:30] 06Release-Engineering-Team (Priority Backlog 📥), 10Development-Metrics: Evaluate sharing indexed repositories between Grimoirelab and Codesearch - https://phabricator.wikimedia.org/T430117#12256426 (10Aklapper) [12:17:31] 06Release-Engineering-Team (Doing 😎), 10Development-Metrics: Automatically export and publish a list of WMF deployed code repositories in Bitergia's JSON format - https://phabricator.wikimedia.org/T385529#12256427 (10Aklapper) [12:22:07] 10Continuous-Integration-Config, 10Quibble: Fully remove deprecated --commands - https://phabricator.wikimedia.org/T321617#12256457 (10hashar) 05Open→03Resolved a:03hashar I switched the remaining uses to `-c` / `--command`. The patch got merged by @Peter , that is one less long time deprecated thing! [12:29:27] (03CR) 10Phedenskog: [C:03+2] castor: Print the load outcome only once [integration/config] - 10https://gerrit.wikimedia.org/r/1329398 (https://phabricator.wikimedia.org/T436032) (owner: 10Pwangai) [12:31:09] (03Merged) 10jenkins-bot: castor: Print the load outcome only once [integration/config] - 10https://gerrit.wikimedia.org/r/1329398 (https://phabricator.wikimedia.org/T436032) (owner: 10Pwangai) [12:32:05] (03CR) 10Hashar: castor: Print the load outcome only once (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1329398 (https://phabricator.wikimedia.org/T436032) (owner: 10Pwangai) [12:32:26] (03CR) 10Hashar: "Too late! I will send it as a follow up :-]" [integration/config] - 10https://gerrit.wikimedia.org/r/1329398 (https://phabricator.wikimedia.org/T436032) (owner: 10Pwangai) [12:36:57] (03PS1) 10Hashar: castor: fix up Finish marker in load and save scripts [integration/config] - 10https://gerrit.wikimedia.org/r/1329552 [12:37:11] (03CR) 10Hashar: "I10063466c51e39bbdbba71af9d1389d7da8f88b9" [integration/config] - 10https://gerrit.wikimedia.org/r/1329398 (https://phabricator.wikimedia.org/T436032) (owner: 10Pwangai) [12:46:12] 10Gerrit (Gerrit 3.13), 07Upstream: Background image of login page is served with a short TTL - https://phabricator.wikimedia.org/T392479#12256546 (10hashar) 05Open→03Stalled My patch got merged and released with Gerrit v3.13.0. I am marking this {nav Stalled} until we upgrade Gerrit. [12:47:27] 10Gerrit (Gerrit 3.13), 07Upstream: Background image of login page is served with a short TTL - https://phabricator.wikimedia.org/T392479#12256553 (10hashar) p:05Triage→03Low [12:48:13] (03CR) 10Jforrester: backend: Use venv for Quibble on Postgres (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/1329356 (owner: 10Jforrester) [12:52:09] (03close) 10hashar: Configure repositories to avoind pruning tags by default [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1107 (https://phabricator.wikimedia.org/T418085) [12:52:39] 06Release-Engineering-Team, 10Scap, 13Patch-For-Review: git pull on a scap deployment repository deletes scap sync tags - https://phabricator.wikimedia.org/T418085#12256567 (10hashar) 05Open→03Resolved Abandoning, that was due to my odd git config which caused tags to be pruned. It is not worth pursu... [13:14:12] (03open) 10phedenskog: Show the graphs first on the core dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/45 [13:14:17] (03merge) 10phedenskog: Show the graphs first on the core dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/45 [13:24:47] (03CR) 10Hashar: [C:03+2] "The coverage report is already published at https://doc.wikimedia.org/cover/quibble/" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1299557 (https://phabricator.wikimedia.org/T428642) (owner: 10Hashar) [13:27:27] development-metrics.wmcloud.org seems to have stopped ingesting Gerrit change data [13:36:36] (03Merged) 10jenkins-bot: build: generate lcov report [integration/quibble] - 10https://gerrit.wikimedia.org/r/1299557 (https://phabricator.wikimedia.org/T428642) (owner: 10Hashar) [13:46:17] 10Beta-Cluster-Infrastructure: Clean up broken legacy maps stack in Beta Cluster - https://phabricator.wikimedia.org/T421449#12256836 (10taavi) [13:46:19] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Priority Backlog 📥), 10Cloud-VPS (Debian Bullseye Deprecation), 07Epic: Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12256837 (10taavi) [13:46:43] 10GitLab, 06Release-Engineering-Team (Priority Backlog 📥), 06Collaboration-Services, 10Development-Metrics: development-metrics crawler bot runs into 404 errors collection GitLab data - https://phabricator.wikimedia.org/T409903#12256849 (10Aklapper) FYI: Things have moved to ourselves hosting this at https... [13:52:58] 10Continuous-Integration-Config, 10tox-wikimedia, 07Test-Coverage: Standardize way to publish test coverage for Python projects in CI - https://phabricator.wikimedia.org/T244650#12256873 (10hashar) [14:08:44] 10GerritBot, 10Phabricator: gerritbot: the icon color of abandoned message is green, expected red - https://phabricator.wikimedia.org/T436101 (10Les4353) 03NEW [14:08:56] 10GerritBot, 10Phabricator: gerritbot: the icon color of abandoned message is green, expected red - https://phabricator.wikimedia.org/T436101#12256973 (10Les4353) p:05Triage→03Low [14:14:13] 10GerritBot, 10Phabricator: gerritbot: the icon color of abandoned message is green, expected red - https://phabricator.wikimedia.org/T436101#12257013 (10Aklapper) 05Open→03Declined Hi, the avatar icon does not express any status but the account which had some activity, it is static like all other avat... [14:15:14] 10GerritBot, 10Phabricator: gerritbot: the icon color of abandoned message is green, expected red - https://phabricator.wikimedia.org/T436101#12257019 (10Les4353) 05Declined→03Open >>! 在T436101#12257013中，@Aklapper写道： > Hi, the avatar icon does not express any status but the account which had some activity,... [14:18:43] 10GerritBot, 10Phabricator, 07Regression: gerritbot: the icon color of abandoned message is green, expected red - https://phabricator.wikimedia.org/T436101#12257035 (10Les4353) Correct sample: T434355#12241040 [14:18:48] 10GerritBot, 10Phabricator, 07Regression: gerritbot: the icon color of abandoned message is green, expected red - https://phabricator.wikimedia.org/T436101#12257039 (10Aklapper) 05Open→03Stalled > Reopen due to another abandon message IS red. Where exactly? Please always provide references for statements. [14:19:26] 10GerritBot, 10Phabricator, 07Regression: gerritbot: the icon color of abandoned message is green, expected red - https://phabricator.wikimedia.org/T436101#12257041 (10Les4353) 05Stalled→03Open >>! 在T436101#12257039中，@Aklapper写道： >> Reopen due to another abandon message IS red. > > Where exactly? Pleas... [14:31:02] 10Phabricator: gerritbot: the icon color of abandoned message is green, expected red - https://phabricator.wikimedia.org/T436101#12257196 (10Aklapper) Thanks. I don't see how this is a regression; #GerritBot has no influence how its messages are rendered by Phabricator as this is some custom downstream CSS. [14:31:14] 10Phabricator, 07CSS: gerritbot: the icon color of abandoned message is green, expected red - https://phabricator.wikimedia.org/T436101#12257200 (10Aklapper) Probably needs adjustments in https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/blob/wmf/stable/webroot/rsrc/css/phui/phui-timeline-view-wikim... [14:31:54] (03CR) 10Phedenskog: [C:03+1] "Ah yep this look correct, sorry I was to fast before." [integration/config] - 10https://gerrit.wikimedia.org/r/1329552 (owner: 10Hashar) [14:34:18] (03CR) 10Hashar: [C:03+2] "No worries it is fine to miss things some time :-]" [integration/config] - 10https://gerrit.wikimedia.org/r/1329552 (owner: 10Hashar) [14:36:05] (03Merged) 10jenkins-bot: castor: fix up Finish marker in load and save scripts [integration/config] - 10https://gerrit.wikimedia.org/r/1329552 (owner: 10Hashar) [14:36:41] (03CR) 10Hashar: "For the record, the image did not build after the patch got merged. I did it and published `docker-registry.wikimedia.org/releng/castor:0." [integration/config] - 10https://gerrit.wikimedia.org/r/1329398 (https://phabricator.wikimedia.org/T436032) (owner: 10Pwangai) [14:43:57] (03PS1) 10Hashar: jjb: update castor 0.4.3..0.4.5 [integration/config] - 10https://gerrit.wikimedia.org/r/1329577 [14:44:29] (03CR) 10Hashar: [C:03+2] jjb: update castor 0.4.3..0.4.5 [integration/config] - 10https://gerrit.wikimedia.org/r/1329577 (owner: 10Hashar) [14:45:33] !log Updating Castor (0.4.3..0.4.5) on all Jenkins jobs | https://gerrit.wikimedia.org/r/1329577 [14:45:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:46:09] 10Continuous-Integration-Config, 10WDAudioLex: BE - Add Gerrit Code Review workflow for this repo - https://phabricator.wikimedia.org/T391419#12257329 (10Collins) 05Open→03Resolved [14:46:44] (03Merged) 10jenkins-bot: jjb: update castor 0.4.3..0.4.5 [integration/config] - 10https://gerrit.wikimedia.org/r/1329577 (owner: 10Hashar) [14:46:47] 06Gerrit-Privilege-Requests, 10WDAudioLex: Request for plus 2 rights - https://phabricator.wikimedia.org/T391414#12257330 (10Collins) 05Open→03Resolved a:03Collins [14:47:08] 10Phabricator, 07CSS: gerritbot: the icon color of abandoned message is green, expected red - https://phabricator.wikimedia.org/T436101#12257334 (10Escargot_rouge) →14Duplicate dup:03T414028 [14:47:10] 10Phabricator, 07CSS: Gerrit comments for patch abandonments without a provided reason are displayed with a green icon (indicating a merged patch) rather than a red icon (indicating an abandoned patch) - https://phabricator.wikimedia.org/T414028#12257336 (10Escargot_rouge) [15:46:55] (03PS2) 10Jack Phoenix: zuul: [mediawiki/extensions/WikiForum] Add Echo for phan [integration/config] - 10https://gerrit.wikimedia.org/r/1329596 [15:51:18] 10Continuous-Integration-Infrastructure (Zuul upgrade): Back up Zuul encryption keys - https://phabricator.wikimedia.org/T436131 (10dduvall) 03NEW [15:55:53] 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12257808 (10bd808) >>! In T435393#12254809, @bd... [16:00:10] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Back up Zuul encryption keys - https://phabricator.wikimedia.org/T436131#12257845 (10LSobanski) [16:23:03] Project mediawiki-core-doxygen build #22415: 04FAILURE in 5 min 1 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/22415/ [16:27:10] 06Release-Engineering-Team, 10Tool-wheres-my-code-running: Handle reverts - https://phabricator.wikimedia.org/T436135 (10hashar) 03NEW [17:02:18] (03PS2) 10Michael Große: zuul: add TestKitchen as GrowthExperiments browser test dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1329537 (https://phabricator.wikimedia.org/T432920) [17:04:34] (03CR) 10Ahmon Dancy: [C:03+2] zuul: add TestKitchen as GrowthExperiments browser test dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1329537 (https://phabricator.wikimedia.org/T432920) (owner: 10Michael Große) [17:07:57] (03Merged) 10jenkins-bot: zuul: add TestKitchen as GrowthExperiments browser test dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1329537 (https://phabricator.wikimedia.org/T432920) (owner: 10Michael Große) [17:09:02] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1329537 [17:09:22] (03CR) 10Ahmon Dancy: [C:03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/1329537 (https://phabricator.wikimedia.org/T432920) (owner: 10Michael Große) [17:09:47] dancy: Thank you 😊 [17:10:09] yw [17:14:46] Ugh, I might have been overly optimistic: [17:14:48] 00:00:52.898 Error: A dependency error was encountered while installing the extension "TestKitchen": Could not find the registration file for the extension "EventBus" [17:15:36] I did not have it on my radar that the dependency resolution is manual for this job [17:15:56] I'll prepare another change right away. Though I'm not exactly able to test that locally :/ [17:16:36] I'm okay with iterating if the cycle is quick [17:20:24] (03PS1) 10Michael Große: zuul: Add EventBus to GrowthExperiments because TK needs it [integration/config] - 10https://gerrit.wikimedia.org/r/1329617 (https://phabricator.wikimedia.org/T432920) [17:20:24] (03CR) 10Michael Große: "This is a follow-up to I7366aa5a636ec865430b3bcba2ecedf78506b0b7" [integration/config] - 10https://gerrit.wikimedia.org/r/1329617 (https://phabricator.wikimedia.org/T432920) (owner: 10Michael Große) [17:21:54] (03CR) 10Ahmon Dancy: [C:03+2] zuul: Add EventBus to GrowthExperiments because TK needs it [integration/config] - 10https://gerrit.wikimedia.org/r/1329617 (https://phabricator.wikimedia.org/T432920) (owner: 10Michael Große) [17:21:58] Next change is up: [zuul: Add EventBus to GrowthExperiments because TK needs it](https://gerrit.wikimedia.org/r/c/integration/config/+/1329617). EventBus does not seem to have a required dependency, so I hope this is good enough 🤞 [17:23:35] (03Merged) 10jenkins-bot: zuul: Add EventBus to GrowthExperiments because TK needs it [integration/config] - 10https://gerrit.wikimedia.org/r/1329617 (https://phabricator.wikimedia.org/T432920) (owner: 10Michael Große) [17:24:27] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1329617 [17:24:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:24:40] MichaelG_WMF: Ready [17:25:02] dancy: Thanks! Trying my change again... [17:30:33] dancy: CI seems to now behave as expected, thank you! (My test is still failing, but that's probably unrelated.) [17:30:49] Great! [17:31:20] Yippee, build fixed! [17:31:20] Project mediawiki-core-doxygen build #22416: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/22416/ [17:51:20] 06Release-Engineering-Team, 10Tool-wheres-my-code-running: Handle reverts - https://phabricator.wikimedia.org/T436135#12258520 (10bd808) The interesting thing about a revert is that the commit is deployed, but the revert commit is also which then nullifies the original patch. > The reverts were NOT done in Ge... [17:52:27] 06Release-Engineering-Team, 10Tool-wheres-my-code-running: Detect when a commit has been reverted by a later commit - https://phabricator.wikimedia.org/T436135#12258522 (10bd808) [18:12:04] FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [18:20:04] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [18:31:25] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/labs/private on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T436020#12258643 (10BCornwall) 05Open→03Resolved Not sure if that's related to my work, but I can confirm that deplo... [18:35:49] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Doing 😎), 10Cloud-VPS (Debian Bullseye Deprecation): Planning for a plan to migrate Beta Cluster again - https://phabricator.wikimedia.org/T433998#12258671 (10ssingh) Traffic will work on upgrading `deployment-cache-(text|upload)08` if that's fine an... [18:51:58] (03open) 10dancy: cli.py: Add prompt_for_message() [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1284 (https://phabricator.wikimedia.org/T412941) [18:55:09] (03merge) 10dancy: cli.py: Add prompt_for_message() [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1284 (https://phabricator.wikimedia.org/T412941) [19:00:28] (03open) 10dancy: Use message_argment property everywhere to access message [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1285 [19:03:27] (03merge) 10dancy: Use message_argment property everywhere to access message [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1285 [19:05:33] (03update) 10dancy: deploy-service: Run helmfile commands, add --confirm-diffs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1273 (https://phabricator.wikimedia.org/T412941) [19:26:32] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: Replace deployment-deploy04 with a Bookworm instance with Java 21 - https://phabricator.wikimedia.org/T421244#12258907 (10bd808) >>! In T421244#11847209, @bd808 wrote: > Booleans were coerced to strings somewhere??? > * `yamle... [19:26:34] FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [19:35:22] (03PS1) 10Ahmon Dancy: README: say when to borrow a plugin path and when not to [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1329645 [19:35:58] (03PS2) 10Ahmon Dancy: README: say when to borrow a plugin path and when not to [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1329645 [19:36:19] (03CR) 10Ahmon Dancy: [C:03+2] README: say when to borrow a plugin path and when not to [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1329645 (owner: 10Ahmon Dancy) [19:36:58] (03Merged) 10jenkins-bot: README: say when to borrow a plugin path and when not to [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1329645 (owner: 10Ahmon Dancy) [19:41:48] (03PS1) 10Ahmon Dancy: wm-wheres-my-code-running: Show branch when hovering over a place chip [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1329648 (https://phabricator.wikimedia.org/T434726) [19:42:46] (03CR) 10Ahmon Dancy: [C:03+2] wm-wheres-my-code-running: Show branch when hovering over a place chip [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1329648 (https://phabricator.wikimedia.org/T434726) (owner: 10Ahmon Dancy) [19:43:36] (03Merged) 10jenkins-bot: wm-wheres-my-code-running: Show branch when hovering over a place chip [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1329648 (https://phabricator.wikimedia.org/T434726) (owner: 10Ahmon Dancy) [19:50:02] (03open) 10dancy: exp/docker-compose.yml: Replace en.wikinews.org with en.wiktionary.org [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/194 (https://phabricator.wikimedia.org/T421796) [19:51:07] (03merge) 10dancy: exp/docker-compose.yml: Replace en.wikinews.org with en.wiktionary.org [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/194 (https://phabricator.wikimedia.org/T421796) [19:51:54] (03open) 10dancy: deploy: Add config files for scap deploy-service [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/195 (https://phabricator.wikimedia.org/T412941) [19:52:38] (03merge) 10dancy: deploy: Add config files for scap deploy-service [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/195 (https://phabricator.wikimedia.org/T412941) [20:14:21] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537#12259070 (10Andrew) >>! In T435537#12248577, @BLiviero-WMF wrote: > can you say more about what "optimizin... [20:27:15] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T436024#12259163 (10bd808) p:05Triage→03High `lang=shell-session,counterexample bd808@deployment-cache-upload08.deployment-prep.eqiad1:~$ sudo -i r... [20:36:20] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12259239 (10BLiviero-WMF) I want to caution on the assertion tha... [21:33:32] 06Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 10ServiceOps-Mediawiki: Deploy multiversion images into pretrain scope Tu-Th at 02:00 UTC - https://phabricator.wikimedia.org/T436178 (10bd808) 03NEW [21:34:48] 06Release-Engineering-Team (Doing 😎): [FY26-27 ST5.1.2] Automated supervision - https://phabricator.wikimedia.org/T427968#12259462 (10bd808) [21:34:50] 06Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 10ServiceOps-Mediawiki: Deploy multiversion images into pretrain scope Tu-Th at 02:00 UTC - https://phabricator.wikimedia.org/T436178#12259461 (10bd808) [21:42:20] 06Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 10ServiceOps-Mediawiki: Deploy multiversion images into pretrain scope Tu-Th at 02:00 UTC - https://phabricator.wikimedia.org/T436178#12259528 (10bd808) Actually enabling this deployment should be considered blocked on communications out to SREs a... [22:03:27] 06Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 10ServiceOps-Mediawiki: Update deployment documentation to factor in pretrain deployment windows - https://phabricator.wikimedia.org/T436187 (10CCiufo-WMF) 03NEW [22:04:27] 06Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 10ServiceOps-Mediawiki: Update deployment documentation to factor in pretrain deployment windows - https://phabricator.wikimedia.org/T436187#12259683 (10CCiufo-WMF) @bd808 @dancy are there other documentation-adjacent things (maybe bots?) that nee... [22:18:14] 06Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 10ServiceOps-Mediawiki: Inform SREs and deployers of upcoming deployment and traffic routing to pretrain - https://phabricator.wikimedia.org/T436189 (10CCiufo-WMF) 03NEW [22:37:13] 06Release-Engineering-Team (Priority Backlog 📥): Exploring moving manual testing workflows from beta cluster to testwiki - https://phabricator.wikimedia.org/T436191 (10CCiufo-WMF) 03NEW [23:10:10] 06Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 10ServiceOps-Mediawiki: Update deployment documentation to factor in pretrain deployment windows - https://phabricator.wikimedia.org/T436187#12259842 (10bd808) [[https://wikitech.wikimedia.org/wiki/Tool:Jouncebot|jouncebot]] is the main bot I can... [23:10:56] 06Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 10ServiceOps-Mediawiki: Update deployment documentation to factor in pretrain deployment windows - https://phabricator.wikimedia.org/T436187#12259843 (10bd808) [23:23:03] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-cache-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [23:23:09] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-cache-text09 in project deployment-prep - https://phabricator.wikimedia.org/T436195 (10wmcs-alerts) 03NEW [23:28:03] FIRING: [2x] PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-cache-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [23:28:36] Hey, I'm spinning up new trixie text/upload nodes and there are bound to be failures. I'll try to claim/close wmcs phab tickets as they show up [23:29:05] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-cache-text09 in project deployment-prep - https://phabricator.wikimedia.org/T436195#12259898 (10BCornwall) a:03BCornwall [23:29:18] brett: I will try not to freak out :) [23:30:33] brett: you may not have seen T280393, but it might be worthwhile to ally a new naming convention to those boxes [23:30:33] T280393: Rename deployment-cache-(text|upload)0x to deployment-cp-(text|upload)0x - https://phabricator.wikimedia.org/T280393 [23:31:07] er, okay, will do [23:32:10] I'm too scared to rename, so will delete/recreate [23:38:03] RESOLVED: [2x] PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-cache-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [23:38:42] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-cache-text09 in project deployment-prep - https://phabricator.wikimedia.org/T436195#12259930 (10BCornwall) 05Open→03Invalid killed this instance in lieu of new naming scheme (https://phabricator.wikimedia.org/T280... [23:53:52] not sure what I was trying to type instead of "ally" ¯\_(ツ)_/¯ [23:56:48] 10Beta-Cluster-Infrastructure, 06SRE, 06Traffic: Name new CDN servers deployment-cp-(text|upload)0x instead of deployment-cache-(text|upload)0x - https://phabricator.wikimedia.org/T280393#12259942 (10bd808) [23:57:36] brett: I updated the task title, because yeah nobody wants to try and rename a Cloud VPS instance after it is built :) [23:57:43] ack, thanks