[01:15:58] (03open) 10dancy: Add /dev/log reader to deploy container [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/143 [01:16:01] (03update) 10dancy: Add /dev/log reader to deploy container [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/143 [01:17:01] (03merge) 10dancy: Add /dev/log reader to deploy container [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/143 [06:47:01] 10Phabricator maintenance bot, 10ContentTranslation: Complete enablement Section Translation in new wikis and make the process less manual for the future - https://phabricator.wikimedia.org/T371420#10940788 (10Nikerabbit) [06:57:54] 10Beta-Cluster-Infrastructure, 10CheckUser, 06Trust and Safety Product Team, 06Trust-and-Safety, 06WMF-Legal: Install CheckUser in Beta Cluster - https://phabricator.wikimedia.org/T357959#10940823 (10Niharika) [06:58:53] 10GitLab (Infrastructure), 10Cloud-VPS, 06collaboration-services, 10cloud-services-team (FY2024/2025-Q3-Q4): Volume is stuck to deleted instance in devtools project - https://phabricator.wikimedia.org/T396739#10940830 (10Jelto) >>! In T396739#10934468, @Andrew wrote: > I'm still wrestling with gitlab-prod-... [06:59:46] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Re-create GitLab test instance - https://phabricator.wikimedia.org/T396622#10940833 (10Jelto) 05Stalled→03Open The volume is available again and attached to the new test instance, I'll proceed setting up the test instance. [07:16:56] 10Phabricator maintenance bot, 10ContentTranslation: Complete enablement Section Translation in new wikis and make the process less manual for the future - https://phabricator.wikimedia.org/T371420#10940892 (10Nikerabbit) That change was partially reverted. `SectionTranslationTargetLanguages` is still in use b... [07:21:31] (03CR) 10Hashar: [C:03+2] Zuul: Add Yvarravy to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1163033 (owner: 10Pppery) [07:22:45] (03Merged) 10jenkins-bot: Zuul: Add Yvarravy to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1163033 (owner: 10Pppery) [07:41:49] (03PS1) 10Jeena Huneidi: Remove all references to patchdemo legacy [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1163289 (https://phabricator.wikimedia.org/T391866) [07:42:35] (03CR) 10CI reject: [V:04-1] Remove all references to patchdemo legacy [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1163289 (https://phabricator.wikimedia.org/T391866) (owner: 10Jeena Huneidi) [07:43:59] (03PS2) 10Jeena Huneidi: Remove all references to patchdemo legacy [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1163289 (https://phabricator.wikimedia.org/T391866) [07:49:42] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Migrate all CI jobs from Bookworm to Trixie or later and drop Bookworm testing support - https://phabricator.wikimedia.org/T397049#10940961 (10hashar) [07:51:22] 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure): TAR_ENTRY_ERROR ENOSPC: no space left on device (January 2025) (integration-agent-docker-1048) - https://phabricator.wikimedia.org/T384209#10940964 (10hashar) 05Open→03Resolved a:03hashar Thanks for the filing. The... [07:51:44] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all CI jobs from Bullseye to Bookworm or later and drop Bullseye testing support - https://phabricator.wikimedia.org/T383337#10940967 (10hashar) [07:53:27] 10Continuous-Integration-Infrastructure: Build failures due to "No space left on device" (August 2024) - https://phabricator.wikimedia.org/T373228#10940974 (10hashar) 05Open→03Resolved a:03hashar Thanks for the filing. The disk must have been cleaned at some point. The workers being overflowed should... [07:55:15] 10Continuous-Integration-Infrastructure, 10LibUp, 10MediaWiki-Vagrant: Uninstall jsduck and related material from CI images etc., once it's no longer needed - https://phabricator.wikimedia.org/T363905#10940980 (10hashar) As part of the #zuul-upgrade we are overhauling all jobs and support for jsduck will be... [08:13:22] 10Phabricator (Upstream), 07Upstream: Setting two workboard columns in one transaction: "RuntimeException: Undefined variable: board_phid" - https://phabricator.wikimedia.org/T397573#10941007 (10Aklapper) I cannot reproduce reliably. It blows up in the line `$board_phid = $column->getProjectPHID()` in `Maniphe... [08:24:33] (03approved) 10jnuche: Update scap beta release scripts [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/827 (owner: 10thcipriani) [08:35:22] (03CR) 10Hashar: [C:03+1] "Looks good to me, I am letting @ebomani@wikimedia.org to review the code since she did wrote the code to support both patch demos system 👍" [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1163289 (https://phabricator.wikimedia.org/T391866) (owner: 10Jeena Huneidi) [08:53:43] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s, 06serviceops: helmfile/scap does not reliably bootstrap mediawiki - https://phabricator.wikimedia.org/T397685 (10JMeybohm) 03NEW [09:11:55] 10Continuous-Integration-Infrastructure (Zuul upgrade): Define where to hold Zuul/Nodepool configuration - https://phabricator.wikimedia.org/T397688 (10hashar) 03NEW [09:53:11] 10Phabricator: Workboard triggers do not fire when using “Move on Workboard” - https://phabricator.wikimedia.org/T397697 (10Lucas_Werkmeister_WMDE) 03NEW [09:53:27] 10Phabricator: Workboard triggers do not fire when using “Move on Workboard” - https://phabricator.wikimedia.org/T397697#10941385 (10Lucas_Werkmeister_WMDE) Upstream task: none yet, I couldn’t find an existing one and my account is currently waiting for approval. [10:05:46] 10GitLab (Upstream pit of despair 🕳️), 07Upstream: Workboard triggers do not fire when using “Move on Workboard” - https://phabricator.wikimedia.org/T397697#10941483 (10Aklapper) [10:06:07] 10Phabricator (Upstream), 07Upstream: Workboard triggers should also fire when using “Move on Workboard” action dropdown - https://phabricator.wikimedia.org/T397697#10941485 (10Aklapper) [10:10:31] 10Phabricator (Upstream), 07Upstream: Workboard triggers should also fire when using “Move on Workboard” action dropdown - https://phabricator.wikimedia.org/T397697#10941501 (10Aklapper) > these triggers currently only fire when dragging a task into the column using drag’n’drop, not when using the “Move on Wor... [10:10:42] 10Phabricator (Upstream), 07Upstream: Workboard triggers should also fire when using “Move on Workboard” action dropdown - https://phabricator.wikimedia.org/T397697#10941502 (10Aklapper) p:05Triage→03Low [10:17:03] 10Phabricator, 06Release-Engineering-Team, 06Traffic: Phabricator videos fail in Firefox ("Range" request gets 503 from Varnish) - https://phabricator.wikimedia.org/T397661#10941540 (10Vgutierrez) FWIW I can't reproduce on Linux with firefox or chrome. With the provided curl reproducer I can see how the requ... [10:24:13] 10Phabricator, 06Security-Team, 07SecTeam-Processed, 07Security: Audit members of acl*security for more than 12 months of no activity (May 2025) - https://phabricator.wikimedia.org/T368224#10941597 (10Aklapper) 05Resolved→03Open @Jly: Hi, see bullet point 4 in the task description [10:58:39] 10Scap, 06serviceops: scap lock seems broken - https://phabricator.wikimedia.org/T397644#10941765 (10Clement_Goubert) Thank you! [11:18:56] (03PS1) 10Majavah: Zuul: Add Arendpieter to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1163334 [12:04:44] hashar: I'm trying to take a look at the quibble changes, but I keep getting rate-limited (429) cloning from diffusion. Is there something new there? My clone / quibble run is slowly making progress, but I keep having to retry with 30 min distance [12:05:51] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, 06Traffic: Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786#10942023 (10Aklapper) a:05Clement_Goubert→03None @Clement_Goubert Removing task assignee as this open task has been assigned for more than t... [12:07:13] 10Release-Engineering-Team (Priority Backlog 📥): Test out alternative k8s platforms for gitlab-cloud-runner - https://phabricator.wikimedia.org/T330520#10942063 (10Aklapper) a:05thcipriani→03None @thcipriani Removing task assignee as this open task has been assigned for more than two years - See the email se... [12:17:23] codders: what are you trying to clone? [12:17:37] quibble is trying to clone the world [12:17:54] it clones from Gerrit [12:18:01] how is diffusion involved? [12:18:38] hmm. Well, maybe I'm missing some environment when I run quibble locally - it clones submodules of Wikibase directly from diffusion [12:18:40] (over http) [12:18:56] for god sake [12:19:11] that stupid "lets migrate off of Gerrit to Differential" project of doom is still haunting us [12:19:38] anyway [12:19:50] yes there is a task to get rid of Phabricator as an intermediate [12:20:12] you are probably rate limited due to a counter measure to attempt to prevent scrapper from killing phabricator in a massive ddos [12:20:17] for sure [12:20:30] but somehow it seems new - I never hit it before [12:20:30] and since the git protocol does a ton of requests, that ends up triggering the limit [12:20:43] I did identify that for Gerrit and we went to use a different limiting factor [12:21:08] 10Beta-Cluster-Infrastructure, 06Data-Persistence, 06SRE, 06Traffic: ATS isn't caching documents in deployment-cache-upload07 - https://phabricator.wikimedia.org/T322575#10942159 (10Aklapper) a:05Vgutierrez→03None @Vgutierrez: Removing task assignee as this open task has been assigned for more than two... [12:21:36] (03PS1) 10Cmelo: Add CLDR as a dependency for CampaingEvents extension [integration/config] - 10https://gerrit.wikimedia.org/r/1163347 (https://phabricator.wikimedia.org/T397271) [12:22:05] (it just cloned the last repo, so for now I'm fine as long as I don't blow away src again) [12:25:22] ah [12:26:48] codders: you can file a task which should be tagged #Phabricator and #collaboration-services , the later is the SRE team managing Phab (they are in #wikimedia-sre-collab ) . Possibly file a security task so you can paste your IP in there [12:26:56] the ref is https://phabricator.wikimedia.org/T365259#10116300 [12:27:04] that is the post I was looking for the last few days [12:27:05] nice - thank you! [12:27:23] that post is for Gerrit [12:27:29] on which being hammered at a high rate is fine [12:27:45] but the number of concurrent threads should be limited [12:31:57] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Decrease number of open Phab tickets with assignee field set for more than two years (aka cookie licking) (Q4/2025 edition) - https://phabricator.wikimedia.org/T397713 (10Aklapper) 03NEW p:05Triage→03Low [12:32:07] 10Phabricator, 10Release-Engineering-Team (Doing 😎): Decrease number of open Phab tickets with assignee field set for more than two years (aka cookie licking) (Q2/2025 edition) - https://phabricator.wikimedia.org/T380312#10942546 (10Aklapper) 05In progress→03Resolved Done. [12:33:00] doing a git clone and submodule I get the same codders [12:33:03] Entering 'view/lib/wikibase-data-values' [12:33:03] fatal: unable to access 'https://phabricator.wikimedia.org/source/datavalues-javascript.git/': The requested URL returned error: 429 [12:33:11] yeah - exactly that. Thanks for the STR :) [12:33:42] have you file a task ?:) [12:33:49] not yet - was just writing in [12:33:52] it [12:33:56] great :) [12:34:01] make sure it is a security one :) [12:34:06] k [12:35:27] that kinda sounds again like https://phabricator.wikimedia.org/T349921 and https://phabricator.wikimedia.org/T362401 [12:35:42] codders, hashar ^ [12:36:00] I did just a few fetches and I got blocked as well [12:36:47] so... new issue? Or just detail for T349921 [12:36:48] T349921: integration-agent-docker machines excessively pull some Wikibase related Git repos in Diffusion - https://phabricator.wikimedia.org/T349921 [12:37:21] T349921 is "internal" traffic being throttled [12:37:39] okay. well, I'll make this issue and everyone can have fun closing or commenting :) [12:38:31] hehe [12:39:14] andre: I think T349921 got fixed. From what I get that was all traffics originating from WMCS (where Jenkins agents are) ended up originating from the same IP address which was not in an allow list and would end up being rate limited [12:39:52] andre: same thing happened for Gerrit, https://phabricator.wikimedia.org/T392467#10770981 . This time that is the allow list that did not get updated when switching over datacenters [12:40:00] ah [12:40:02] so essentially I think T349921 can be resolved [12:40:54] I would love to eventually remove serving git repos from Phabricator [12:40:57] https://phabricator.wikimedia.org/T397714 [12:41:00] there must not be that many uses cases [13:01:21] I have poked SRE about the task :) [13:01:31] thank you! [13:02:42] codders: and if you feel like playing with Quibble I have a couple changes pending that could use testing/a second pair of eyes https://gerrit.wikimedia.org/r/q/project:integration/quibble+is:open [13:02:51] none of them are urgent :] [13:03:19] yeah - that's why I started playing in the first place - because you tagged me on the review for the revert [13:05:19] :-] [13:06:43] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, 06Traffic: Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786#10942721 (10Clement_Goubert) a:03Clement_Goubert [13:12:56] (03CR) 10Arthur taylor: [C:03+1] "tested locally at it seems to work for me. Looks good from my side!" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1162903 (https://phabricator.wikimedia.org/T396670) (owner: 10Hashar) [13:25:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [13:25:38] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T397717 (10wmcs-alerts) 03NEW [13:32:18] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, 06Traffic: Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786#10942871 (10Clement_Goubert) 05In progress→03Resolved [13:37:54] (03CR) 10Hashar: [C:03+2] "Great, thank you for the additional verification!! :)" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1162903 (https://phabricator.wikimedia.org/T396670) (owner: 10Hashar) [13:40:22] (03CR) 10Arthur taylor: [C:03+1] "tested locally - seems to improve things without breaking anything. Thanks!" [integration/quibble] - 10https://gerrit.wikimedia.org/r/903256 (owner: 10Hashar) [13:56:18] (03Merged) 10jenkins-bot: Revert "Add support for importlib.resources for pkg_resources deprecation" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1162903 (https://phabricator.wikimedia.org/T396670) (owner: 10Hashar) [14:22:12] 10Continuous-Integration-Infrastructure (Zuul upgrade): Artifact storage for updated Zuul CI - https://phabricator.wikimedia.org/T397385#10943097 (10hashar) > From the S3 point of view, Zuul would need a user account on the `apus` cluster, and it would then create the `cilogs` (or whichever name) bucket and set... [14:24:00] 10Continuous-Integration-Infrastructure (Zuul upgrade): Artifact storage for updated Zuul CI - https://phabricator.wikimedia.org/T397385#10943105 (10hashar) Side track, Zuul has a playbook to upload to Swift: [[ https://zuul-ci.org/docs/zuul-jobs/latest/log-roles.html#role-upload-logs-swift | upload-logs-swift ]... [14:33:25] (03CR) 10Hashar: [C:03+2] "Awesome, thank you for the test!!!" [integration/quibble] - 10https://gerrit.wikimedia.org/r/903256 (owner: 10Hashar) [14:33:52] 10Continuous-Integration-Infrastructure (Zuul upgrade): Artifact storage for updated Zuul CI - https://phabricator.wikimedia.org/T397385#10943125 (10Corvus) I believe that, in general, radosgw can support the swift object expiration header, but I don't know if support for that needs to be explicitly enabled, or... [14:37:18] (03Merged) 10jenkins-bot: Accept multiple --run --skip [integration/quibble] - 10https://gerrit.wikimedia.org/r/903256 (owner: 10Hashar) [14:38:44] that patch has merged in less than 4 minutes thanks to dduvall success cache system :] [14:48:28] (03PS1) 10Hashar: Drop six dependency by using Python 3 dict.items() [integration/quibble] - 10https://gerrit.wikimedia.org/r/1163388 [14:49:48] 10Phabricator (Upstream), 07Upstream: Workboard triggers should also fire when using “Move on Workboard” action dropdown - https://phabricator.wikimedia.org/T397697#10943178 (10Lucas_Werkmeister_WMDE) >>! In T397697#10941385, @Lucas_Werkmeister_WMDE wrote: > Upstream task: none yet, I couldn’t find an existing... [14:51:23] 10Phabricator (phabricator-next), 06Release-Engineering-Team: Deploy Phabricator/Phorge 2025-06-24 - https://phabricator.wikimedia.org/T397726 (10brennen) 03NEW [14:51:41] 10Phabricator (Upstream), 07Upstream: Workboard triggers should also fire when using “Move on Workboard” action dropdown - https://phabricator.wikimedia.org/T397697#10943196 (10Aklapper) @Lucas_Werkmeister_WMDE I added you to Trusted-Contributors in upstream, feel free to try again [14:54:06] hashar: See T397653 - there might be new and exciting CI demands around ARM support coming from SRE soon. [14:54:06] T397653: Incorporate new arm64 host in our tooling - https://phabricator.wikimedia.org/T397653 [14:54:13] (03open) 10brennen: update submodules for 2025-06-24 release [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/72 (https://phabricator.wikimedia.org/T397726) [14:56:27] James_F: are they racking MacBooks? :] [14:56:51] hashar: No, I think they're probably from Dell/HP like our other prod servers. :-) [14:57:07] 10Phabricator (Upstream), 07Upstream: Workboard triggers should also fire when using “Move on Workboard” action dropdown - https://phabricator.wikimedia.org/T397697#10943235 (10Lucas_Werkmeister_WMDE) Thanks! [14:57:11] 10Phabricator (Upstream), 07Upstream: Workboard triggers should also fire when using “Move on Workboard” action dropdown - https://phabricator.wikimedia.org/T397697#10943236 (10Lucas_Werkmeister_WMDE) [14:58:14] (03CR) 10Ahmon Dancy: [C:03+1] "Typo in the commit message. Otherwise looks good to me." [integration/quibble] - 10https://gerrit.wikimedia.org/r/1163388 (owner: 10Hashar) [14:58:38] (03PS2) 10Hashar: Drop six dependency by using Python 3 dict.items() [integration/quibble] - 10https://gerrit.wikimedia.org/r/1163388 [14:59:30] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#10943253 (10Corvus) Note: Zuul doesn't have the ability to use passphrases with any of the ssh keys it uses (for nodepool, or gerrit). [15:03:03] (03merge) 10brennen: update submodules for 2025-06-24 release [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/72 (https://phabricator.wikimedia.org/T397726) [15:03:46] 10Phabricator (2025-06-24), 06Release-Engineering-Team, 13Patch-For-Review: Deploy Phabricator/Phorge 2025-06-24 - https://phabricator.wikimedia.org/T397726#10943293 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=52ad7c30-8002-4f7c-8efb-4650d2a3e70a) set by aokoth@cumin1002 for 0:30:00 o... [15:04:13] (03CR) 10Ahmon Dancy: Drop six dependency by using Python 3 dict.items() (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/1163388 (owner: 10Hashar) [15:05:07] 10Phabricator (2025-06-24), 06Release-Engineering-Team, 13Patch-For-Review: Deploy Phabricator/Phorge 2025-06-24 - https://phabricator.wikimedia.org/T397726#10943305 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a2029b60-687f-49e5-afb6-af916e2cb164) set by aokoth@cumin1002 for 0:30:00 o... [15:07:35] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog 📥): Experiment with Quibble job migration - https://phabricator.wikimedia.org/T392117#10943311 (10hashar) Things to tackle: * injecting dependencies (we use parameters functions) * reproducing `wmf-quibble*` us... [15:08:26] 10Phabricator (2025-06-24), 10Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Deploy Phabricator/Phorge 2025-06-24 - https://phabricator.wikimedia.org/T397726#10943315 (10Aklapper) 05Open→03Resolved a:03brennen This deployment included: * AVA: {T396607} * A fix for upstream https://we.phorg... [15:11:12] 10Beta-Cluster-Infrastructure, 10GitLab (CI & Job Runners), 10m3api: Unable to connect to Beta Cluster from WMCS GitLab CI runners - https://phabricator.wikimedia.org/T397591#10943336 (10LucasWerkmeister) According to https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner/Security_Evaluation#Firewall_confi... [15:34:39] 10Beta-Cluster-Infrastructure, 10GitLab (CI & Job Runners), 06collaboration-services, 10m3api: Unable to connect to Beta Cluster from WMCS GitLab CI runners - https://phabricator.wikimedia.org/T397591#10943435 (10Jelto) [15:38:56] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog 📥): Experiment with Quibble job migration - https://phabricator.wikimedia.org/T392117#10943458 (10thcipriani) >>! In T392117#10943311, @hashar wrote: > Things to tackle: > * injecting dependencies (we use param... [16:00:33] 10Phabricator, 06Security-Team, 07SecTeam-Processed, 07Security: Audit members of acl*security for more than 12 months of no activity (May 2025) - https://phabricator.wikimedia.org/T368224#10943527 (10Jly) 05Open→03Resolved My apologies. T397734 has been opened and I will proceed to close this one... [18:15:38] (03PS2) 10Daimona Eaytoy: zuul: [mediawiki/extensions/CampaignEvents] add CLDR as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1163347 (https://phabricator.wikimedia.org/T397271) (owner: 10Cmelo) [18:15:56] (03CR) 10Daimona Eaytoy: [C:03+1] zuul: [mediawiki/extensions/CampaignEvents] add CLDR as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1163347 (https://phabricator.wikimedia.org/T397271) (owner: 10Cmelo) [19:55:48] 10Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T392178#10944565 (10jeena) [20:04:53] 10Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T392178#10944590 (10jeena) [20:05:03] 10Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T392177#10944593 (10jeena) [20:42:28] FIRING: InstanceDown: Project deployment-prep instance deleteme is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:42:39] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deleteme is down - https://phabricator.wikimedia.org/T397777 (10wmcs-alerts) 03NEW [20:47:28] RESOLVED: InstanceDown: Project deployment-prep instance deleteme is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:47:33] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deleteme is down - https://phabricator.wikimedia.org/T397777#10944875 (10wmcs-alerts) [21:18:19] (03open) 10annet: releases: Bump Codex to 2.2.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/82 (https://phabricator.wikimedia.org/T397759) [21:18:50] (03approved) 10catrope: releases: Bump Codex to 2.2.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/82 (https://phabricator.wikimedia.org/T397759) (owner: 10annet) [21:18:55] (03merge) 10catrope: releases: Bump Codex to 2.2.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/82 (https://phabricator.wikimedia.org/T397759) (owner: 10annet) [22:13:20] (03open) 10dancy: spiderpig: Revise HTTP access logging [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/858 [22:13:23] (03update) 10dancy: spiderpig: Revise HTTP access logging [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/858 [22:15:29] (03update) 10dancy: spiderpig: Revise HTTP access logging [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/858 [22:15:36] (03update) 10dancy: spiderpig: Revise HTTP access logging [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/858 [22:15:56] (03update) 10dancy: spiderpig: Revise HTTP access logging [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/858 [22:50:03] 10Beta-Cluster-Infrastructure, 10GitLab (CI & Job Runners), 06collaboration-services, 10m3api: Unable to connect to Beta Cluster from WMCS GitLab CI runners - https://phabricator.wikimedia.org/T397591#10945262 (10bd808) I'm not really sure that I agree with the analysis by @Jelto from T317341#8324825 that... [22:53:40] 10Beta-Cluster-Infrastructure, 10GitLab (CI & Job Runners), 06collaboration-services, 10m3api: Unable to connect to Beta Cluster from WMCS GitLab CI runners - https://phabricator.wikimedia.org/T397591#10945277 (10bd808) {T374129} is the task to remove the split horizon DNS resolution magic from Cloud VPS g... [23:36:05] 10Continuous-Integration-Infrastructure (Zuul upgrade), 07Upstream: terraform-provider-openstack >v3.0.0 uses a gophercloud client that does not work with WMCS Magnum APIs - https://phabricator.wikimedia.org/T397106#10945398 (10bd808) >>! In T397106#10924766, @Andrew wrote: > Opened gophercloud issue here: htt... [23:56:05] 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Provision Kubernetes cluster and bastion using OpenTofu and Magnum - https://phabricator.wikimedia.org/T396936#10945434 (10bd808) >>! In T396936#10937326, @taavi wrote: > If Magnum doesn't support dual-stack clusters then I think I con...