[02:11:36] 10Beta-Cluster-Infrastructure, 10NavigationTiming, 06SRE Observability: navtiming: Loss of Kafka connection fills multiple log files with identical stack traces - https://phabricator.wikimedia.org/T391273#10882405 (10Krinkle) >>! In T391273#10880614, @taavi wrote: > I think the ultimate issue here is tha... [05:11:56] 10Continuous-Integration-Infrastructure, 10Gerrit, 07Jenkins, 10MediaWiki-extensions-Springboard: Commits to Springboard repo on Gerrit not getting handled - https://phabricator.wikimedia.org/T395981 (10Yaron_Koren) 03NEW [05:17:48] 10Continuous-Integration-Infrastructure, 10Gerrit, 07Jenkins, 10MediaWiki-extensions-Springboard: Commits to Springboard repo on Gerrit not getting handled - https://phabricator.wikimedia.org/T395981#10882520 (10Samwilson) Has it been added to the integratin/config repo? https://www.mediawiki.org/wiki/Cont... [05:18:10] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 06Data-Persistence, 10Data-Persistence-Backup: Request mariadb database for Zuul - https://phabricator.wikimedia.org/T394844#10882522 (10Marostegui) @Dzahn did you choose m3 for any specific reason? We only have phabricator i... [05:38:59] 10Continuous-Integration-Infrastructure, 10Gerrit, 07Jenkins, 10MediaWiki-extensions-Springboard: Commits to Springboard repo on Gerrit not getting handled - https://phabricator.wikimedia.org/T395981#10882529 (10Yaron_Koren) Oh, thanks, I should have checked that Zuul file. Yes, indeed, Springboard is miss... [07:09:38] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#10882674 (10LSobanski) p:05Triage→03High [07:44:00] 10Continuous-Integration-Config: Commits to Springboard repo on Gerrit not getting handled - https://phabricator.wikimedia.org/T395981#10882742 (10taavi) [07:44:18] 10Continuous-Integration-Config, 10MediaWiki-extensions-Springboard: Commits to Springboard repo on Gerrit not getting handled - https://phabricator.wikimedia.org/T395981#10882743 (10taavi) [07:59:47] (03CR) 10Hashar: [C:03+2] Add start-zuul-console to pre-run playbook [integration/config] (zuul3) - 10https://gerrit.wikimedia.org/r/1153391 (owner: 10Corvus) [07:59:54] (03CR) 10Hashar: [V:03+2 C:03+2] Add start-zuul-console to pre-run playbook [integration/config] (zuul3) - 10https://gerrit.wikimedia.org/r/1153391 (owner: 10Corvus) [08:01:35] (03CR) 10CI reject: [V:04-1] Add start-zuul-console to pre-run playbook [integration/config] (zuul3) - 10https://gerrit.wikimedia.org/r/1153391 (owner: 10Corvus) [08:30:33] 10Deployments, 10Release-Engineering-Team (Radar), 06serviceops, 13Patch-For-Review, 07Wikimedia-production-error: httpb sometimes fails upon deployment with a HTTP 503 - https://phabricator.wikimedia.org/T380958#10882837 (10akosiaris) This is surfacing once every couple of days or so, at least per [Logs... [09:02:03] 10Beta-Cluster-Infrastructure: CampaignEvents: Error 1091: Can't DROP COLUMN `event_type`; check that it exists - https://phabricator.wikimedia.org/T394418#10882915 (10hashar) Thank you to have digged into it :) [09:19:05] 10GitLab (Infrastructure), 10Release-Engineering-Team (Radar), 10ChangeProp, 06cloud-services-team, and 10 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#10882980 (10hashar) 05Open→03Resolved a:03hashar After chatting wit... [09:31:27] 10Continuous-Integration-Infrastructure, 10Castor: Castor rsync causes: rsync: failed to set times on "/cache/.": Operation not permitted (1) - https://phabricator.wikimedia.org/T188488#10883021 (10hashar) >>! In T188488#10876686, @matmarex wrote: > Thanks, this has been bugging me for years! All credits... [09:34:01] how do I tell debian-glue that bookworm is in fact a debian release? https://gerrit.wikimedia.org/r/c/labs/toollabs/+/1153344 [09:40:16] 10Continuous-Integration-Infrastructure (Zuul upgrade): Install a microK8s instance on WMCS project - https://phabricator.wikimedia.org/T395826#10883033 (10hashar) My first pass was to have Nodepool to provide a Kubernetes namespace, and then the job playbook would create the container. But that means the contai... [09:43:49] 10Continuous-Integration-Infrastructure (Zuul upgrade): Install a microK8s instance on WMCS project - https://phabricator.wikimedia.org/T395826#10883037 (10hashar) The Nodepool provider section looks like: ` lang=yaml,name=nodepool.yaml providers: - name: microk8s driver: kubernetes context: microk8s... [09:55:19] 10Continuous-Integration-Infrastructure (Zuul upgrade): Install a microK8s instance on WMCS project - https://phabricator.wikimedia.org/T395826#10883051 (10hashar) 05Open→03Resolved a:03hashar [10:36:49] 10Continuous-Integration-Infrastructure (Zuul upgrade): Install a microK8s instance on WMCS project - https://phabricator.wikimedia.org/T395826#10883203 (10hashar) >>! In T395826#10883032, @hashar wrote: > There were some issue due to our images having `USER nobody`: > * Ansible attempts to create `/root/.an... [12:10:01] !log lucaswerkmeister@deployment-deploy04:~$ mwscript createAndPromote commonswiki --interface-admin --force 'Lucas Werkmeister' # w-beta.wmflabs.org/mt [12:10:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:10:46] w-beta doesn't seem to work [12:12:10] yeah I noticed that too [12:14:26] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012 (10LucasWerkmeister) 03NEW [12:14:26] filed T396012 [12:14:27] T396012: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012 [12:16:25] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10883527 (10LucasWerkmeister) Workaround: manually rewrite the URL to the correct Special:UrlRedirector URL, such as https://meta.wikimedia.beta... [12:31:23] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10883650 (10LucasWerkmeister) I tried to look into this but didn’t get anywhere. I can’t even find the “Web Proxies” menu under “Network” anymor... [12:48:30] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10883699 (10Jelto) Total number of artifacts was reduced from 400k to around 100k in T395014. I enabled o... [12:48:44] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10883700 (10Jelto) [13:08:06] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Check GitLab artifact retention time - https://phabricator.wikimedia.org/T395014#10883769 (10Jelto) The cleanup of old artifacts is done, the number of artifacts was reduced from 400k to around 100k. With `... [13:17:28] <_joe_> A job on a merge request of mine keeps failing at the kokkuri:setup-variables stage https://gitlab.wikimedia.org/oblivian/airflow-dags/-/jobs/527368 [13:17:57] <_joe_> I see a 500 internal server error response there, but the logs aren't very clear about what's failing [13:18:38] <_joe_> actually, that's gitlab I'd say [13:18:41] <_joe_> jelto: ^^ [13:21:30] Hmmm I'm currently migrating artifacts from disk to object storage for GitLab. This should not impact upload of new artifacts but apparently it is. I'll open a sub task [13:24:15] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018 (10Jelto) 03NEW [13:28:26] <_joe_> jelto: as long as I know it's a known migration, I'm happy to wait [13:29:28] yes I started the migration 45 minutes ago, rough guess is it needs another 120 minutes. I'd like to avoid blocking CI for that long. I'll check if there is anything we can do [13:32:50] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10883933 (10Jelto) p:05Triage→03High The migration of objects is running currently. 25k of the 100k artifacts are migrated already and this... [13:49:26] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10884065 (10Jelto) I stopped the migration, the expected time would be Wed 04 Jun 2025 14:54:15 UTC (so over an hour). I re-tried the failing j... [14:03:23] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10884137 (10Jelto) Unfortunately migrating back to local storage is even slower, estimated date is Wed 04 Jun 2025 17:07:15 UTC. So GitLab CI ar... [14:17:11] (03open) 10daimona: releases: Bump all wdio packages to v9 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/77 [14:17:38] (03update) 10daimona: releases: Bump all wdio packages to v9 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/77 (https://phabricator.wikimedia.org/T363704) [14:18:09] (03update) 10daimona: releases: Bump all wdio packages to v9 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/77 (https://phabricator.wikimedia.org/T363704) [14:30:04] Daimona: I am pretty sure we can not just bump wdio :) [14:31:09] Peter & Testing Platform are working on it though :) [14:31:45] Why not? It seemed to work basically everywhere. Except for the weird stuff (like using removed functionality), but those will fail and then the patch can be refined manually as we usually do (same as phan, for instance) [14:52:35] (03update) 10daimona: releases: Bump all wdio packages to v9 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/77 (https://phabricator.wikimedia.org/T363704) [14:53:32] (03update) 10daimona: releases: Bump all wdio packages to v9 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/77 (https://phabricator.wikimedia.org/T363704) [14:53:48] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10884348 (10Jelto) Migration is still running, estimated date is 17:07 UTC. Logstash dashboard with some of the 500 requests: https://logstash.... [14:56:51] Daimona: because Testing Platform team has a roadmap to upgrade webdriver already ( 8 is https://phabricator.wikimedia.org/T324685 and 9 is https://phabricator.wikimedia.org/T363704 ) [14:57:22] but you can sync up with them for sure! :) [14:57:43] maybe it is indeed as easy has bumping the wdio package and wdio-mediawiki [14:58:41] (03CR) 10Hashar: [C:03+2] Revert "jjb: drop obsolete creation of log & src dirs" [integration/config] - 10https://gerrit.wikimedia.org/r/1152761 (https://phabricator.wikimedia.org/T282893) (owner: 10Hashar) [14:58:47] I'm under the impression that we were going to v9 straight. At any rate, I think LibUp would provide a useful starting point with the patches, regardless of who does the migration. If anything, it should save that person from making dozens of identical commits and let them only worry about stuff that needs to be cleaned up manually ;) [14:59:42] (03CR) 10Hashar: [C:03+2] "This was deployed immediately, but I forgot to +2 😂" [integration/config] - 10https://gerrit.wikimedia.org/r/1152761 (https://phabricator.wikimedia.org/T282893) (owner: 10Hashar) [15:00:08] (03Merged) 10jenkins-bot: Revert "jjb: drop obsolete creation of log & src dirs" [integration/config] - 10https://gerrit.wikimedia.org/r/1152761 (https://phabricator.wikimedia.org/T282893) (owner: 10Hashar) [15:04:10] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10884399 (10Jelto) In https://docs.gitlab.com/administration/object_storage/ I found the information that `host` parameter is deprecated in `git... [15:06:51] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 13Patch-For-Review: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10884402 (10Ladsgroup) With those patches cherry-picked https://w.beta.wmcloud.org/1 works but TLS cert doesn't include th... [15:12:59] 10Beta-Cluster-Infrastructure: Determine if cherry-pick for kafka_config.rb can be removed/replaced in deployment-prep - https://phabricator.wikimedia.org/T396042 (10bd808) 03NEW [15:15:34] 10Beta-Cluster-Infrastructure: Determine if cherry-pick for kafka_config.rb can be removed/replaced in deployment-prep - https://phabricator.wikimedia.org/T396042#10884472 (10bd808) The patch looks like it was really just a replacement for making some hiera changes in deployment-prep. Maybe @Ottomata can help us... [15:16:43] 10Beta-Cluster-Infrastructure: Determine if cherry-pick for kafka_config.rb can be removed/replaced in deployment-prep - https://phabricator.wikimedia.org/T396042#10884473 (10taavi) IHMO that function should be changed to properly fail if the cluster in case is not defined. The current behaviour of defaulting to... [15:20:13] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10884482 (10Jelto) I tried different combinations of `endpoint`, `enable_signature_v4_streaming` and `path_style` with no success. I'll continue... [16:38:49] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 13Patch-For-Review: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10884960 (10LucasWerkmeister) Okay… is any of it going to make https://w-beta.wmflabs.org/1 work, i.e. the URL generated b... [16:38:56] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 13Patch-For-Review: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10884962 (10LucasWerkmeister) [16:40:53] 10Phabricator: Vanishing request for User Snowyamur9889 - https://phabricator.wikimedia.org/T396053 (10MdsShakil) 03NEW [16:44:37] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 13Patch-For-Review: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10884999 (10Ladsgroup) >>! In T396012#10884960, @LucasWerkmeister wrote: > Okay… is any of it going to make https://w-beta... [16:46:30] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 13Patch-For-Review: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10885004 (10Jdforrester-WMF) >>! In T396012#10884999, @Ladsgroup wrote: >>>! In T396012#10884960, @LucasWerkmeister wrote:... [17:04:04] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 13Patch-For-Review: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10885051 (10Ladsgroup) If I were to be very picky, no since this is w-beta (not a subdomain) but regardless, wmflabs is de... [17:16:25] 10Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T392174#10885123 (10Dreamy_Jazz) [17:47:41] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 06Data-Persistence, 10Data-Persistence-Backup: Request mariadb database for Zuul - https://phabricator.wikimedia.org/T394844#10885262 (10Dzahn) Yes, we can do m1 instead. It does not have to be m3. It's up to you. The string... [17:51:33] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 13Patch-For-Review: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10885293 (10Ladsgroup) I love debugging TLS, I barely can contain myself right now. I added the domain to acme-chief confi... [18:38:37] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10885455 (10Jelto) The migration is still running [18:50:37] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10885474 (10Jelto) p:05High→03Medium I disabled object storage again and the issue is fixed. I'll follow up tomorrow a... [18:50:57] GitLab CI issue is fixed, you should be able to upload artifacts again [19:08:37] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10885527 (10Jdforrester-WMF) Thanks, confirm this has fixed CI / merges for us. [19:10:23] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06SRE, 10SRE-Access-Requests: Requesting access to contint-roots for Corvus - https://phabricator.wikimedia.org/T395167#10885542 (10KFrancis) Hi all, the NDA has been signed. Thanks! [19:13:45] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06SRE, 10SRE-Access-Requests: Requesting access to contint-roots for Corvus - https://phabricator.wikimedia.org/T395167#10885557 (10Dzahn) @KFrancis Thank you. Has it been added to the NDA/MOU spreadsheet? [19:29:17] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06SRE, 10SRE-Access-Requests: Requesting access to contint-roots for Corvus - https://phabricator.wikimedia.org/T395167#10885633 (10KFrancis) Just added! [19:30:08] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06SRE, 10SRE-Access-Requests: Requesting access to contint-roots for Corvus - https://phabricator.wikimedia.org/T395167#10885637 (10Dzahn) Confirmed:) thanks again. Moving the ticket forward. [19:30:49] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06SRE, 10SRE-Access-Requests: Requesting access to contint-roots for Corvus - https://phabricator.wikimedia.org/T395167#10885641 (10Dzahn) [19:31:24] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06SRE, 10SRE-Access-Requests: Requesting access to contint-roots for Corvus - https://phabricator.wikimedia.org/T395167#10885645 (10Dzahn) [20:12:13] 10Beta-Cluster-Infrastructure, 10MediaWiki-User-management, 07Beta-Cluster-reproducible, 13Patch-For-Review: Inconsistent user permissions for users who were recently added to a new group (June 2025 edition) - https://phabricator.wikimedia.org/T396061#10885746 (10A_smart_kitten) [20:39:28] 10Deployments, 10Release-Engineering-Team (Radar), 06serviceops, 07Wikimedia-production-error: httpb sometimes fails upon deployment with a HTTP 503 - https://phabricator.wikimedia.org/T380958#10885815 (10cjming) encountered this deploying today - https://spiderpig.wikimedia.org/jobs/154 {F61547159} not... [20:49:51] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to contint-roots for Corvus - https://phabricator.wikimedia.org/T395167#10885837 (10Dzahn) a:05KFrancis→03None [21:13:43] (03PS1) 10QChris: Zuul: [mediawiki/extensions/Springboard] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1153708 (https://phabricator.wikimedia.org/T395981) [21:25:19] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/Springboard] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1153708 (https://phabricator.wikimedia.org/T395981) (owner: 10QChris) [21:27:23] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Springboard] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1153708 (https://phabricator.wikimedia.org/T395981) (owner: 10QChris) [21:27:44] !log Zuul: [mediawiki/extensions/Springboard] Add basic CI, for T395981 [21:27:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:27:46] T395981: Commits to Springboard repo on Gerrit not getting handled - https://phabricator.wikimedia.org/T395981 [21:49:24] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 13Patch-For-Review: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10886033 (10Ladsgroup) Yup, the new certificate has been issued, I'm seeing the new one with this domain included, works o... [21:53:29] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 13Patch-For-Review: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10886037 (10LucasWerkmeister) So my [SAL message](https://sal.toolforge.org/log/jTfZOpcBffdvpiTrMUWl) is just going to hav... [22:01:10] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 13Patch-For-Review: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10886055 (10Dzahn) >>! In T396012#10885051, @Ladsgroup wrote: > Also this is a testing setup I don't see the point of tryi... [22:01:47] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 13Patch-For-Review: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10886057 (10taavi) I set up a redirect via the https://wikitech.wikimedia.org/wiki/Nova_Resource:Redirects infrastructure. [22:03:03] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 13Patch-For-Review: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10886060 (10LucasWerkmeister) >>! In T396012#10886055, @Dzahn wrote: > But in the future just use the actual production UR... [22:07:28] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 13Patch-For-Review: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10886062 (10Dzahn) >>! In T396012#10886060, @LucasWerkmeister wrote: > I tried that one first, of course, but it doesn’t a... [22:31:21] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 13Patch-For-Review: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10886093 (10bd808) >>! In T396012#10886062, @Dzahn wrote: >>>! In T396012#10886060, @LucasWerkmeister wrote: >> I tried th... [22:32:31] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 13Patch-For-Review: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10886096 (10LucasWerkmeister) >>! In T396012#10886057, @taavi wrote: > I set up a redirect via the https://wikitech.wikime... [23:17:56] 10Beta-Cluster-Infrastructure: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318#10886173 (10Krinkle) [23:31:07] 10Beta-Cluster-Infrastructure: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318#10886188 (10Krinkle) Strawman: 1. **MediaWiki**. Update multiversion routing to handle both domain suffixes. This is fairly straight forward (The "hard" problem with renames in MediaWiki is w... [23:40:06] (03update) 10dduvall: artifacts: Support `exclude` patterns for copies/requirements [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/123 [23:40:09] (03update) 10dduvall: artifacts: Support `exclude` patterns for copies/requirements [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/123 [23:40:10] (03update) 10dduvall: artifacts: Support `exclude` patterns for copies/requirements [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/123 [23:49:37] 10Beta-Cluster-Infrastructure: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318#10886208 (10bd808) >>! In T289318#10886188, @Krinkle wrote: > 2. **TLS (Puppet?)**. The various layers will need to acquire the right certificates, which, like prod, is non-trivial because the...