[01:12:11] 10Continuous-Integration-Config, 06MediaWiki-Platform-Team, 10MediaWiki-Vendor, 10Quibble: mediawiki/core and mediawiki/vendor both skip composer.lock checks - https://phabricator.wikimedia.org/T370380#10005434 (10Krinkle) Thanks @hashar. Moving back to our workboard to verify on the MediaWiki side. When... [06:54:41] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [Mirji] - https://phabricator.wikimedia.org/T370744 (10M-J) 03NEW [06:57:07] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [mirji] - https://phabricator.wikimedia.org/T370744#10005711 (10M-J) [07:36:26] (03PS1) 10Gehel: Add credentials for publishing Maven artifacts to Gitlab. [integration/config] - 10https://gerrit.wikimedia.org/r/1056079 (https://phabricator.wikimedia.org/T367403) [07:37:25] 06Release-Engineering-Team, 06Data-Platform-SRE, 06Discovery-Search, 06Java-Scala-Standardization, and 2 others: Validate CI integration so that Ci can release Maven artifacts on user's demand - https://phabricator.wikimedia.org/T367403#10005759 (10Gehel) >>! In T367403#10003586, @amastilovic wrote: > Also... [07:37:49] hashar: 2 questions for your ^ (openssh host keys and additional configuration in maven's settings.xml) [08:19:34] 06Release-Engineering-Team, 10Castor, 07ci-test-error: Wikibase CI failing due to npm cache corruption issue - https://phabricator.wikimedia.org/T370748 (10Jakob_WMDE) 03NEW [08:28:34] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 10Wikimedia-Apache-configuration, and 3 others: Apache 2.4.61 throws a 403 Forbidden for links containing %3F - https://phabricator.wikimedia.org/T370110#10005864 (10hashar) Is the `B` flag the reason the issue trigge... [08:31:00] gehel: sorry I was in Apache Rewrite Rule config [08:31:49] hashar: no emergency here! [08:31:55] I will review, but at first glance I would add that gitlab server to the parent pom [08:32:05] but I need a coffee :D [08:32:16] As usual, I've sent a CR so that you can tell me what's wrong! [08:32:48] That part cannot be configured in a pom. All the authentication / authorization has to be separate from the project and in settings.xml [08:34:54] 06Release-Engineering-Team, 10Castor, 10Wikidata, 10ci-test-error (WMF-deployed Build Failure): Wikibase CI failing due to npm cache corruption issue - https://phabricator.wikimedia.org/T370748#10005908 (10Lucas_Werkmeister_WMDE) [09:06:16] (03CR) 10Hashar: "I thought it would be a better fit for the parent pom, my memory was of I1582af65cb39ec02a49dd402c3c43ffc994d0e61 which was proposing to a" [integration/config] - 10https://gerrit.wikimedia.org/r/1056079 (https://phabricator.wikimedia.org/T367403) (owner: 10Gehel) [09:07:46] oh and the image needs the gitlab host key :/ [09:18:30] gehel: do you still need to publish over ssh for the maven release? [09:18:38] cause I can surely add the gitlab ssh host key to the image [09:18:50] yes, we should have those! [09:18:57] so both [09:19:11] ssh to push the commit / tag [09:19:30] and the http header for pushing to the artifact store? [09:19:52] * hashar shakes fists at Docker [09:42:04] grr [09:42:09] I am going to refactor those maven images a bit [09:52:06] hashar: yes, both [09:55:38] 10Release-Engineering-Team (Seen), 10Data Pipelines, 06Data-Engineering, 10Data-Platform-SRE (2024.07.08 - 2024.07.28), 13Patch-For-Review: Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#10006086 (10Stevemunene) We are now upgrading to the latest version v2.9.3. There are still some... [09:59:04] (03PS1) 10Hashar: dockerfiles: add in maven-java17, share Gerrit host key [integration/config] - 10https://gerrit.wikimedia.org/r/1056127 (https://phabricator.wikimedia.org/T367403) [10:00:12] (03CR) 10CI reject: [V:04-1] dockerfiles: add in maven-java17, share Gerrit host key [integration/config] - 10https://gerrit.wikimedia.org/r/1056127 (https://phabricator.wikimedia.org/T367403) (owner: 10Hashar) [10:00:16] of course [10:05:20] 10Release-Engineering-Team (Seen), 10Data Pipelines, 06Data-Engineering, 10Data-Platform-SRE (2024.07.08 - 2024.07.28), 13Patch-For-Review: Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#10006135 (10Stevemunene) [10:11:11] bonjour hashar, wikibase CI has been failing since yesterday evening due to the recurring npm cache corruption issue. could you nuke the cache again to unblock us? (T370748) [10:11:12] T370748: Wikibase CI failing due to npm cache corruption issue - https://phabricator.wikimedia.org/T370748 [10:21:18] jakob_WMDE: ouch :/ [10:22:50] 06Release-Engineering-Team, 10Castor, 10Wikidata, 10ci-test-error (WMF-deployed Build Failure): Wikibase CI failing due to npm cache corruption issue - https://phabricator.wikimedia.org/T370748#10006218 (10hashar) p:05Triage→03Unbreak! a:03hashar [10:25:23] !log integration: nuked Castor cache /srv/castor/castor-mw-ext-and-skins/master/mwgate-node18 [10:25:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:28:08] jakob_WMDE: done [10:28:28] jakob_WMDE: and I think we should add a few WMDE people ot the integration project with `sudo` permission so that you can easily delete the caches [10:28:39] hashar: thank you! [10:28:50] yeah that might be helpful [10:29:19] 06Release-Engineering-Team, 10Castor, 10Wikidata, 10ci-test-error (WMF-deployed Build Failure): Wikibase CI failing due to npm cache corruption issue - https://phabricator.wikimedia.org/T370748#10006237 (10hashar) 05Open→03Resolved Hi, I have deleted the faulty cache. The process is to look up in... [10:30:15] jakob_WMDE: if you can ask around WMDE and file a task against #continuous-integration-infrastructure I will add to the WMCS project [10:30:34] short of having a better solution to solve the root cause :/ [10:31:03] I'll do that, thanks! [10:31:10] Danke Schon! [10:31:37] (03PS2) 10Hashar: dockerfiles: add in maven-java17, share Gerrit host key [integration/config] - 10https://gerrit.wikimedia.org/r/1056127 (https://phabricator.wikimedia.org/T367403) [11:06:42] (03CR) 10Hashar: [C:03+2] "I have tested all three images locally and they can ssh to Gerrit. That will make it easier to add the Gitlab host key in Icbe44b7609aecd" [integration/config] - 10https://gerrit.wikimedia.org/r/1056127 (https://phabricator.wikimedia.org/T367403) (owner: 10Hashar) [11:08:26] (03Merged) 10jenkins-bot: dockerfiles: add in maven-java17, share Gerrit host key [integration/config] - 10https://gerrit.wikimedia.org/r/1056127 (https://phabricator.wikimedia.org/T367403) (owner: 10Hashar) [11:19:31] (03PS1) 10Hashar: jjb: update Maven jobs for Gerrit host key refactoring [integration/config] - 10https://gerrit.wikimedia.org/r/1056135 (https://phabricator.wikimedia.org/T367403) [11:19:47] (03CR) 10Hashar: [C:03+2] jjb: update Maven jobs for Gerrit host key refactoring [integration/config] - 10https://gerrit.wikimedia.org/r/1056135 (https://phabricator.wikimedia.org/T367403) (owner: 10Hashar) [11:20:58] (03Merged) 10jenkins-bot: jjb: update Maven jobs for Gerrit host key refactoring [integration/config] - 10https://gerrit.wikimedia.org/r/1056135 (https://phabricator.wikimedia.org/T367403) (owner: 10Hashar) [12:13:26] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-sessionstore04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370461#10006532 (10Southparkfan) I didn't get a response in `-sre`, but A... [12:19:45] (03PS2) 10Hashar: dockerfiles: add GitLab ssh host keys to Maven images [integration/config] - 10https://gerrit.wikimedia.org/r/1056139 (https://phabricator.wikimedia.org/T367403) [12:20:04] (03CR) 10Hashar: [C:03+2] "I have tested it locally and the host keys do work ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/1056139 (https://phabricator.wikimedia.org/T367403) (owner: 10Hashar) [12:21:11] (03Merged) 10jenkins-bot: dockerfiles: add GitLab ssh host keys to Maven images [integration/config] - 10https://gerrit.wikimedia.org/r/1056139 (https://phabricator.wikimedia.org/T367403) (owner: 10Hashar) [12:34:16] 10Continuous-Integration-Infrastructure: Add WMDE staff working on development of Wikibase software - https://phabricator.wikimedia.org/T370766 (10WMDE-leszek) 03NEW [13:04:33] (03PS1) 10Pwangai: Zuul: [mediawiki/extensions/QuickSurveys] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1056149 (https://phabricator.wikimedia.org/T321837) [13:05:43] (03PS1) 10Pwangai: Zuul: [mediawiki/extensions/Quiz] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1056150 (https://phabricator.wikimedia.org/T321837) [13:06:53] (03PS1) 10Pwangai: Zuul: [mediawiki/extensions/Score] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1056151 (https://phabricator.wikimedia.org/T321837) [13:07:52] (03PS1) 10Pwangai: Zuul: [mediawiki/extensions/PropertySuggester] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1056152 (https://phabricator.wikimedia.org/T321837) [13:18:45] (03PS1) 10Hashar: jjb: update Maven jobs to add GitLab ssh host key [integration/config] - 10https://gerrit.wikimedia.org/r/1056154 (https://phabricator.wikimedia.org/T367403) [13:19:56] (03CR) 10Hashar: [C:03+2] jjb: update Maven jobs to add GitLab ssh host key [integration/config] - 10https://gerrit.wikimedia.org/r/1056154 (https://phabricator.wikimedia.org/T367403) (owner: 10Hashar) [13:21:15] (03Merged) 10jenkins-bot: jjb: update Maven jobs to add GitLab ssh host key [integration/config] - 10https://gerrit.wikimedia.org/r/1056154 (https://phabricator.wikimedia.org/T367403) (owner: 10Hashar) [13:55:06] (03CR) 10Hashar: [C:04-1] "After looking at the GitLab merge request https://gitlab.wikimedia.org/repos/maven/maven-test-project/-/merge_requests/1/diffs#note_95865 " [integration/config] - 10https://gerrit.wikimedia.org/r/1056079 (https://phabricator.wikimedia.org/T367403) (owner: 10Gehel) [13:55:42] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/PropertySuggester] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1056152 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [13:55:50] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/QuickSurveys] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1056149 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [13:55:55] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/Quiz] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1056150 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [13:56:00] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/Score] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1056151 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [13:56:53] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/PropertySuggester] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1056152 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [13:56:58] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/QuickSurveys] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1056149 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [13:57:12] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Quiz] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1056150 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [13:57:16] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Score] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/1056151 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [14:02:17] !log Reloaded Zuul to enable Sonar Codehalth on PropertySuggester, QuickSurveys, Quiz, Score # T321837 [14:02:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:02:19] T321837: Repositories integrated into Codehealth Pipeline (Production) - https://phabricator.wikimedia.org/T321837 [14:21:32] (03merge) 10brennen: Log creation when duplicating an EditEngine form [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/64 (https://phabricator.wikimedia.org/T369568) (owner: 10aklapper) [14:22:14] (03update) 10brennen: Disallow creating commit-based Herald rules [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/60 (https://phabricator.wikimedia.org/T367651) (owner: 10aklapper) [14:22:58] (03merge) 10brennen: Disallow creating commit-based Herald rules [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/60 (https://phabricator.wikimedia.org/T367651) (owner: 10aklapper) [14:25:33] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services: Deploy Phabricator/Phorge 2024-07-23 - https://phabricator.wikimedia.org/T370776 (10brennen) 03NEW [14:26:21] (03update) 10brennen: Uninstall Audit application [repos/phabricator/deployment] (work/2024-07-23-release) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/48 (https://phabricator.wikimedia.org/T330794) (owner: 10aklapper) [14:26:35] (03update) 10brennen: Uninstall Audit application [repos/phabricator/deployment] (work/2024-07-23-release) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/48 (https://phabricator.wikimedia.org/T330794) (owner: 10aklapper) [14:27:05] (03merge) 10brennen: Uninstall Audit application [repos/phabricator/deployment] (work/2024-07-23-release) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/48 (https://phabricator.wikimedia.org/T330794) (owner: 10aklapper) [14:43:02] (03open) 10brennen: 2024-07-23 release [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/49 (https://phabricator.wikimedia.org/T330794 https://phabricator.wikimedia.org/T370776) [14:43:26] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phabricator/Phorge 2024-07-23 - https://phabricator.wikimedia.org/T370776#10007052 (10brennen) Release branch deployed to https://phabricator.wmcloud.org/ [14:46:16] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phabricator/Phorge 2024-07-23 - https://phabricator.wikimedia.org/T370776#10007053 (10brennen) a:03brennen [15:02:40] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phabricator/Phorge 2024-07-23 - https://phabricator.wikimedia.org/T370776#10007082 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=f58b5d71-46e5-4b24-8cdf-b84111b86989) set... [15:03:10] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phabricator/Phorge 2024-07-23 - https://phabricator.wikimedia.org/T370776#10007084 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a64bc67c-cf99-43e3-8b53-04a7a9363298) set... [15:03:20] 10Phabricator: Make phorge.wikimedia.org a redirect to phabricator.wikimedia.org - https://phabricator.wikimedia.org/T370778 (10GTrang) 03NEW [15:03:49] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phabricator/Phorge 2024-07-23 - https://phabricator.wikimedia.org/T370776#10007095 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=f91c576f-6803-47c8-8e7d-3d27b69f41f4) set... [15:04:10] (03merge) 10brennen: 2024-07-23 release [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/49 (https://phabricator.wikimedia.org/T330794 https://phabricator.wikimedia.org/T370776) [15:05:30] 06Release-Engineering-Team, 10Castor, 10Wikidata, 10ci-test-error (WMF-deployed Build Failure): Wikibase CI failing due to npm cache corruption issue - https://phabricator.wikimedia.org/T370748#10007105 (10WMDE-leszek) Thanks @hashar, I've requested the addition in T370766. Do you think it makes sense... [15:12:47] 10Phabricator (2024-07-23), 10Release-Engineering-Team (Priority Backlog 📥): Duplicating an EditEngine form does not log its creation - https://phabricator.wikimedia.org/T369568#10007162 (10Aklapper) 05Open→03Resolved This was deployed today (2024-07-23). [15:12:53] 10Phabricator (2024-07-23), 10Release-Engineering-Team (Priority Backlog 📥): Disallow creating commit-based Herald rules - https://phabricator.wikimedia.org/T367651#10007165 (10Aklapper) 05Open→03Resolved This was deployed today (2024-07-23). [15:14:11] 10Phabricator (2024-07-23): Disable Audit application in Phabricator - https://phabricator.wikimedia.org/T330794#10007170 (10Aklapper) [15:16:42] 10Phabricator (2024-07-23): Disable Audit application in Phabricator - https://phabricator.wikimedia.org/T330794#10007194 (10Aklapper) 05Open→03Resolved [15:17:01] 10Phabricator (2024-07-23), 06Release-Engineering-Team, 06collaboration-services: Deploy Phabricator/Phorge 2024-07-23 - https://phabricator.wikimedia.org/T370776#10007178 (10Aklapper) 05Open→03Resolved This was deployed today (2024-07-23) - thanks! [15:26:35] 10Phabricator: Make phorge.wikimedia.org a redirect to phabricator.wikimedia.org - https://phabricator.wikimedia.org/T370778#10007273 (10Aklapper) 05Open→03Declined No, it should not. (This ticket also lacks reasoning - please use the feature request template for requesting features. Thanks.) [15:27:39] heads up - I am going to upgrade and migrate Beta's sessionstorage/kask as part of https://phabricator.wikimedia.org/T370461 [15:28:15] I'll try to reduce the breakage to a minimum, but no guarantees :) [15:28:39] Southparkfan: <3, godspeed :) [15:29:33] one day, everything in deployment-prep is Buster-free :) [15:30:02] and we shall see each other again to do the same for Bullseye [15:32:55] Error: /Stage[main]/Puppet::Agent/Service[puppet]: Could not evaluate: Cannot allocate memory - fork(2) [15:33:07] ah yes, this upgrade is totally not doomed [15:46:42] Southparkfan: I bet that would be "fixed" by setting `vm.overcommit_memory` [15:46:53] but most probably the instance needs more RAM [15:48:36] * hashar oh and an explanation from the first time I encountered that issue while forking: https://phabricator.wikimedia.org/T219114#5084302 [15:50:10] or maybe even further in the past https://phabricator.wikimedia.org/T125050#3153574 [15:50:18] anyway thanks for rebuilding the VMs! [15:53:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Radar): puppet CI task failing with: Cannot allocate memory - https://phabricator.wikimedia.org/T284998#10007461 (10hashar) 05Open→03Resolved a:03hashar Looks like that got fixed. For the resource usage, I don't think there is anyt... [15:54:20] hashar: oh right, I'll do it with a swapfile for the old VM I guess [15:54:43] I was distracted a bit by a genuine breakage of AuthDNS in production [15:55:05] (fortunately, only for 2/3rd, at least the 3rd part was still up) [15:57:28] so far, the whole puppetization of Cassandra has been nothing but a pain to me [15:57:34] push 1 in /proc/sys/vm/overcommit_memory and linux will always pretends there is enough memory [15:57:46] sudo bash -c 'echo 1 > /proc/sys/vm/overcommit_memory' [15:58:08] nice, applied it on the old host [15:58:36] but of course, if the forked process ends up actually needing all the memory, it pretends it need it will eventually fail [16:01:04] aha - puppet tries to install cassandra=4.1.5, which is present in the bullseye-wikimedia cassandra41 component, but buster-wikimedia cassandra41 stops at 4.1.1 [16:01:05] classic [16:09:53] nvm, 4.1.1 isn't packaged either [16:18:07] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, and 2 others: Spin down api_appserver and appserver clusters - https://phabricator.wikimedia.org/T367949#10007582 (10Scott_French) Silenced ProbeDown for api-https:443 and appservers-https:443 for 24h: * f6f67d8d-6381-43b3-9262-9a8cf58f2b... [16:37:37] 10Release-Engineering-Team (Seen), 06collaboration-services, 10Data Pipelines, 06Data-Engineering, and 2 others: Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#10007638 (10LSobanski) [16:55:30] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-sessionstore04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370461#10007736 (10Southparkfan) Couldn't upgrade Buster to 4.x, because... [16:58:22] 10Phabricator: Make phorge.wikimedia.org a redirect to phabricator.wikimedia.org - https://phabricator.wikimedia.org/T370778#10007740 (10GTrang) [16:58:52] 10Phabricator: Make phorge.wikimedia.org a redirect to phabricator.wikimedia.org - https://phabricator.wikimedia.org/T370778#10007743 (10GTrang) [17:02:43] 10Phabricator: Make phorge.wikimedia.org a redirect to phabricator.wikimedia.org - https://phabricator.wikimedia.org/T370778#10007766 (10GTrang) >>! In T370778#10007273, @Aklapper wrote: > No, it should not. (This ticket also lacks reasoning - please use the feature request template for requesting features.... [17:03:30] (03merge) 10dancy: make-container-image: Add FORCE_MW_VERSION support [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/95 (https://phabricator.wikimedia.org/T369115) [17:05:52] 10Release-Engineering-Team (Seen), 06collaboration-services, 10Data Pipelines, 06Data-Engineering, and 2 others: Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#10007775 (10thcipriani) This is an interesting problem. It seems that the trusted runners should have picked up the job. 1.... [17:50:37] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 10Wikimedia-Apache-configuration, and 4 others: Apache 2.4.61 throws a 403 Forbidden for links containing %3F - https://phabricator.wikimedia.org/T370110#10007941 (10Dzahn) >>! In T370110#10005864, @hashar wrote: > *... [18:51:45] 10Phabricator: Make phorge.wikimedia.org a redirect to phabricator.wikimedia.org - https://phabricator.wikimedia.org/T370778#10008228 (10brennen) I agree that it would be nice if we didn't have a domain pinned to the old name of the software, but there is so much legacy stuff inter-tangled with the existing... [19:27:48] 10Phabricator: Make phorge.wikimedia.org a redirect to phabricator.wikimedia.org - https://phabricator.wikimedia.org/T370778#10008316 (10Aklapper) > Why not (contrary to my reasoning, which I have just added)? Because it creates long-term maintenance costs so I request arguments why to change/add something.... [19:37:21] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, and 2 others: Spin down api_appserver and appserver clusters - https://phabricator.wikimedia.org/T367949#10008340 (10Volans) [19:41:34] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, and 2 others: Spin down api_appserver and appserver clusters - https://phabricator.wikimedia.org/T367949#10008343 (10Volans) I took the liberty to add a cleanup item to the task description. If that should be part of another task feel to... [20:56:18] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, and 2 others: Spin down api_appserver and appserver clusters - https://phabricator.wikimedia.org/T367949#10008652 (10Scott_French) Many thanks, all who helped get this out the door. At this point, the LVS service turndown is done, and we... [20:56:41] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, and 2 others: Spin down api_appserver and appserver clusters - https://phabricator.wikimedia.org/T367949#10008653 (10Scott_French) [21:12:12] 06Release-Engineering-Team, 06Data-Engineering, 06Discovery-Search, 06Java-Scala-Standardization: Java projects hosted on Gerrit should publish artifacts to Gitlab - https://phabricator.wikimedia.org/T370400#10008694 (10brennen) > Do you have an opinion on where to host artifacts coming from projects hoste... [21:41:26] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, 06Traffic: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#10008833 (10Krinkle) [23:26:34] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Code-Stewardship-Reviews: deployment-prep (beta cluster): Code stewardship request - https://phabricator.wikimedia.org/T215217#10009077 (10bd808) >>! In T215217#6610236, @thcipriani wrote: > Sunsetting beta requires a plan to replace the us...