[07:23:42] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto: Scribunto does not run unit tests on REL branches - https://phabricator.wikimedia.org/T322506 (10Bawolff) [07:25:55] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto: Scribunto does not run unit tests on REL branches - https://phabricator.wikimedia.org/T322506 (10Bawolff) [08:38:53] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10ci-test-error: Scribunto does not run unit tests on REL branches - https://phabricator.wikimedia.org/T322506 (10hashar) [08:48:50] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10ci-test-error: Scribunto does not run unit tests on REL branches - https://phabricator.wikimedia.org/T322506 (10hashar) A change send to Scribunto REL branch triggers three jobs: | mwgate-node14-docker | `npm test` (linters, style checkers... [09:47:02] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10ci-test-error: Scribunto does not run unit tests on REL branches - https://phabricator.wikimedia.org/T322506 (10hashar) a:03hashar [10:00:22] (03PS1) 10Hashar: zuul: run PHPUnit standalone tests on REL branches [integration/config] - 10https://gerrit.wikimedia.org/r/853929 (https://phabricator.wikimedia.org/T322506) [10:02:15] (03CR) 10CI reject: [V: 04-1] zuul: run PHPUnit standalone tests on REL branches [integration/config] - 10https://gerrit.wikimedia.org/r/853929 (https://phabricator.wikimedia.org/T322506) (owner: 10Hashar) [10:03:37] (03PS2) 10Hashar: zuul: run PHPUnit standalone tests on REL branches [integration/config] - 10https://gerrit.wikimedia.org/r/853929 (https://phabricator.wikimedia.org/T322506) [10:05:18] (03CR) 10CI reject: [V: 04-1] zuul: run PHPUnit standalone tests on REL branches [integration/config] - 10https://gerrit.wikimedia.org/r/853929 (https://phabricator.wikimedia.org/T322506) (owner: 10Hashar) [10:07:17] (03PS3) 10Hashar: zuul: run PHPUnit standalone tests on REL branches [integration/config] - 10https://gerrit.wikimedia.org/r/853929 (https://phabricator.wikimedia.org/T322506) [10:23:38] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10CirrusSearch, and 3 others: Add a PHPUnit group to skip test on gated CI runs - https://phabricator.wikimedia.org/T225068 (10hashar) Note {nav Standalone} PHPUni... [10:23:50] 10Continuous-Integration-Config, 10SRE: cergen CI fails to run on Debian Stretch because cryptography dependency cannot be built against newer openssl version - https://phabricator.wikimedia.org/T212395 (10LSobanski) 05Open→03Resolved a:03LSobanski With no response in 2 years and the fact that we've migr... [10:27:45] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Data-Engineering-Planning, 10Discovery-Search, 10Event-Platform Value Stream: cirrusSearchCheckerJob JobQueueErrors (Could not enqueue jobs) on Beta Cluster - https://phabricator.wikimedia.org/T322491 (10EChetty) [10:28:41] 10GitLab, 10Release-Engineering-Team, 10Data-Engineering-Planning, 10serviceops-collab: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10EChetty) [10:38:04] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️), 10DBA, 10serviceops-collab, 10Patch-For-Review: sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Marostegui) >>! In T315713#8334341, @Dzahn wrote: > @Marostegui Sorry for the delay here. I made some... [10:44:43] 10GitLab, 10Release-Engineering-Team, 10Data Pipelines, 10Data-Engineering-Planning, 10serviceops-collab: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10EChetty) [12:33:25] 10Beta-Cluster-Infrastructure: Enable ContentTranslation for Wikivoyage on the Beta Cluster - https://phabricator.wikimedia.org/T322325 (10Pginer-WMF) [12:48:24] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Create mw-api-int helmfile deployment - https://phabricator.wikimedia.org/T321895 (10Clement_Goubert) p:05Triage→03High [12:48:43] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Create mw-jobrunner helmfile deployment - https://phabricator.wikimedia.org/T321897 (10Clement_Goubert) p:05Triage→03High [12:48:56] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Create mw-web helmfile deployment - https://phabricator.wikimedia.org/T321900 (10Clement_Goubert) p:05Triage→03High [12:49:11] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Create mw-web helmfile deployment - https://phabricator.wikimedia.org/T321900 (10Clement_Goubert) 05Open→03In progress [12:49:19] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786 (10Clement_Goubert) [12:50:11] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Create mw-api-int helmfile deployment - https://phabricator.wikimedia.org/T321895 (10Clement_Goubert) 05Open→03In progress [12:50:19] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786 (10Clement_Goubert) [12:50:31] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Create mw-api-ext helmfile deployment - https://phabricator.wikimedia.org/T321896 (10Clement_Goubert) 05Open→03In progress [12:50:41] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786 (10Clement_Goubert) [12:50:57] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Create mw-jobrunner helmfile deployment - https://phabricator.wikimedia.org/T321897 (10Clement_Goubert) 05Open→03In progress [12:51:07] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786 (10Clement_Goubert) [13:39:04] 10Release-Engineering-Team (Blocking 🧱), 10Discovery-Search, 10Projects-Cleanup: Archive or delete wikidata/query/flink-swift-plugin - https://phabricator.wikimedia.org/T314273 (10Gehel) [14:08:49] 10Continuous-Integration-Infrastructure, 10OOUI: OOUI PHP demos page is broken again - https://phabricator.wikimedia.org/T322357 (10taavi) [14:42:19] (03PS2) 10Majavah: zuul: Move PCC to its own Zuul pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/824424 [15:05:39] (03CR) 10Jbond: zuul: Move PCC to its own Zuul pipeline (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/824424 (owner: 10Majavah) [15:11:55] (03CR) 10Jforrester: [C: 03+1] "Oops, good find." [integration/config] - 10https://gerrit.wikimedia.org/r/853929 (https://phabricator.wikimedia.org/T322506) (owner: 10Hashar) [15:16:25] (03CR) 10Hashar: [C: 03+2] "Thanks James :)" [integration/config] - 10https://gerrit.wikimedia.org/r/853929 (https://phabricator.wikimedia.org/T322506) (owner: 10Hashar) [15:20:25] (03Merged) 10jenkins-bot: zuul: run PHPUnit standalone tests on REL branches [integration/config] - 10https://gerrit.wikimedia.org/r/853929 (https://phabricator.wikimedia.org/T322506) (owner: 10Hashar) [15:31:09] 10Continuous-Integration-Config, 10Abstract Wikipedia team, 10CirrusSearch, 10Discovery-Search, and 4 others: Scribunto does not run unit tests on REL branches - https://phabricator.wikimedia.org/T322506 (10hashar) PHPUnit Standalone groups not running in CI for release branches also affects #CirrusSearch... [15:45:50] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops-collab, 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10jijiki) [15:50:55] 10Release-Engineering-Team (Blocking 🧱), 10Discovery-Search, 10Projects-Cleanup: Archive or delete wikidata/query/flink-swift-plugin - https://phabricator.wikimedia.org/T314273 (10dcausse) [15:57:06] !log shutting down deployment-ms-be05 - T322231 [15:57:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:57:08] T322231: Create new deployment-ms-be instances running Debian Bullseye - https://phabricator.wikimedia.org/T322231 [15:58:23] ^^ that shouldn't impact the beta cluster as that node it's apparently properly removed from deployment-prep's swift ring [15:58:27] (last famous words) [16:00:13] (03CR) 10Hashar: zuul: Move PCC to its own Zuul pipeline (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/824424 (owner: 10Majavah) [16:02:04] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641 (10Vgutierrez) [16:02:07] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Create new deployment-ms-be instances running Debian Bullseye - https://phabricator.wikimedia.org/T322231 (10Vgutierrez) [16:02:51] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641 (10Vgutierrez) [16:05:35] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): Create new deployment-ms-fe instance running Debian Bullseye - https://phabricator.wikimedia.org/T322554 (10Vgutierrez) [16:06:11] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641 (10Vgutierrez) [16:18:30] 10Release-Engineering-Team (Blocking 🧱), 10Discovery-Search, 10Projects-Cleanup: Archive or delete wikidata/query/flink-swift-plugin - https://phabricator.wikimedia.org/T314273 (10MPhamWMF) p:05Triage→03Medium [16:19:03] 10Release-Engineering-Team (Blocking 🧱), 10Projects-Cleanup, 10Wikidata-Query-Service: Archive or delete wikidata/query/flink-swift-plugin - https://phabricator.wikimedia.org/T314273 (10Gehel) [16:34:51] 10Beta-Cluster-Infrastructure, 10Data-Engineering-Planning, 10Event-Platform Value Stream: cirrusSearchCheckerJob JobQueueErrors (Could not enqueue jobs) on Beta Cluster - https://phabricator.wikimedia.org/T322491 (10Gehel) Removing the Search team from this, it looks to be related to JobQueue and not touchi... [16:35:30] 10Continuous-Integration-Config, 10Abstract Wikipedia team, 10Discovery-Search, 10MediaWiki-extensions-Scribunto, and 3 others: Scribunto does not run unit tests on REL branches - https://phabricator.wikimedia.org/T322506 (10Gehel) [16:35:46] 10Continuous-Integration-Config, 10Abstract Wikipedia team, 10MediaWiki-extensions-Scribunto, 10WikiLambda, and 2 others: Scribunto does not run unit tests on REL branches - https://phabricator.wikimedia.org/T322506 (10Gehel) [16:36:53] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Stretch Deprecation): Cloud VPS "deployment-prep" project Stretch deprecation - https://phabricator.wikimedia.org/T306068 (10Vgutierrez) [16:40:35] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Stretch Deprecation): Cloud VPS "deployment-prep" project Stretch deprecation - https://phabricator.wikimedia.org/T306068 (10Vgutierrez) [16:50:21] 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops, 10Release Pipeline (Blubber): WMF container registry does not accept a manifest list (aka OCI manifest index, or "fat" manifest) - https://phabricator.wikimedia.org/T322453 (10JMeybohm) I think this might require an update of our docker-registry... [17:20:03] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Patch-For-Review, 10ci-test-error: Scribunto does not run unit tests on REL branches - https://phabricator.wikimedia.org/T322506 (10Jdforrester-WMF) [17:22:47] 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops, 10Release Pipeline (Blubber): WMF container registry does not accept a manifest list (aka OCI manifest index, or "fat" manifest) - https://phabricator.wikimedia.org/T322453 (10dduvall) >>! In T322453#8375676, @JMeybohm wrote: > I think this might... [17:38:59] (03Restored) 10Deni: Zuul: Add Denis Smajlović (@deni), and company, to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/853078 (owner: 10Deni) [18:01:09] 10Release-Engineering-Team (GitLab-a-thon 🦊): Bronislawskich - https://phabricator.wikimedia.org/T322571 (10Smallpox25) [18:18:59] !log let deployment-cache-upload07 use deployment-ms-fe04 - T322554 [18:19:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:19:01] T322554: Create new deployment-ms-fe instance running Debian Bullseye - https://phabricator.wikimedia.org/T322554 [18:31:01] (03CR) 10Deni: "Hi Antoine," [integration/config] - 10https://gerrit.wikimedia.org/r/853078 (owner: 10Deni) [19:11:57] 10Beta-Cluster-Infrastructure, 10Traffic: ATS isn't caching documents in deployment-cache-upload07 - https://phabricator.wikimedia.org/T322575 (10Vgutierrez) [19:12:07] 10Beta-Cluster-Infrastructure, 10Traffic: ATS isn't caching documents in deployment-cache-upload07 - https://phabricator.wikimedia.org/T322575 (10Vgutierrez) p:05Triage→03Medium [19:16:14] 10Beta-Cluster-Infrastructure, 10Traffic: ATS isn't caching documents in deployment-cache-upload07 - https://phabricator.wikimedia.org/T322575 (10Vgutierrez) For some reason swift isn't setting a `Cache-Control` header in deployment-prep: ` vgutierrez@deployment-ms-fe04:~$ curl --connect-to upload.wikimedia.be... [19:22:53] 10Beta-Cluster-Infrastructure, 10Data-Persistence, 10Traffic: ATS isn't caching documents in deployment-cache-upload07 - https://phabricator.wikimedia.org/T322575 (10Vgutierrez) on Swift the Cache-Control header is appended on the ms-fe instances by a filter called `ensure_max_age` with the following config:... [19:59:35] 10Beta-Cluster-Infrastructure, 10Editing-team, 10MediaWiki-Configuration, 10VisualEditor, and 2 others: Configure VE backend to use Parsoid directly on the beta cluster and testwiki - https://phabricator.wikimedia.org/T320531 (10daniel) [20:00:30] 10Beta-Cluster-Infrastructure, 10Editing-team, 10MediaWiki-Configuration, 10VisualEditor, and 2 others: Configure VE backend to use Parsoid directly on the beta cluster and testwiki - https://phabricator.wikimedia.org/T320531 (10daniel) 05Open→03Resolved a:03daniel [20:09:53] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️), 10DBA, 10serviceops-collab, 10Patch-For-Review: sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Dzahn) 05Open→03In progress [20:09:59] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️), 10serviceops, 10serviceops-collab, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) [20:10:06] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️), 10DBA, 10serviceops-collab, 10Patch-For-Review: sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Dzahn) a:05Marostegui→03Dzahn Thank you! Yes, I will check again. [20:10:29] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️), 10DBA, 10serviceops-collab, 10Patch-For-Review: sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Dzahn) p:05Medium→03High [20:56:12] (03PS1) 10Ahmon Dancy: create-databases: Don't create labtestwiki db [tools/train-dev] - 10https://gerrit.wikimedia.org/r/854102 [20:57:50] (03CR) 10Ahmon Dancy: [C: 03+2] create-databases: Don't create labtestwiki db [tools/train-dev] - 10https://gerrit.wikimedia.org/r/854102 (owner: 10Ahmon Dancy) [20:58:17] (03Merged) 10jenkins-bot: create-databases: Don't create labtestwiki db [tools/train-dev] - 10https://gerrit.wikimedia.org/r/854102 (owner: 10Ahmon Dancy) [20:59:19] (03CR) 10Ahmon Dancy: [V: 03+2 C: 03+2] "Tested in a fresh workspace." [tools/train-dev] - 10https://gerrit.wikimedia.org/r/853394 (https://phabricator.wikimedia.org/T321847) (owner: 10Ahmon Dancy) [20:59:45] (03Merged) 10jenkins-bot: train-dev: Clone scap from gitlab [tools/train-dev] - 10https://gerrit.wikimedia.org/r/853394 (https://phabricator.wikimedia.org/T321847) (owner: 10Ahmon Dancy) [21:00:56] (03CR) 10Ahmon Dancy: [C: 03+2] php7.2 to php7.4 [tools/train-dev] - 10https://gerrit.wikimedia.org/r/853390 (owner: 10Ahmon Dancy) [21:01:25] (03Merged) 10jenkins-bot: php7.2 to php7.4 [tools/train-dev] - 10https://gerrit.wikimedia.org/r/853390 (owner: 10Ahmon Dancy) [21:40:02] 10Release-Engineering-Team (Radar), 10serviceops-collab: give releng access to logs to debug buildkit-to-wmf-registry publishing - https://phabricator.wikimedia.org/T322579 (10dduvall) [21:52:37] 10Release-Engineering-Team (Priority Backlog 📥), 10Epic, 10Patch-For-Review, 10Release Pipeline (Blubber): Upgrade docker on integration hosts for fixes to BuildKit builder - https://phabricator.wikimedia.org/T318382 (10dduvall) >>! In T318382#8317695, @dduvall wrote: > Thanks, Moritz. Can we do the same f... [21:54:38] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab: Move cloud runner CI jobs to trusted runners - https://phabricator.wikimedia.org/T322344 (10dduvall) Thanks for the review/merge, @MoritzMuehlenhoff and @Dzahn! I don't see the packages yet but I'm assuming the... [22:00:36] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab: Move cloud runner CI jobs to trusted runners - https://phabricator.wikimedia.org/T322344 (10Dzahn) Looks like this is needed: https://wikitech.wikimedia.org/wiki/Reprepro#Adding_a_new_external_repository but pu... [22:00:36] Hello o/ I'm trying to run the production mediawiki helm chart locally on minikube. I'm currently stuck insofar as when I go to http://CLUSTER_IP:8080, I simply get "File not found." I'm not sure where to get more detailed logs. Any tips? [22:01:26] Here's my configuration: https://gitlab.wikimedia.org/kindrobot/aw-test-env [22:11:29] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab: Move cloud runner CI jobs to trusted runners - https://phabricator.wikimedia.org/T322344 (10Dzahn) ` apt1001:~] $ sudo -E reprepro --verbose --component thirdparty/terraform-bullseye update bullseye-wikimedia... [22:12:25] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Phonos, 10SRE-swift-storage, 10Community-Tech (CommTech-Sprint-36), and 2 others: Phonos links to an unauthorized URL - https://phabricator.wikimedia.org/T317417 (10JMcLeod_WMF) [22:25:35] 10Release-Engineering-Team (Radar), 10serviceops, 10serviceops-collab: give releng access to logs to debug buildkit-to-wmf-registry publishing - https://phabricator.wikimedia.org/T322579 (10Dzahn) [22:32:44] 10Beta-Cluster-Infrastructure: File on betacommons shows usage on production and links nonexistent beta project - https://phabricator.wikimedia.org/T301997 (10matmarex) a:03matmarex