[00:02:13] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11747228 (10bd808) >>! In T420833, @AlexisJazz wrote: > Unsure if related, but these ping results are strange? Why are different domains answering? >>! In T420833#11735601, @AlexisJazz... [00:27:49] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [MaryyannL] - https://phabricator.wikimedia.org/T421180#11747261 (10Peachey88) 05Openβ†’03Resolved a:03glaab The bot has done it's magic: https://wikitech.wikimedia.org/w/index.php?title=Tool%3AGitlab-acco... [00:35:13] Project beta-code-update-eqiad build #593107: 04FAILURE in 2 min 12 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/593107/ [00:45:18] Yippee, build fixed! [00:45:18] Project beta-code-update-eqiad build #593108: 09FIXED in 2 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/593108/ [05:04:01] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Doing 😎): Jenkins: In the integration project in WMCS, test out larger workers for CI - https://phabricator.wikimedia.org/T383254#11747487 (10Peter) FYI in T420185 I changed how many webdriverIO tests we run in parallel. With the change we se... [06:03:45] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11747532 (10AlexisJazz) Thanks for the info. Beta cluster is slow as sludge again. ;-( [06:18:41] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: gerrit: Add envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11747534 (10ABran-WMF) [06:22:18] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: gerrit: Add envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11747537 (10ABran-WMF) >>! In T420909#11742909, @ABran-WMF wrote: > `gerrit-spare` now uses Envoy to expose its service to the CDN... [06:28:23] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: gerrit: Add Envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11747543 (10ABran-WMF) [08:37:20] 10Phabricator: Have Phabricator link Gerrit change-id to Gerrit search query - https://phabricator.wikimedia.org/T420363#11747831 (10hashar) @Aklapper that code reads like a book, that is nice :-] I have filed this as an idea, I don't know though whether that is a good idea or something we actually want to proc... [08:39:11] seeing npm cache corruption on mediawiki-node20 jobs. Will clear the cache [08:40:05] !log integration: integration-castor05: rm -fR /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20/ [08:40:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:46:06] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11747846 (10ItsNyoty) I am experiencing the same issues for a month. This has been happening on nl.wikipedia.beta too (which is normal since it uses the same cache) Request served via... [08:47:51] 10Continuous-Integration-Infrastructure: Add Python 3.14 to Wikimedia CI - https://phabricator.wikimedia.org/T421209 (10hashar) 03NEW [09:01:33] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [MaryyannL] - https://phabricator.wikimedia.org/T421180#11747916 (10Aklapper) Was this banner not displayed on GitLab? It has instructions for WMF folks. {F65766561,size=full} [09:32:17] (03PS1) 10Hashar: dockerfiles: add support for Python 3.14 [integration/config] - 10https://gerrit.wikimedia.org/r/1260631 (https://phabricator.wikimedia.org/T421209) [09:32:20] (03PS1) 10Hashar: jjb: update tox jobs to support Python 3.14 [integration/config] - 10https://gerrit.wikimedia.org/r/1260632 (https://phabricator.wikimedia.org/T421209) [09:32:55] (03CR) 10Hashar: [C:03+2] "I have successfully built them locally." [integration/config] - 10https://gerrit.wikimedia.org/r/1260631 (https://phabricator.wikimedia.org/T421209) (owner: 10Hashar) [09:34:51] (03Merged) 10jenkins-bot: dockerfiles: add support for Python 3.14 [integration/config] - 10https://gerrit.wikimedia.org/r/1260631 (https://phabricator.wikimedia.org/T421209) (owner: 10Hashar) [10:01:52] !log Updating tox Jenkins jobs to add support for Python 3.14 | https://gerrit.wikimedia.org/r/1260632 | T421209 [10:01:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:01:56] T421209: Add Python 3.14 to Wikimedia CI - https://phabricator.wikimedia.org/T421209 [10:06:37] (03CR) 10Hashar: [C:03+2] "I have updated all the jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/1260632 (https://phabricator.wikimedia.org/T421209) (owner: 10Hashar) [10:09:48] (03Merged) 10jenkins-bot: jjb: update tox jobs to support Python 3.14 [integration/config] - 10https://gerrit.wikimedia.org/r/1260632 (https://phabricator.wikimedia.org/T421209) (owner: 10Hashar) [11:06:34] (03open) 10hoo: Remove (broken) develop from default XDEBUG_MODE [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/643 [11:16:04] (03open) 10hoo: Default XDEBUG_CONFIG to client_host=xdebug://gateway [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/644 [11:19:41] 10Continuous-Integration-Infrastructure: Add Python 3.14 to Wikimedia CI - https://phabricator.wikimedia.org/T421209#11748317 (10hashar) 05Openβ†’03Resolved a:03hashar [11:20:19] 10Continuous-Integration-Infrastructure: Rebuild all Jenkins agents VM to Bookworm to support Java 21 - https://phabricator.wikimedia.org/T421114#11748319 (10hashar) [12:43:10] 10Phabricator, 06Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Deprecate phabricator's fixed_settings.yaml (in puppet) and just use phabricator's built in configuration tools - https://phabricator.wikimedia.org/T239355#11748650 (10A_smart_kitten) While working on {T388022} (& [[https://gerrit.wikim... [12:58:18] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07ci-test-error (WMF-deployed Build Failure): Tests failing with "23:12:20 stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/REPO: GnuTLS recv error (-54): Erro... - https://phabricator.wikimedia.org/T420865#11748691 [13:03:59] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: zuul1002 fails to compile in PCC - https://phabricator.wikimedia.org/T421232 (10taavi) 03NEW [13:05:16] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: zuul1002 fails to compile in PCC - https://phabricator.wikimedia.org/T421232#11748756 (10taavi) That secret is set in labs/private, but only for `role(zuul::main)`, and this machine is `role(zuul::executor)`. [13:30:02] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services, 10Cloud-VPS (Quota-requests): Large temporary quota increase for integration project to ease Jenkins migration - https://phabricator.wikimedia.org/T421139#11748838 (10hashar) 05Openβ†’03Declined I will rebuild them to Bookw... [13:31:39] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services, 13Patch-For-Review: Update Jenkins hosts from Java 17 to Java 21 - https://phabricator.wikimedia.org/T418109#11748862 (10hashar) > 2.or build a new fleet of instances based on Trixie (which has Java 21) and only connect the new c... [13:32:49] 10Continuous-Integration-Infrastructure, 07Jenkins: Switch CI agent config from IPs to FQDN now it's possible - https://phabricator.wikimedia.org/T300224#11748893 (10hashar) [13:48:14] (03PS6) 10Zfilipin: Add devcontainer.json [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250501 (https://phabricator.wikimedia.org/T421236) [13:48:59] (03CR) 10Zfilipin: "PS6 updates relevant task in the commit message." [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250501 (https://phabricator.wikimedia.org/T421236) (owner: 10Zfilipin) [13:51:46] 10Continuous-Integration-Infrastructure, 07Jenkins: Switch CI agent config from IPs to FQDN now it's possible - https://phabricator.wikimedia.org/T300224#11749040 (10hashar) 05Openβ†’03Resolved a:03hashar Some hosts already used the FQDN (puppet-diffs, deployment-deploy04). I have updated the configura... [13:51:57] it is never too late to fix a 4 years old bug! [14:00:10] (03CR) 10Zfilipin: Add devcontainer.json (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250501 (https://phabricator.wikimedia.org/T421236) (owner: 10Zfilipin) [14:01:49] (03PS7) 10Zfilipin: Add devcontainer.json [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250501 (https://phabricator.wikimedia.org/T421236) [14:06:10] (03CR) 10Zfilipin: Add devcontainer.json (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250501 (https://phabricator.wikimedia.org/T421236) (owner: 10Zfilipin) [14:20:06] 06Project-Admins: Clarify the status of Structured Content / Structured Data team - https://phabricator.wikimedia.org/T404286#11749160 (10Aklapper) 05Openβ†’03Resolved > If Structured Content team no longer exists or is merged to other team: Everything in this section has happened, thus resolving. [14:54:06] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, 10Cloud-VPS (Quota-requests): New flavor for the integration project with more vCPU and ephemeral disk space - https://phabricator.wikimedia.org/T421242 (10hashar) 03NEW [14:55:37] 10Phabricator, 06collaboration-services, 06DC-Ops, 10ops-codfw: phab2002: SEL System Event:, System Board Front LED Panel, Critical, management controller unavailable - https://phabricator.wikimedia.org/T420228#11749332 (10Jhancock.wm) @Dzahn looks like that worked. it rebooted after the update and there... [15:04:15] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, 10Browser Test Platform, 10Cloud-VPS (Quota-requests): New flavor for the integration project with more vCPU and ephemeral disk space - https://phabricator.wikimedia.org/T421242#11749366 (10Peter) [15:13:00] (03update) 10dancy: Replace nginx ingress with traefik [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/562 (https://phabricator.wikimedia.org/T420743) [15:13:42] 10Phabricator, 10Bitu, 06cloud-services-team, 06Infrastructure-Foundations, and 2 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11749383 (10bd808) >>! In T420599#11744332, @Dondersmooi wrote: > My question is: should Bitu... [15:18:02] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Rebuild deployment-deploy04 with Bookworm to get Java 21 - https://phabricator.wikimedia.org/T421244 (10hashar) 03NEW [15:18:24] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Rebuild deployment-deploy04 with Bookworm to get Java 21 - https://phabricator.wikimedia.org/T421244#11749415 (10hashar) [15:18:25] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Rebuild all Jenkins agents VM to Bookworm to support Java 21 - https://phabricator.wikimedia.org/T421114#11749414 (10hashar) [15:20:54] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Rebuild all Jenkins agents VM to Bookworm to support Java 21 - https://phabricator.wikimedia.org/T421114#11749439 (10hashar) [15:26:17] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Rebuild deployment-deploy04 with Bookworm to get Java 21 - https://phabricator.wikimedia.org/T421244#11749488 (10bd808) [15:26:22] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 10Cloud-VPS (Debian Bullseye Deprecation), 07Epic: Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#11749487 (10bd808) [15:31:18] (03update) 10dancy: Replace nginx ingress with traefik [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/562 (https://phabricator.wikimedia.org/T420743) [15:32:07] !log Migrated gitlab-cloud-runners (staging) from nginx-ingress to traefik (T420743) [15:32:07] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, 10Browser Test Platform, 10Cloud-VPS (Quota-requests): New flavor for the integration project with more vCPU and ephemeral disk space - https://phabricator.wikimedia.org/T421242#11749518 (10zeljkofilipin) [15:32:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:32:18] T420743: Replace nginx ingress with traefik for gitlab-cloud-runners - https://phabricator.wikimedia.org/T420743 [15:32:34] (03merge) 10dancy: Replace nginx ingress with traefik [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/562 (https://phabricator.wikimedia.org/T420743) [15:41:05] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, 06Quality-and-Test-Engineering-Team (Test Infrastructure): Move beta cluster automatic deployment to a dedicated infrastructure - https://phabricator.wikimedia.org/T256168#11749554 (10bd808... [15:44:34] 10Phabricator, 06collaboration-services, 06DC-Ops, 10ops-codfw: phab2002: SEL System Event:, System Board Front LED Panel, Critical, management controller unavailable - https://phabricator.wikimedia.org/T420228#11749564 (10Dzahn) 05In progressβ†’03Resolved a:03Dzahn Great! Thank you, @Jhancock.wm... [15:44:54] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, 06Quality-and-Test-Engineering-Team (Test Infrastructure): Move beta cluster automatic deployment to a dedicated infrastructure - https://phabricator.wikimedia.org/T256168#11749568 (10bd808... [15:45:05] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: zuul1002 fails to compile in PCC - https://phabricator.wikimedia.org/T421232#11749569 (10Dzahn) a:03Dzahn [15:46:15] !log Migrated gitlab-cloud-runners (prod) from nginx-ingress to traefik (T420743) [15:46:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:46:17] T420743: Replace nginx ingress with traefik for gitlab-cloud-runners - https://phabricator.wikimedia.org/T420743 [15:47:31] (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331) [15:53:07] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Rebuild deployment-deploy04 with Bookworm to get Java 21 - https://phabricator.wikimedia.org/T421244#11749588 (10hashar) From a discussion I had with bd808 and others, we might instead stop using to update the #beta-cluster-infrastructure in favor of a... [15:54:47] 10Phabricator, 10Bitu, 06cloud-services-team, 06Infrastructure-Foundations, and 2 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11749593 (10Dondersmooi) 05Openβ†’03Declined [15:55:03] 10Phabricator, 10Bitu, 06cloud-services-team, 06Infrastructure-Foundations, and 2 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11749594 (10Dondersmooi) Thank you all for the thorough answers to all my questions! [15:55:16] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 13Patch-For-Review: Replace nginx ingress with traefik for gitlab-cloud-runners - https://phabricator.wikimedia.org/T420743#11749607 (10dancy) 05Openβ†’03Resolved p:05Triageβ†’03Medium a:03dancy [16:00:33] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Rebuild deployment-deploy04 with Bookworm to get Java 21 - https://phabricator.wikimedia.org/T421244#11749622 (10bd808) We need to build a new instance for {T401839} too, so the quicker thing is probably to do that now and also unblock the Jenkins stuf... [16:01:32] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Rebuild deployment-deploy04 with Bookworm to get Java 21 - https://phabricator.wikimedia.org/T421244#11749629 (10bd808) p:05Triageβ†’03High a:03bd808 [16:02:13] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Replace deployment-deploy04 with a Bookworm instance with Java 21 - https://phabricator.wikimedia.org/T421244#11749636 (10bd808) [16:03:55] (03open) 10dancy: ingress/main.tf: Bump traefik chart to 39.0.6 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/564 [16:03:58] (03update) 10dancy: ingress/main.tf: Bump traefik chart to 39.0.6 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/564 [16:05:52] (03merge) 10dancy: ingress/main.tf: Bump traefik chart to 39.0.6 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/564 [16:06:52] 10Continuous-Integration-Config, 06Test Platform (Črnomerec 24): Add devcontainer to integration/config - https://phabricator.wikimedia.org/T421249 (10zeljkofilipin) 03NEW [16:07:14] 10Continuous-Integration-Config, 06Test Platform (Črnomerec 24): Add devcontainer to integration/config - https://phabricator.wikimedia.org/T421249#11749660 (10zeljkofilipin) 05Openβ†’03In progress p:05Triageβ†’03Medium a:03zeljkofilipin [16:08:02] 10Continuous-Integration-Config, 06Test Platform (Črnomerec 24): Add devcontainer to integration/config - https://phabricator.wikimedia.org/T421249#11749679 (10zeljkofilipin) [16:10:34] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Replace deployment-deploy04 with a Bookworm instance with Java 21 - https://phabricator.wikimedia.org/T421244#11749712 (10bd808) In a quick chat with @hashar I asked about the cut-over from the Java 17 compatible Jenkins to Java 21 compatible Jenkins.... [16:22:33] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work, 13Patch-For-Review: Buildkit v0.28.1 released - https://phabricator.wikimedia.org/T415284#11749809 (10dancy) [16:28:55] (03open) 10dancy: staging.tfvars: Bump buildkit to wmf-v0.28.1 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/565 [16:28:58] (03update) 10dancy: staging.tfvars: Bump buildkit to wmf-v0.28.1 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/565 [16:30:48] (03merge) 10dancy: staging.tfvars: Bump buildkit to wmf-v0.28.1 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/565 [16:44:43] 10GitLab, 06Release-Engineering-Team: Slack Automation with Gerrit and Gitlab - https://phabricator.wikimedia.org/T421259 (10DSantamaria) 03NEW [16:48:27] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11749954 (10bd808) >>! In T420833#11747846, @ItsNyoty wrote: > I am experiencing the same issues for a month. This has been happening on nl.wikipedia.beta too (which is normal since it u... [17:08:17] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, 10Browser Test Platform, 10Cloud-VPS (Quota-requests): New flavor for the integration project with more vCPU and ephemeral disk space - https://phabricator.wikimedia.org/T421242#11750096 (10hashar) [17:21:59] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services, 13Patch-For-Review: Update Jenkins hosts from Java 17 to Java 21 - https://phabricator.wikimedia.org/T418109#11750205 (10Dzahn) >>! In T418109#11661882, @MoritzMuehlenhoff wrote: > @Dzahn : I enabled Java 21 on build2002 (which i... [17:39:39] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services, 13Patch-For-Review: Update Jenkins hosts from Java 17 to Java 21 - https://phabricator.wikimedia.org/T418109#11750275 (10Dzahn) ` root@releases1003:/etc/apt/sources.list.d# cat component-jdk21-apt.wikimedia.org-wikimedia-bookworm... [17:49:54] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services, 13Patch-For-Review: Update Jenkins hosts from Java 17 to Java 21 - https://phabricator.wikimedia.org/T418109#11750312 (10Dzahn) found in `modules/profile/manifests/bigtop/apt.pp` which seems the only other place using a `apt::rep... [18:19:24] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services, 13Patch-For-Review: Update Jenkins hosts from Java 17 to Java 21 - https://phabricator.wikimedia.org/T418109#11750468 (10Dzahn) So yea, this seems the same thing as this: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1194... [18:26:32] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services, 13Patch-For-Review: Update Jenkins hosts from Java 17 to Java 21 - https://phabricator.wikimedia.org/T418109#11750480 (10Dzahn) 05Openβ†’03In progress p:05Triageβ†’03High [20:05:19] 10GitLab, 06Release-Engineering-Team: Slack Automation with Gerrit and Gitlab - https://phabricator.wikimedia.org/T421259#11750820 (10dancy) p:05Triageβ†’03Medium [20:06:38] 10GitLab, 06Release-Engineering-Team: Slack Automation with Gerrit and Gitlab - https://phabricator.wikimedia.org/T421259#11750824 (10dancy) Hi @DSantamaria. Can you fill in the requested bot name and team info in the description please? [20:23:06] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Wyslijp16 Toolforge - https://phabricator.wikimedia.org/T417394#11750910 (10Aklapper) 05Stalledβ†’03Declined Unfortunately closing this Phabricator task as no reply has been provided. [20:23:29] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11750912 (10ItsNyoty) Pff, that sucks. There's no way that we can help right? Otherwise please share, I use the beta quite often 😣 [20:28:05] 10Continuous-Integration-Config, 10AbuseFilter, 10MediaWiki-extensions-CodeMirror, 07ci-test-error, and 2 others: CI failure on REL1_44 - https://phabricator.wikimedia.org/T421291#11750943 (10Reedy) [20:29:52] 10Continuous-Integration-Config, 10AbuseFilter, 10MediaWiki-extensions-CodeMirror, 07ci-test-error, and 2 others: CodeMirror CI failure on REL1_43/REL1_44 - https://phabricator.wikimedia.org/T421291#11750959 (10Reedy) [20:33:47] 10Continuous-Integration-Config, 10AbuseFilter, 10MediaWiki-extensions-CodeMirror, 07ci-test-error, and 2 others: CodeMirror CI failure on REL1_43/REL1_44 - https://phabricator.wikimedia.org/T421291#11750983 (10Reedy) Caused by {7d7884f7da5cc67a07e4a3f62cbb2a8add3a6772} for {T399673}.... [20:35:53] (03PS1) 10Reedy: Revert "Zuul: [mediawiki/extensions/AbuseFilter] Add dependency on CodeMirror" [integration/config] - 10https://gerrit.wikimedia.org/r/1260796 (https://phabricator.wikimedia.org/T421283) [20:36:23] (03PS2) 10Reedy: Revert "Zuul: [mediawiki/extensions/AbuseFilter] Add dependency on CodeMirror" [integration/config] - 10https://gerrit.wikimedia.org/r/1260796 (https://phabricator.wikimedia.org/T421283) [20:36:27] 10Continuous-Integration-Config, 10AbuseFilter, 10MediaWiki-extensions-CodeMirror, 07ci-test-error, and 2 others: CodeMirror CI failure on REL1_43/REL1_44 - https://phabricator.wikimedia.org/T421291#11750993 (10Reedy) β†’14Duplicate dup:03T421283 [20:36:58] 10Continuous-Integration-Config, 10AbuseFilter, 10MediaWiki-extensions-CodeMirror, 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: AbuseFilter CI is broken due to unconditional module ext.visualEditor.mwcore from CodeMirror - https://phabricator.wikimedia.org/T421283#11750997 (10Reedy) [20:38:48] (03CR) 10Reedy: [C:03+2] Revert "Zuul: [mediawiki/extensions/AbuseFilter] Add dependency on CodeMirror" [integration/config] - 10https://gerrit.wikimedia.org/r/1260796 (https://phabricator.wikimedia.org/T421283) (owner: 10Reedy) [20:40:28] (03Merged) 10jenkins-bot: Revert "Zuul: [mediawiki/extensions/AbuseFilter] Add dependency on CodeMirror" [integration/config] - 10https://gerrit.wikimedia.org/r/1260796 (https://phabricator.wikimedia.org/T421283) (owner: 10Reedy) [20:40:43] (03CR) 10Reedy: "AbuseFilter doesn't load VisualEditor, so this doesn't actually work" [integration/config] - 10https://gerrit.wikimedia.org/r/1256318 (https://phabricator.wikimedia.org/T399673) (owner: 10Bhsd) [20:41:29] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1256318 T421283 [20:41:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:41:32] T421283: AbuseFilter CI is broken due to unconditional module ext.visualEditor.mwcore from CodeMirror - https://phabricator.wikimedia.org/T421283 [20:43:51] 10Continuous-Integration-Config, 10AbuseFilter, 10MediaWiki-extensions-CodeMirror, 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: AbuseFilter CI is broken due to unconditional module ext.visualEditor.mwcore from CodeMirror - https://phabricator.wikimedia.org/T421283#11751025 (10Reed... [20:49:11] (03open) 10jhuneidi: catalyst-mw-chart-base: [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/97 (https://phabricator.wikimedia.org/T420441) [21:08:11] 10GitLab, 06Release-Engineering-Team: Slack Automation with Gerrit and Gitlab - https://phabricator.wikimedia.org/T421259#11751163 (10DSantamaria) [21:08:22] 10GitLab, 06Release-Engineering-Team: Slack Automation with Gerrit and Gitlab - https://phabricator.wikimedia.org/T421259#11751168 (10DSantamaria) Done! [21:09:21] 10GitLab, 06Release-Engineering-Team: Slack Automation with Gerrit and Gitlab - https://phabricator.wikimedia.org/T421259#11751176 (10dancy) >>! In T421259#11751163, @DSantamaria wrote: > Done! Thanks! Does the Abstract Wikipedia team have an internal mailing list that I can reference? [21:12:10] 10GitLab, 06Release-Engineering-Team: Slack Automation with Gerrit and Gitlab - https://phabricator.wikimedia.org/T421259#11751205 (10DSantamaria) Yes, but does this need to be attached to the team? I mean, can I have a specific email list for the bot (if I ask tech support, of course)? I would prefer to avoi... [21:14:17] 10Continuous-Integration-Config, 05Goal: Add CI to all Gerrit repositories - https://phabricator.wikimedia.org/T180317#11751230 (10Pppery) [21:14:59] 10GitLab, 06Release-Engineering-Team: Slack Automation with Gerrit and Gitlab - https://phabricator.wikimedia.org/T421259#11751237 (10dancy) >>! In T421259#11751205, @DSantamaria wrote: > Yes, but does this need to be attached to the team? I mean, can I have a specific email list for the bot (if I ask tech sup... [21:19:38] 10Beta-Cluster-Infrastructure, 06SRE, 05Goal, 07Technical-Debt: Minimize infrastructure differences between Beta Cluster and production - https://phabricator.wikimedia.org/T87220#11751304 (10Pppery) [21:21:19] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11751325 (10bd808) Let's block `57.141.0.0/16` which is facebook and has sent 98,072 requests today. That is the highest volume from a `/16` today by an order of magnitude. https://gerr... [21:21:28] 10Beta-Cluster-Infrastructure, 10Cloud-Services, 07Tracking-Neverending: Beta Cluster <-> WMF Labs policy compliance (tracking) - https://phabricator.wikimedia.org/T114615#11751327 (10Pppery) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.w... [21:26:59] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11751400 (10bd808) 74.235.192.0/18 is still more Microsoft. https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/1f1f276801415e198730cac0cda7cd2b2f039b28%5E%21/#F0 ` di... [21:42:33] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11751617 (10bd808) 188.166.0.0/17 is EU-DIGITALOCEAN-NL1 https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/62cb0aea5eed0d0aa3e19e085b338b56ad9b670d%5E%21/#F0 ` diff... [21:59:10] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: zuul1002 fails to compile in PCC - https://phabricator.wikimedia.org/T421232#11751769 (10Dzahn) Thanks for reporting, @taavi . It's exactly what you already said, added for the main role but not for the exec... [22:03:49] 10GitLab, 06Release-Engineering-Team: Slack Automation with Gerrit and Gitlab - https://phabricator.wikimedia.org/T421259#11751811 (10Dzahn) To get a mailing list (or alias) you have choices: If you prefer a Google list: ask ITS per techsupport@ to set one up for you. If you prefer a Mailman list (https:/... [22:06:39] 10GitLab, 06Release-Engineering-Team: Slack Automation with Gerrit and Gitlab - https://phabricator.wikimedia.org/T421259#11751818 (10Dzahn) Beyond that there are also shared inboxes (group inbox in Google). What I would not recommend is to just make a group address an alias for one specific user though. [22:11:38] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: zuul1002 fails to compile in PCC - https://phabricator.wikimedia.org/T421232#11751833 (10Dzahn) 05Openβ†’03Resolved https://puppet-compiler.wmflabs.org/output/1260816/8343/zuul1002.eqiad.wmnet/prod.zuu... [22:23:28] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11751863 (10bd808) 2.59.220.0/22 is CZ-ONEHOSTPLANET-20190327 https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/a88b7a7c74f328f6637a06c0561f2e91a8904ae8%5E%21/#F0 `... [22:28:54] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11751892 (10bd808) 74.243.224.0/19 is still more Microsoft https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/d4477039a0a591c5cdc3c7531e24c2baff6ff9cb%5E%21/#F0 ` dif... [22:36:55] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11751959 (10bd808) 212.237.120.0/22 is KurdEstanNetSubnet7 https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/5f0c3aa390e9d81602be7a5a9d16543e6ac18f82%5E%21/#F0 ` dif... [23:00:00] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11752056 (10bd808) I nerd sniped @thcipriani into looking at this and he has spotted a pattern in the traffic that looks like something we could block on, but unfortunately it is not IP... [23:04:59] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work, 13Patch-For-Review: Set up zuul web on zuul1001/zuul2001 - https://phabricator.wikimedia.org/T405119#11752074 (10Dzahn) >>! In T405119#11746081, @dduvall wrote: > @Dzahn circling back to this. Thanks for pus... [23:08:46] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work, 13Patch-For-Review: Set up zuul web on zuul1001/zuul2001 - https://phabricator.wikimedia.org/T405119#11752106 (10dduvall) Per our IRC discussion, it may be a good idea long-term to refactor `profile::pki::ge... [23:12:19] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Radar), 06SRE, 06Traffic: Beta cluster seems to be extremely slow for logged in user during page navigation - https://phabricator.wikimedia.org/T267435#11752112 (10bd808) 05Openβ†’03Declined let's stop chasing this one. [23:35:41] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work, 13Patch-For-Review: Set up zuul web on zuul1001/zuul2001 - https://phabricator.wikimedia.org/T405119#11752163 (10Dzahn) After the cert issues were fixed in puppet we got to that database error I mentioned be... [23:36:10] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work, 13Patch-For-Review: Set up zuul web on zuul1001/zuul2001 - https://phabricator.wikimedia.org/T405119#11752166 (10Dzahn) 05Openβ†’03Resolved [23:36:59] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11752168 (10AlexisJazz) Microsoft I get, as they sell Azure cloud services. DigitalOcean is also a cloud provider, Czech "ONEHOSTPLANET" sounds like one and Earthlink.iq offers at least... [23:50:25] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#11752176 (10Dzahn) zuul-web now works (details in subtask!) :) We have a new issue with zuul-scheduler (T421330). It's similar to T405119#11752163 Thi... [23:54:42] 10Phabricator: Move "Task search" to the top of the default phabricator sidebar nav - https://phabricator.wikimedia.org/T420801#11752192 (10A_smart_kitten) > The search in the upper-right is the global search, not the task search. > > But I still use it to search for tasks because: > > 1. it's RIGHT THERE, and...