[01:12:37] 10Release-Engineering-Team (Quid Pro Crow 🦃), 10Release, 10Train Deployments: 1.42.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T350081 (10Jdlrobson) Heads up that T351362 is likely to trigger our client error alerting systems with roll out of the train. User impact appears relatively low,... [01:16:02] 10Phabricator, 10I18n: Phabricator translate export script should not overwrite manual edits to qqq.json - https://phabricator.wikimedia.org/T351363 (10Pppery) [01:39:34] 10Phabricator, 10I18n: Phabricator translate export script should not overwrite manual edits to qqq.json - https://phabricator.wikimedia.org/T351363 (10Pppery) Relevant lost manual changes seem to be: (more to be added) - Phabricator:phabricator-dashboard-f090ac2cb105c374/qqq - Phabricator:phabricator-das... [02:15:52] 10Phabricator, 10I18n: Phabricator translate export script should not overwrite manual edits to qqq.json - https://phabricator.wikimedia.org/T351363 (10Pppery) >>! In T351363#9335927, @Pppery wrote: > Phabricator:phabricator-audit-d8de960f1cc3d44b/qqq (although this is a FIXME that really shouldn't be in qqq)... [05:59:18] 10Fresh, 10ARM support, 10User-zeljkofilipin: ECONNREFUSED error when running Selenium tests on M1 Mac - https://phabricator.wikimedia.org/T308889 (10Peter) I was able to hack Fresh so it worked for me. Changing the base image to a version that was built for linux/arm64 and then removed the --platform linux/... [10:28:19] 10Phabricator: Phabricator: Simplify the multifactor auth reset procedure - https://phabricator.wikimedia.org/T231667 (10Aklapper) 05Open→03Declined Whatever may happen or not in the future regarding this ticket, it will not happen in downstream by maintaining a custom patch, thus declining this ticket. See... [10:50:21] 10Beta-Cluster-Infrastructure, 10RESTBase, 10Patch-For-Review: RESTbase broken on beta cluster - https://phabricator.wikimedia.org/T351193 (10hnowlan) It seems that restbase has had `graphite_host` set as an option since forever, and in production instead of removing that as a parameter, `graphite_host` was... [10:55:04] 10Beta-Cluster-Infrastructure, 10RESTBase, 10Patch-For-Review: RESTbase broken on beta cluster - https://phabricator.wikimedia.org/T351193 (10TheresNoTime) Thanks @hnowlan, all working afaict too :-) [10:55:51] 10Beta-Cluster-Infrastructure, 10RESTBase, 10Patch-For-Review: RESTbase broken on beta cluster - https://phabricator.wikimedia.org/T351193 (10TheresNoTime) p:05High→03Triage [11:58:38] 10Continuous-Integration-Infrastructure, 10Code-Health, 10Developer Productivity: Regularly generate a dependency graph for MediaWiki code - https://phabricator.wikimedia.org/T351336 (10daniel) [12:08:05] 10GitLab (Infrastructure), 10collaboration-services: Record metrics for Gitlab backup/restore times - https://phabricator.wikimedia.org/T347593 (10eoghan) 05In progress→03Resolved [12:08:19] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Code-Health, 10Developer Productivity: In CI, only run tests for code that is affected by a given change - https://phabricator.wikimedia.org/T351357 (10kostajh) > Currently, we run all (core) test for every iteration of every change on gerr... [12:09:49] 10Continuous-Integration-Infrastructure, 10Code-Health, 10Developer Productivity: Regularly generate a dependency graph for MediaWiki code - https://phabricator.wikimedia.org/T351336 (10daniel) [13:12:57] 10Beta-Cluster-Infrastructure: No entries at all in beta-logs.wmcloud.org since 2023-11-06 Z 12:15:39 - https://phabricator.wikimedia.org/T350786 (10kostajh) In the past, restarting Logstash ({T274593}) got things working again. [13:16:52] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: No entries at all in beta-logs.wmcloud.org since 2023-11-06 Z 12:15:39 - https://phabricator.wikimedia.org/T350786 (10kostajh) p:05Triage→03High I don't know if this should be considered a train blocking task (tagging #release-engineering-team for... [13:28:38] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Code-Health, 10Developer Productivity: In CI, only run tests for code that is affected by a given change - https://phabricator.wikimedia.org/T351357 (10daniel) >>! In T351357#9336802, @kostajh wrote: > Sorry to nitpick, but I think it's clo... [13:29:05] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Code-Health, 10Developer Productivity: In CI, only run tests for code that is affected by a given change - https://phabricator.wikimedia.org/T351357 (10daniel) [13:32:07] 10Beta-Cluster-Infrastructure, 10Wikimedia-Logstash, 10observability, 10User-DannyS712: Logstash beta is not getting any events - https://phabricator.wikimedia.org/T274593 (10jbond) i have logged into `logging-logstash-02.logging.eqiad1.wikimedia.cloud` and ran `systemctl restart logstash.service` hopefull... [13:32:37] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: No entries at all in beta-logs.wmcloud.org since 2023-11-06 Z 12:15:39 - https://phabricator.wikimedia.org/T350786 (10jbond) i have logged into `logging-logstash-02.logging.eqiad1.wikimedia.cloud` and ran `systemctl restart logstash.service` hopefully... [13:32:46] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Code-Health, 10Developer Productivity: In CI, only run tests for code that is affected by a given change - https://phabricator.wikimedia.org/T351357 (10Jdforrester-WMF) This has been discussed extensively many times, with lots of discussion... [13:33:30] 10GitLab, 10Release-Engineering-Team (Quid Pro Crow 🦃), 10cloud-services-team, 10Cloud-VPS (Project-requests), 10User-brennen: Request creation of devel-stats VPS project - https://phabricator.wikimedia.org/T351330 (10rook) 05Open→03In progress a:03rook [13:38:27] 10GitLab, 10Release-Engineering-Team (Quid Pro Crow 🦃), 10cloud-services-team, 10Cloud-VPS (Project-requests), 10User-brennen: Request creation of devel-stats VPS project - https://phabricator.wikimedia.org/T351330 (10rook) C'est fait! ` root@cloudcontrol1005:~# openstack project create --description 'd... [13:38:43] 10GitLab, 10Release-Engineering-Team (Quid Pro Crow 🦃), 10cloud-services-team, 10Cloud-VPS (Project-requests), 10User-brennen: Request creation of devel-stats VPS project - https://phabricator.wikimedia.org/T351330 (10rook) 05In progress→03Resolved [13:42:54] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: No entries at all in beta-logs.wmcloud.org since 2023-11-06 Z 12:15:39 - https://phabricator.wikimedia.org/T350786 (10kostajh) @jbond discovered that Logstash is in a crash loop, see {P53517} [13:44:21] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Code-Health, 10Developer Productivity: In CI, only run tests for code that is affected by a given change - https://phabricator.wikimedia.org/T351357 (10daniel) >>! In T351357#9337139, @Jdforrester-WMF wrote: > This has been discussed extens... [13:45:01] hashar: Why did you push two patches to extensions.git's REL1_23(!) branch? https://gerrit.wikimedia.org/r/q/project:mediawiki%252Fextensions+branch:REL1_23+status:merged [13:45:14] Did you mean to push to master? Nothing anywhere uses REL1_23. [14:25:06] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Code-Health, 10Developer Productivity: In CI, only run tests for code that is affected by a given change - https://phabricator.wikimedia.org/T351357 (10Jdforrester-WMF) >>! In T351357#9337226, @daniel wrote: >>>! In T351357#9337139, @Jdforr... [14:25:38] 10Beta-Cluster-Infrastructure, 10RESTBase, 10Patch-For-Review: RESTbase broken on beta cluster - https://phabricator.wikimedia.org/T351193 (10hnowlan) `graphite_host` hack removed! [14:31:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: Remove Java 8 images from integration/config - https://phabricator.wikimedia.org/T350587 (10hashar) [14:34:59] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: Remove Java 8 images from integration/config - https://phabricator.wikimedia.org/T350587 (10hashar) [14:41:34] James_F: I had to delete a couple empty mediawiki/extensions which were attached as submodules of the superproject mediawiki/extensions which in turns prevent deletion :) [14:41:48] Ah, actual gerrit repo deletion? Wow. [14:42:02] yeah :) [14:42:15] it is smart enough to detect the to be deleted project is registered as a submodule [14:42:38] for the REL branches, I guess the release branch script used to cut a branches for the super projects smediawiki/extensions and mediawiki/skins and eventually that got phased out after REL1_35 [14:42:45] I have filed their deletion as https://phabricator.wikimedia.org/T351341 [14:42:54] but I have no clue who might know whether they are of any use [14:43:12] Joy. [14:43:31] I have dug a bit in phabricator history and the branch release script but haven't found anything :\ [14:44:42] I felt that maybe that could have been used to generate mediawiki tarballs, but the release branches seem to have all extension that existed at that point [14:44:47] so most probably, that never got used [14:45:06] Yeah, we merged the scripts together. [14:45:06] and the REL branch script was just blindly cutting the branch for any repo under mediawiki/** [14:45:19] Ah. [14:47:32] so I think I will just delete them at some point :) [15:22:35] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Code-Health, 10Developer Productivity: In CI, only run tests for code that is affected by a given change - https://phabricator.wikimedia.org/T351357 (10daniel) >>! In T351357#9337449, @Jdforrester-WMF wrote: > Given the monolithic nature of... [15:31:39] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Observability-Logging, 10SRE Observability (FY2023/2024-Q2): No entries at all in beta-logs.wmcloud.org since 2023-11-06 Z 12:15:39 - https://phabricator.wikimedia.org/T350786 (10colewhite) 05Open→03Resolved a:03colewhite Logstash was crash... [15:32:32] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Code-Health, 10Developer Productivity: In CI, only run tests for code that is affected by a given change - https://phabricator.wikimedia.org/T351357 (10daniel) [15:34:52] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Code-Health, 10Developer Productivity: In CI, only run tests for code that is affected by a given change - https://phabricator.wikimedia.org/T351357 (10thcipriani) >>! In T351357#9337449, @Jdforrester-WMF wrote: > Given the monolithic natur... [15:37:01] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services, 10Patch-For-Review: Migrate to using new GitLab CI runner authentication scheme - https://phabricator.wikimedia.org/T344951 (10Jelto) [15:44:07] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services, 10Patch-For-Review: Migrate to using new GitLab CI runner authentication scheme - https://phabricator.wikimedia.org/T344951 (10Jelto) I migrated all WMCS Shared Runners to the new authentication scheme.... [15:52:16] 10Continuous-Integration-Infrastructure: Split out the PHPUnit Database tests to their own job (like we did for Selenium), as they take so long - https://phabricator.wikimedia.org/T351426 (10Jdforrester-WMF) [15:57:31] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Code-Health, 10Developer Productivity: In CI, only run tests for code that is affected by a given change - https://phabricator.wikimedia.org/T351357 (10Jdforrester-WMF) >>! In T351357#9337874, @thcipriani wrote: >>>! In T351357#9337449, @Jd... [15:57:31] Gerrit emails for replies to inline comments no longer show me the reply text, is that a new issue? [15:57:39] I feel like emails like https://tmp.lucaswerkmeister.de/gerrit-email.png used to include the comment text [15:57:57] (the comment in question is at https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/974994/1/docs/topics/options.md#253) [16:00:56] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services, 10Patch-For-Review: Migrate to using new GitLab CI runner authentication scheme - https://phabricator.wikimedia.org/T344951 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab... [16:03:35] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Observability-Logging, 10SRE Observability (FY2023/2024-Q2): No entries at all in beta-logs.wmcloud.org since 2023-11-06 Z 12:15:39 - https://phabricator.wikimedia.org/T350786 (10Jdforrester-WMF) >>! In T350786#9337868, @colewhite wrote: > Logsta... [16:10:49] 10Release-Engineering-Team (Onboarding 🚀), 10Scap: scap git LFS only works for submodules - https://phabricator.wikimedia.org/T324664 (10hashar) 05Open→03Resolved a:03hashar I guess scap can be made to use the canonical git URL on the target host instead of the deployment server git url, the workaround i... [16:11:05] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Observability-Logging, 10SRE Observability (FY2023/2024-Q2): No entries at all in beta-logs.wmcloud.org since 2023-11-06 Z 12:15:39 - https://phabricator.wikimedia.org/T350786 (10kostajh) @colewhite thank you! Would you mind adding notes about ho... [16:14:23] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services: Migrate to using new GitLab CI runner authentication scheme - https://phabricator.wikimedia.org/T344951 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_... [16:23:09] 10Release-Engineering-Team (Onboarding 🚀), 10Scap: scap backport should remove code for removed submodules - https://phabricator.wikimedia.org/T341292 (10hashar) 05Open→03Resolved a:03hashar Claimed that to be resolved after doing a manual `git clean`. I don't think we need to invest time in making scap... [16:36:25] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services: Migrate to using new GitLab CI runner authentication scheme - https://phabricator.wikimedia.org/T344951 (10Jelto) [16:38:32] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services: Migrate to using new GitLab CI runner authentication scheme - https://phabricator.wikimedia.org/T344951 (10Jelto) Cloud Runners have been migrated to the new authentication scheme. Thanks @dancy for the he... [16:42:46] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for mabualruz - https://phabricator.wikimedia.org/T345186 (10hashar) 05Open→03Resolved a:03hashar We did a training and I have been told @Mabualruz successfully deployed. [16:54:16] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: Request access to deployment-prep for DJackson - https://phabricator.wikimedia.org/T351433 (10Djackson-ctr) [17:09:25] 10Release-Engineering-Team (Quid Pro Crow 🦃): Migrate R2115 Diffusion repo to 1password - https://phabricator.wikimedia.org/T351321 (10Aklapper) [17:10:16] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services, 10Patch-For-Review: Migrate to using new GitLab CI runner authentication scheme - https://phabricator.wikimedia.org/T344951 (10CodeReviewBot) dduvall opened https://gitlab.wikimedia.org/repos/releng/gitl... [17:15:57] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: Request access to deployment-prep for DJackson - https://phabricator.wikimedia.org/T351433 (10Jrbranaa) I am Derrick's manager and approve his request. [17:16:55] 10Release-Engineering-Team (Quid Pro Crow 🦃): Migrate R2115 Diffusion repo to 1password - https://phabricator.wikimedia.org/T351321 (10thcipriani) Hrm. Done. But got an exception when running `sudo /srv/phab/phabricator/bin/remove destroy ` ` Destroying objects... Destroying PhabricatorRepository .... [17:18:19] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services, 10Patch-For-Review: Migrate to using new GitLab CI runner authentication scheme - https://phabricator.wikimedia.org/T344951 (10CodeReviewBot) dduvall merged https://gitlab.wikimedia.org/repos/releng/gitl... [17:29:23] 10Release-Engineering-Team (Quid Pro Crow 🦃): Migrate R2115 Diffusion repo to 1password - https://phabricator.wikimedia.org/T351321 (10Aklapper) 05Open→03Resolved That's similar to https://we.phorge.it/T15125 and can be ignored (some older DB edges not being handled but the data is gone). [17:29:31] 10GitLab (Project Migration), 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services, and 3 others: Migrate active repositories in Phabricator Differential to GitLab - https://phabricator.wikimedia.org/T191182 (10Aklapper) [18:33:47] 10GitLab (Project Migration), 10Traffic, 10Patch-For-Review: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10BCornwall) p:05Triage→03Medium [19:23:36] 10GitLab (Project Migration), 10Traffic, 10Patch-For-Review: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10BCornwall) [20:09:59] 10GitLab (Account Approval): Question: Volunteer and WMDE staff work with two separate GitLab accounts or with a combined one? - https://phabricator.wikimedia.org/T351462 (10Celenduin) [20:57:22] 10Continuous-Integration-Infrastructure, 10Zuul: Find a way to significantly speed up the zuul-cloner step - https://phabricator.wikimedia.org/T351466 (10Jdforrester-WMF) [20:57:47] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Code-Health, 10Developer Productivity: In CI, only run tests for code that is affected by a given change - https://phabricator.wikimedia.org/T351357 (10Jdforrester-WMF) >>! In T351357#9337934, @Jdforrester-WMF wrote: > I think your work on... [21:02:05] 10Phabricator, 10DC-Ops: Update SRE Procurement Request template - https://phabricator.wikimedia.org/T305366 (10RobH) a:03ayounsi Arhzel, Can you view my proposed change above and provide feedback if that would work for you? [21:07:56] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Security: Implement an outdated modules check for golang - https://phabricator.wikimedia.org/T309997 (10sbassett) **Fake gitlab bot:** https://gitlab.wikimedia.org/repos/security/gitlab-ci-security-templates/-/merge_requests/24 [21:22:37] (03PS4) 10Jforrester: [DNM] dockerfiles: Provide php83 test image [integration/config] - 10https://gerrit.wikimedia.org/r/954337 (https://phabricator.wikimedia.org/T339350) [21:55:51] PROBLEM - SSH on contint2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [21:57:01] RECOVERY - SSH on contint2002 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [23:16:16] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Onboarding 🚀): Automatically clean up unused buildkitd PersistentVolumeClaims periodically - https://phabricator.wikimedia.org/T351478 (10dancy) [23:20:10] 10GitLab (Account Approval): Question: Volunteer and WMDE staff work with two separate GitLab accounts or with a combined one? - https://phabricator.wikimedia.org/T351462 (10bd808) There is currently no community or Wikimedia Foundation requirement for separating paid and volunteer contributions to Wikimedia's t... [23:46:15] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: Request access to deployment-prep for DJackson - https://phabricator.wikimedia.org/T351433 (10TheresNoTime) 05Open→03Resolved a:03TheresNoTime @Djackson-ctr you should now have access to [[ https://wikitech.wikimedia.org/wiki/Nova_Resource:Deploy...