[00:00:34] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:02:14] 10Phabricator, 10I18n: Phabricator translate export script should not overwrite manual edits to qqq.json - https://phabricator.wikimedia.org/T351363 (10Pppery) Plus the following from the October 2021 export: - Phabricator:phabricator-policy-83423c198b6099ed/qqq - Phabricator:phabricator-files-170fd7e84a1b2... [00:05:34] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:12:54] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: Request access to deployment-prep for DJackson - https://phabricator.wikimedia.org/T351433 (10Djackson-ctr) Great!!! Thank you @TheresNoTime [02:03:31] 10Phabricator, 10translatewiki.net, 10I18n, 10Patch-For-Review: Adding sicilian language (scn) - https://phabricator.wikimedia.org/T299694 (10Pppery) a:03Pppery [02:03:35] 10Phabricator, 10translatewiki.net, 10I18n, 10Patch-For-Review: Slovenian language in Phabricator - https://phabricator.wikimedia.org/T318762 (10Pppery) a:03Pppery [02:03:41] 10Phabricator, 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Patch-For-Review, 10User-brennen: Work out how to bring in new Phabricator translations again - https://phabricator.wikimedia.org/T318763 (10Pppery) a:03Pppery [03:26:57] 10Beta-Cluster-Infrastructure, 10Machine-Learning-Team, 10ORES, 10PageTriage, 10Patch-For-Review: Special:NewPagesFeed broken on beta cluster testwiki - https://phabricator.wikimedia.org/T349635 (10MPGuy2824) a:03MPGuy2824 [05:16:52] (03PS1) 10Subramanya Sastry: Re-enable kartographer since Parsoid supports it now [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/975112 [05:16:54] (03PS1) 10Subramanya Sastry: Re-enable DiscussionTools since Parsoid supports it now [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/975113 [06:54:08] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Code-Health, 10Developer Productivity: In CI, only run tests for code that is affected by a given change - https://phabricator.wikimedia.org/T351357 (10daniel) > But the key question is: would that speed up overall test times (for a subset... [07:25:54] 10Phabricator: Uninstall Differential (Phabricator application) - https://phabricator.wikimedia.org/T330797 (10Aklapper) [07:48:41] 10Phabricator, 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10User-brennen: Move phabricator/deployments repository from Diffusion to GitLab/Gerrit - https://phabricator.wikimedia.org/T321380 (10Aklapper) [08:02:35] hashar, jnuche: there's a request for an emergency deploy in #wikimedia-operations, also a note that scap backport is broken [08:05:10] RhinosF1: I can't get to my computer for at least an hour [08:05:17] jnuche: ok [08:05:29] hashar: can you help? [08:09:56] 10Phabricator, 10DC-Ops: Update SRE Procurement Request template - https://phabricator.wikimedia.org/T305366 (10ayounsi) a:05ayounsi→03RobH Sounds good! [08:23:39] 10Phabricator, 10I18n: Phabricator translate export script should not overwrite manual edits to qqq.json - https://phabricator.wikimedia.org/T351363 (10Nikerabbit) The current system is not sustainable. I've been scanning over the qqq changes, but there are thousands (because basically every qqq message change... [08:39:10] (03CR) 10Isabelle Hurbain-Palatin: [C: 03+2] Re-enable kartographer since Parsoid supports it now [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/975112 (owner: 10Subramanya Sastry) [08:39:54] (03Merged) 10jenkins-bot: Re-enable kartographer since Parsoid supports it now [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/975112 (owner: 10Subramanya Sastry) [08:56:06] RhinosF1: jnuche yes I can [08:56:16] sorry I haven't opened irc til now [09:04:09] hashar: thank you, will still be a while until I'm at my desk [09:14:04] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10collaboration-services, 10Patch-For-Review: Migrate to using new GitLab CI runner authentication scheme - https://phabricator.wikimedia.org/T344951 (10Jelto) [09:23:01] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10collaboration-services, 10Patch-For-Review: Migrate to using new GitLab CI runner authentication scheme - https://phabricator.wikimedia.org/T344951 (10ops-monitoring-bot) Cookbook cookbooks.sre.gitlab.upgrade was started by jel... [09:24:12] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10collaboration-services, 10Patch-For-Review: Migrate to using new GitLab CI runner authentication scheme - https://phabricator.wikimedia.org/T344951 (10ops-monitoring-bot) Cookbook cookbooks.sre.gitlab.upgrade started by jelto@c... [09:40:21] 10Phabricator, 10Patch-For-Review: Uninstall Differential (Phabricator application) - https://phabricator.wikimedia.org/T330797 (10CodeReviewBot) aklapper opened https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/28 Remove unused "Differential" application from default settings [10:03:18] 10Phabricator: Uninstall Almanac (Phabricator application) - https://phabricator.wikimedia.org/T351502 (10Aklapper) p:05Triage→03Low [10:04:48] hashar: thanks [10:04:52] Just out of a meeting [10:04:57] I see no reply from mo [10:58:47] thcipriani: please check your email [11:10:15] RhinosF1: that is handled via private messages :D [11:10:35] eventually the patch got cherry picked to the wmf branch https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MobileFrontend/+/975037 [11:10:42] which `scap backport` should be able to handle [12:40:39] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Deployment pipeline (GitLab/Kokkuri/Blubber) documentation improvements - https://phabricator.wikimedia.org/T342317 (10KBach) First version of the overview document is written and is being edited. I expect to share it for review next week... [12:46:11] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Projects-Cleanup, 10collaboration-services: Migrate SRE repositories to GitLab - Archiving unused Gerrit repositories - https://phabricator.wikimedia.org/T343707 (10LSobanski) operations/software/wmfbackups [13:07:20] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10collaboration-services, 10Patch-For-Review: Migrate to using new GitLab CI runner authentication scheme - https://phabricator.wikimedia.org/T344951 (10Jelto) [13:36:29] 10GitLab (Project Migration), 10collaboration-services: Migrate SRE repositories to GitLab - operations/software - https://phabricator.wikimedia.org/T341504 (10LSobanski) [13:40:55] 10Continuous-Integration-Infrastructure, 10Diffusion-Repository-Administrators, 10Projects-Cleanup: Consider archiving Gerrit repository "integration/junitdiff" (20130307) - https://phabricator.wikimedia.org/T351524 (10Aklapper) [13:53:51] 10Continuous-Integration-Infrastructure, 10Diffusion-Repository-Administrators, 10Projects-Cleanup: Consider archiving Gerrit repository "integration/junitdiff" (20130307) - https://phabricator.wikimedia.org/T351524 (10hashar) That was made in 2013 for Parsoid regressions testing ( https://gerrit.wikimedia.o... [13:57:09] (03PS1) 10Hashar: Archive the repository [integration/junitdiff] - 10https://gerrit.wikimedia.org/r/975278 (https://phabricator.wikimedia.org/T351524) [13:58:11] (03PS1) 10Hashar: Archive integration/junitdiff [integration/config] - 10https://gerrit.wikimedia.org/r/975280 (https://phabricator.wikimedia.org/T351524) [13:58:30] (03CR) 10Hashar: [V: 03+2 C: 03+2] Archive the repository [integration/junitdiff] - 10https://gerrit.wikimedia.org/r/975278 (https://phabricator.wikimedia.org/T351524) (owner: 10Hashar) [13:59:03] (03CR) 10Hashar: [C: 03+2] Archive integration/junitdiff [integration/config] - 10https://gerrit.wikimedia.org/r/975280 (https://phabricator.wikimedia.org/T351524) (owner: 10Hashar) [13:59:14] 10Beta-Cluster-Infrastructure, 10Wikistories: Creating a Wikistory on English betacluster results in a 502 error - https://phabricator.wikimedia.org/T351131 (10SBisson) 05Open→03Resolved a:03SBisson This appears to be fixed for the time being. [13:59:17] 10Beta-Cluster-Infrastructure, 10RESTBase: RESTbase broken on beta cluster - https://phabricator.wikimedia.org/T351193 (10SBisson) [14:00:45] (03Merged) 10jenkins-bot: Archive integration/junitdiff [integration/config] - 10https://gerrit.wikimedia.org/r/975280 (https://phabricator.wikimedia.org/T351524) (owner: 10Hashar) [14:02:03] 10Continuous-Integration-Infrastructure, 10Diffusion-Repository-Administrators, 10Projects-Cleanup, 10Patch-For-Review: Consider archiving Gerrit repository "integration/junitdiff" (20130307) - https://phabricator.wikimedia.org/T351524 (10hashar) [14:03:16] 10Continuous-Integration-Infrastructure, 10Diffusion-Repository-Administrators, 10Projects-Cleanup, 10Patch-For-Review: Consider archiving Gerrit repository "integration/junitdiff" (20130307) - https://phabricator.wikimedia.org/T351524 (10hashar) What is left to do is delete the GitHub mirror: https://gith... [14:08:54] 10Continuous-Integration-Infrastructure, 10Diffusion-Repository-Administrators, 10Projects-Cleanup, 10Patch-For-Review: Consider archiving Gerrit repository "integration/junitdiff" (20130307) - https://phabricator.wikimedia.org/T351524 (10hashar) [14:20:28] 10Continuous-Integration-Infrastructure, 10Diffusion-Repository-Administrators, 10Projects-Cleanup: Consider archiving Gerrit repository "integration/junitdiff" (20130307) - https://phabricator.wikimedia.org/T351524 (10hashar) [14:20:45] 10Continuous-Integration-Infrastructure, 10Diffusion-Repository-Administrators, 10Projects-Cleanup: Consider archiving Gerrit repository "integration/junitdiff" (20130307) - https://phabricator.wikimedia.org/T351524 (10hashar) 05Open→03Resolved a:03hashar Done! Thanks @Aklapper [14:39:49] 10Release-Engineering-Team (Quid Pro Crow đŸĻƒ), 10CAS-SSO, 10Infrastructure-Foundations, 10Patch-For-Review: Correct IDP login page Privacy Policy - https://phabricator.wikimedia.org/T350129 (10Aklapper) ping - could this get a review please? [15:14:07] 10Gerrit, 10Project-Admins, 10Projects-Cleanup, 10Wikidata: Deactivate/archive WikibaseQuery extension - https://phabricator.wikimedia.org/T134091 (10hashar) >>! In T134091#2483604, @demon wrote: > Hidden in Gerrit, deleted on Github, archived in Phab. As part of {T349543} I have changed the Gerrit reposi... [15:26:20] 10Gerrit, 10Release-Engineering-Team, 10Projects-Cleanup: Prevent Gerrit archived repositories from being replicated to GitHub - https://phabricator.wikimedia.org/T351543 (10hashar) [15:27:39] 10Gerrit, 10Release-Engineering-Team, 10Projects-Cleanup: Prevent Gerrit archived repositories from being replicated to GitHub - https://phabricator.wikimedia.org/T351543 (10hashar) [16:06:11] Hi, RelEng! I am having an issue with disk space on CI runners. You can see an example here: https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/jobs/166511 . The job is failing because there's "no disk space left on device." I have been advised that this is a shared runner. I am working to reduce the size of my image, but I'm wondering if there's anything that can be done on your side. [16:16:49] Daimona: regarding phpunit loading settings, I'm actually wondering about the opposite. From what I can see, the only thing I (personally) "want" from local settings during test is extension enablement and db config. Anything else is just asking for test failures to happen for some people but not others. [16:17:42] We could in theory keep expanding the list of rerereoverrides in TestSetup/IntegrationTestCase but maybe it's easier to move toward not loading it fully in the first place. [16:17:53] Krinkle: Extension enablement is the only thing for which we'd use LocalSettings in unit tests [16:18:06] We could build on your sub process trick and export extensions and db config only or something like that [16:18:16] And also the reason why I did the awful hack of shelling out to a separate entry point [16:18:24] Yeah I mean integration tests [16:18:33] -> so that we can get a list of extension to load, without any random side effects [16:18:41] Ah well, but for integration tests we're already loading all settings [16:18:47] And then not have to load als at all in the main process [16:19:02] Even for integration tess [16:19:27] We might consider ignoring LS for integration tests... But I feel like that would be its own can of worms [16:19:59] But in general yes, I agree: tests should not depend on random stuff in LS [16:20:27] In fact, loading LS.php *does* already cause random failures. For instance, if your local wiki is in a language other than English, you'll immediately get a few dozen test failures [16:20:29] Yeah, I think many of those worms are bugs in disguise worth knowing about, even if we do keep loading LS. [16:20:34] Yep [16:20:54] And the testing environment not being stable is also a problem in itself [16:21:02] Maybe a one off exercise to see what kinds of stuff we learn from it [16:22:25] Yes, it would definitely be interesting [16:22:34] Also [16:22:56] I think it would help with paratest [16:23:22] Or the other way around -- paratest might help us identify tests which rely on the environment. Albeit perhaps in slightly different ways. [16:29:14] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Observability-Logging, 10Quality-and-Test-Engineering-Team, 10SRE Observability (FY2023/2024-Q2): No entries at all in beta-logs.wmcloud.org since 2023-11-06 Z 12:15:39 - https://phabricator.wikimedia.org/T350786 (10kostajh) 05Resolved→03Ope... [16:29:44] 10Phabricator, 10Infrastructure-Foundations, 10Packaging, 10collaboration-services, 10Patch-For-Review: build python-phabricator package for bullseye (and bookworm?) - https://phabricator.wikimedia.org/T351333 (10Dzahn) @Volans Thank you. For some reason I never expected this phabricator specific class w... [16:46:04] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Observability-Logging, 10Quality-and-Test-Engineering-Team, 10SRE Observability (FY2023/2024-Q2): No entries at all in beta-logs.wmcloud.org since 2023-11-06 Z 12:15:39 - https://phabricator.wikimedia.org/T350786 (10colewhite) 05Open→03Resol... [17:00:37] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Observability-Logging, 10Quality-and-Test-Engineering-Team, 10SRE Observability (FY2023/2024-Q2): No entries at all in beta-logs.wmcloud.org since 2023-11-06 Z 12:15:39 - https://phabricator.wikimedia.org/T350786 (10kostajh) >>! In T350786#9341... [18:22:09] (03PS1) 10Hashar: Add Java 11 to wmf-jvm-utils [integration/config] - 10https://gerrit.wikimedia.org/r/975352 (https://phabricator.wikimedia.org/T350587) [18:35:58] 10Phabricator, 10DC-Ops: Update SRE Procurement Request template - https://phabricator.wikimedia.org/T305366 (10RobH) 05Open→03Resolved Form defaults updated for the racking section netowrking to: **Networking Setup:** # of Connections:**1**/2 - Speed:1G/10G. - VLAN:**Private**/Public/Other(Specify) : A... [19:05:51] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests: Improve speed of codehealth checks - https://phabricator.wikimedia.org/T351561 (10Daimona) [19:21:59] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10Code-Health, 10Quality-and-Test-Engineering-Team (SonarCloud Admin): Improve speed of codehealth checks - https://phabricator.wikimedia.org/T351561 (10Jdforrester-WMF) [19:22:22] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10SonarQube Bot, 10Code-Health: Improve speed of codehealth checks - https://phabricator.wikimedia.org/T351561 (10Jdforrester-WMF) [19:24:48] 10Release-Engineering-Team (Quid Pro Crow đŸĻƒ), 10Release, 10Train Deployments: 1.42.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T350081 (10thcipriani) 05Open→03Resolved [19:54:58] 10Phabricator, 10Release-Engineering-Team, 10serviceops: move "releng-secrets" git repo away from Phabricator (stop pushing over ssh to phab) - https://phabricator.wikimedia.org/T301170 (10Aklapper) What's described in the task title actually got resolved in {T351321} instead. [19:56:13] 10Phabricator, 10Release-Engineering-Team, 10serviceops: Change from ssh to pushing via https for "releng-secrets" git repo - https://phabricator.wikimedia.org/T301170 (10Aklapper) [19:56:42] 10Phabricator, 10Release-Engineering-Team, 10serviceops: Change from ssh to pushing via https for "releng-secrets" git repo - https://phabricator.wikimedia.org/T301170 (10Aklapper) [19:56:52] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10collaboration-services, 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Aklapper) [19:57:00] 10Release-Engineering-Team (Yak Shaving 🐃đŸĒ’): Migrate from pws gpg to 1password - https://phabricator.wikimedia.org/T290337 (10Aklapper) [19:57:04] 10Phabricator, 10Release-Engineering-Team, 10serviceops: Change from ssh to pushing via https for "releng-secrets" git repo - https://phabricator.wikimedia.org/T301170 (10Aklapper) [19:57:16] 10Release-Engineering-Team (Quid Pro Crow đŸĻƒ): Migrate R2115 Diffusion repo to 1password - https://phabricator.wikimedia.org/T351321 (10Aklapper) [19:57:20] 10Release-Engineering-Team (Yak Shaving 🐃đŸĒ’): Migrate from pws gpg to 1password - https://phabricator.wikimedia.org/T290337 (10Aklapper) [19:57:43] 10Release-Engineering-Team (Quid Pro Crow đŸĻƒ): Migrate R2115 Diffusion repo to 1password - https://phabricator.wikimedia.org/T351321 (10Aklapper) @thcipriani: I assume this de-facto also resolved {T290337}? [19:58:59] 10Release-Engineering-Team (Yak Shaving 🐃đŸĒ’): Migrate from pws gpg to 1password - https://phabricator.wikimedia.org/T290337 (10Dzahn) Since T351321 happened and is resolved. Is this ticket now outdated / also resolved? [20:29:37] Project beta-update-databases-eqiad build #71775: 04FAILURE in 9 min 36 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/71775/ [20:36:35] Project beta-scap-sync-world build #129844: 04FAILURE in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/129844/ [20:46:33] Yippee, build fixed! [20:46:33] Project beta-scap-sync-world build #129845: 09FIXED in 1 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/129845/ [21:19:43] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10SonarQube Bot, 10Code-Health, 10Patch-For-Review: Improve speed of codehealth checks - https://phabricator.wikimedia.org/T351561 (10kostajh) It also looks like the taint analysis takes 8 minutes ` 22:08:12 INFO: Taint analysis for php: Starting... [21:26:21] 10Release-Engineering-Team (Quid Pro Crow đŸĻƒ): Migrate R2115 Diffusion repo to 1password - https://phabricator.wikimedia.org/T351321 (10thcipriani) [21:26:26] 10Release-Engineering-Team (Yak Shaving 🐃đŸĒ’): Migrate from pws gpg to 1password - https://phabricator.wikimedia.org/T290337 (10thcipriani) 05Open→03Resolved a:03thcipriani >>! In T290337#9341910, @Dzahn wrote: > Since T351321 happened and is resolved. Is this ticket now outdated / also resolved? Sure enough! [21:26:55] 10Release-Engineering-Team (Quid Pro Crow đŸĻƒ): Migrate R2115 Diffusion repo to 1password - https://phabricator.wikimedia.org/T351321 (10thcipriani) >>! In T351321#9341900, @Aklapper wrote: > @thcipriani: I assume this de-facto also resolved {T290337}? yep, two times the productivity. [21:30:39] Yippee, build fixed! [21:30:39] Project beta-update-databases-eqiad build #71776: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/71776/ [22:00:26] (03PS1) 10BCornwall: Archive operations/debs/pybal [integration/config] - 10https://gerrit.wikimedia.org/r/975386 (https://phabricator.wikimedia.org/T347623) [22:01:37] (03CR) 10CI reject: [V: 04-1] Archive operations/debs/pybal [integration/config] - 10https://gerrit.wikimedia.org/r/975386 (https://phabricator.wikimedia.org/T347623) (owner: 10BCornwall) [22:03:07] (03PS2) 10BCornwall: Archive operations/debs/pybal [integration/config] - 10https://gerrit.wikimedia.org/r/975386 (https://phabricator.wikimedia.org/T347623) [22:15:29] 10Fresh: Composer (and PHP) support in Fresh - https://phabricator.wikimedia.org/T280914 (10Krinkle) If they are identical otherwise, I think it'd also be fine to use them by default/instead. Sounds good to me either way! [22:16:30] 10GitLab (Project Migration), 10Traffic, 10Patch-For-Review: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10BCornwall) [22:19:42] 10GitLab (Project Migration), 10Traffic, 10Patch-For-Review: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10BCornwall) [22:20:59] 10GitLab (Project Migration), 10Traffic, 10Patch-For-Review: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10BCornwall) [22:27:16] 10GitLab (Project Migration), 10Traffic, 10Patch-For-Review: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10BCornwall) [22:31:55] !log Removed 2018 cherry-pick from beta-cluster: "[BETA HACK] Attempt to secure Puppet DB better" https://gerrit.wikimedia.org/r/c/operations/puppet/+/941476 [22:31:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL