[06:28:42] (03CR) 10Hashar: [C:03+2] "πŸŽ‰" [integration/docroot] - 10https://gerrit.wikimedia.org/r/1183206 (https://phabricator.wikimedia.org/T402398) (owner: 10Krinkle) [06:29:24] (03Merged) 10jenkins-bot: CoveragePage: Refactor $crumbs by slug instead of page title [integration/docroot] - 10https://gerrit.wikimedia.org/r/1183206 (https://phabricator.wikimedia.org/T402398) (owner: 10Krinkle) [06:30:17] (03PS2) 10Krinkle: CoveragePage: Nicer page titles [integration/docroot] - 10https://gerrit.wikimedia.org/r/1183207 (https://phabricator.wikimedia.org/T402398) [06:30:25] (03PS2) 10Krinkle: CoveragePage: Hide link to cover-skins [integration/docroot] - 10https://gerrit.wikimedia.org/r/1183208 (https://phabricator.wikimedia.org/T402398) [06:30:42] (03CR) 10Hashar: [C:03+2] CoveragePage: Nicer page titles [integration/docroot] - 10https://gerrit.wikimedia.org/r/1183207 (https://phabricator.wikimedia.org/T402398) (owner: 10Krinkle) [06:31:07] (03CR) 10Hashar: [C:03+2] CoveragePage: Hide link to cover-skins (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/1183208 (https://phabricator.wikimedia.org/T402398) (owner: 10Krinkle) [06:31:36] (03Merged) 10jenkins-bot: CoveragePage: Nicer page titles [integration/docroot] - 10https://gerrit.wikimedia.org/r/1183207 (https://phabricator.wikimedia.org/T402398) (owner: 10Krinkle) [06:31:47] (03Merged) 10jenkins-bot: CoveragePage: Hide link to cover-skins [integration/docroot] - 10https://gerrit.wikimedia.org/r/1183208 (https://phabricator.wikimedia.org/T402398) (owner: 10Krinkle) [06:42:30] (03CR) 10Hashar: [C:04-1] "The v48 codesniffer seems to cause a wrong indentation?" [integration/docroot] - 10https://gerrit.wikimedia.org/r/1185446 (owner: 10Libraryupgrader) [06:48:26] cd/clear [06:48:29] :) [06:49:57] 10Continuous-Integration-Infrastructure (Zuul upgrade): Investigate how Zuul finger gateway works - https://phabricator.wikimedia.org/T403734#11156116 (10hashar) 05Openβ†’03Resolved a:03hashar Done: we do not have to setup the finger gateway. [07:22:58] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Phorge setup check caching is misbehaving, leading to many duck-sound=quack requests - https://phabricator.wikimedia.org/T401157#11156213 (10hashar) Note that despite `apc.shm_size` being set to `4096M`, the UI shows it as: | Memory Us... [08:33:19] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Phorge setup check caching is misbehaving, leading to many duck-sound=quack requests - https://phabricator.wikimedia.org/T401157#11156425 (10Aklapper) @hashar: Uh, thanks for digging into that. I guess one of us should file an upstream... [09:50:02] 10GitLab, 10bacula, 10Data-Persistence-Backup: Split repository backups (gerrit and gitlab) into its own dedicated storage daemons - https://phabricator.wikimedia.org/T403946 (10jcrespo) 03NEW [09:50:13] 10GitLab, 10bacula, 10Data-Persistence-Backup: Split repository backups (gerrit and gitlab) into its own dedicated storage daemons - https://phabricator.wikimedia.org/T403946#11156795 (10jcrespo) p:05Triageβ†’03High [09:52:04] 10GitLab, 10bacula, 10Data-Persistence-Backup: Split repository backups (gerrit and gitlab) into its own dedicated storage daemons - https://phabricator.wikimedia.org/T403946#11156802 (10jcrespo) This needs to be done ASAP due to the storage size constraints that started this weekend (backups got stuck for 4... [10:07:13] 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on the test Phabricator instance (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11156855 (10LSobanski) [10:16:48] (03merge) 10aklapper: Set a name for our custom config source [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/57 (https://phabricator.wikimedia.org/T403909) [10:19:53] 10Phabricator (phabricator-next), 10Release-Engineering-Team (Doing 😎), 10Wikimedia-Phabricator-Extensions: Unnamed configuration source listed in all Settings - https://phabricator.wikimedia.org/T403909#11156930 (10Aklapper) [10:35:28] 10Release-Engineering-Team (Doing 😎), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on the test Phabricator instance (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11156958 (10Aklapper) p:05Tria... [10:46:10] 10Release-Engineering-Team (Doing 😎), 06collaboration-services, 10VPS-project-Phabricator, 13Patch-For-Review: 'Fulltext' searches fail on the test Phabricator instance (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11156998 (... [12:31:42] (03PS1) 10Hashar: jjb: move skins test coverage to /cover-extensions/ [integration/config] - 10https://gerrit.wikimedia.org/r/1185915 (https://phabricator.wikimedia.org/T402398) [12:37:10] (03CR) 10Hashar: [C:03+2] "On the doc hosts I have done some manually changes:" [integration/config] - 10https://gerrit.wikimedia.org/r/1185915 (https://phabricator.wikimedia.org/T402398) (owner: 10Hashar) [12:38:49] (03Merged) 10jenkins-bot: jjb: move skins test coverage to /cover-extensions/ [integration/config] - 10https://gerrit.wikimedia.org/r/1185915 (https://phabricator.wikimedia.org/T402398) (owner: 10Hashar) [13:07:58] (03PS1) 10Hashar: jjb: switch skins coverage to mwext-phpunit-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/1185925 (https://phabricator.wikimedia.org/T395470) [13:10:00] (03PS1) 10Hashar: zuul: switch skins coverage to same job as extensions [integration/config] - 10https://gerrit.wikimedia.org/r/1185926 (https://phabricator.wikimedia.org/T395470) [13:13:57] (03PS2) 10Hashar: zuul: switch skins coverage to same job as extensions [integration/config] - 10https://gerrit.wikimedia.org/r/1185926 (https://phabricator.wikimedia.org/T395470) [13:19:31] FIRING: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:19:38] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T403963 (10phaultfinder) 03NEW [13:20:35] (03PS3) 10Hashar: zuul: switch skins coverage to same job as extensions [integration/config] - 10https://gerrit.wikimedia.org/r/1185926 (https://phabricator.wikimedia.org/T395470) [13:20:35] (03PS1) 10Hashar: jjb: delete mwskin-phpunit-coverage-publish, unused [integration/config] - 10https://gerrit.wikimedia.org/r/1185928 (https://phabricator.wikimedia.org/T402398) [13:20:36] (03PS1) 10Hashar: zuul: remove remnant of 'mwskin' [integration/config] - 10https://gerrit.wikimedia.org/r/1185929 (https://phabricator.wikimedia.org/T402398) [13:20:38] (03PS1) 10Hashar: quibble-coverage: remove mwskin-phpunit-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/1185930 (https://phabricator.wikimedia.org/T402398) [13:20:40] (03PS1) 10Hashar: jjb: update jobs to remove mwskin-phpunit-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/1185931 (https://phabricator.wikimedia.org/T402398) [13:21:41] its me ↑ no worries, not prod [13:23:01] (03CR) 10CI reject: [V:04-1] jjb: delete mwskin-phpunit-coverage-publish, unused [integration/config] - 10https://gerrit.wikimedia.org/r/1185928 (https://phabricator.wikimedia.org/T402398) (owner: 10Hashar) [13:28:18] Krinkle: I think I have finished the series of patches that would remove the last usage of phpunit.php AND get rid of mwskin [13:28:22] T395470 [13:28:23] T395470: Drop deprecated phpunit.php and suite.xml - https://phabricator.wikimedia.org/T395470 [13:28:28] T402398 [13:28:28] T402398: Phase out "skin" terminology from CI config, use "extension" instead - https://phabricator.wikimedia.org/T402398 [13:29:15] I have to make a change to mediawiki core so that the phpunit `extensions:unit`Β test suite recognizes skins [13:29:16] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1185862 [13:30:21] earlier today I have also made skins in https://doc.wikimedia.org/cover-skins/ to be symbolic links to the equivalent in /cover-extensions/ where I have moved the skins coverage reports [13:30:50] so we can switch the mwskin job to use the same entry point as extensions https://gerrit.wikimedia.org/r/c/integration/config/+/1185925 [13:30:58] and the rest of the patches in the series are merely clean up patches [13:31:05] [13:32:25] hashar: I initially had the symlinks in the other direction because I assume docpublisher jobs would overwrite a symlink in cover-skins after the next gerrit merge in those repos [13:32:34] but maybe rsync follows the symlink for the first directory? [13:32:39] maybe [13:32:43] I haven't checked :] [13:32:49] either way, after that change goes live we can fix it if needed [13:32:53] so I have swapped the coverage reports and symlinks [13:32:59] k [13:33:03] (03CR) 10Krinkle: [C:03+1] jjb: switch skins coverage to mwext-phpunit-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/1185925 (https://phabricator.wikimedia.org/T395470) (owner: 10Hashar) [13:33:29] my idea is to trigger the coverage job for one of the skins and verify how it behaves [13:33:46] I also rediscovered code health [13:33:54] and I am wondering whether anyone is actively using those [13:34:13] hashar: do we need the core change in other branches or only master? [13:34:19] * Krinkle lands https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1185862 [13:34:31] only master afaik [13:34:36] okay [13:35:01] are there other jobs that use skins:unit or something like that besides coverage jobs? [13:35:10] I'm aware code coverage is only master [13:35:30] tests/phpunit/suites/ExtensionsUnitTestSuite.php is used by the `extension:unit`Β test suite and I could only find one usage which is the code health coverage report [13:35:47] okay [13:36:07] and it looks like that is active on all branches grr [13:37:22] and I could only find one usecase for skins:unit which is the phpunit coverage report for the codehealth system [13:37:45] (because sonarqube does not have a full install of mediawiki, the enrtypoint was added to let it run solely the unit tests) [14:01:01] hashar: Hello. [14:01:01] RESOLVED: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:01:11] yeah [14:01:14] hold please :) [14:01:15] mic issue [14:01:24] I blame the Romans. [14:01:44] I blame audio on linux [14:10:54] but I thought 2025 was the Year of the Linux Desktop? [14:13:58] (03PS1) 10Sergio Gimeno: zuul: [mediawiki/extensions/MetricsPlatform] Add to gate [integration/config] - 10https://gerrit.wikimedia.org/r/1185945 [14:14:25] James_F: ^ :) [14:15:36] (03CR) 10Jforrester: [C:04-1] "This needs a significant discussion first, and then sign-off from Release Engineering. Where is the Phab task?" [integration/config] - 10https://gerrit.wikimedia.org/r/1185945 (owner: 10Sergio Gimeno) [14:17:49] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T403963#11157777 (10ABran-WMF) 05Openβ†’03Resolved p:05Triageβ†’03Medium a:03ABran-WMF caused by: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1185888 revert has been merged [14:18:15] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - http_gerrit_tls_ip4 gerrit2003 - https://phabricator.wikimedia.org/T403963#11157783 (10ABran-WMF) [14:18:31] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - http_gerrit_tls_ip4 gerrit2003 - https://phabricator.wikimedia.org/T403963#11157784 (10ABran-WMF) [14:44:52] 10GitLab (Integrations), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, 06serviceops: Container image reports in debmonitor are broken - https://phabricator.wikimedia.org/T348876#11157919 (10LSobanski) a:03elukey [14:47:36] 06Release-Engineering-Team, 10dev-images, 10MediaWiki-Docker: Switch PHP 8.3 image of MediaWiki-Docker from Sury to WMF - https://phabricator.wikimedia.org/T401255#11157946 (10Jdforrester-WMF) Unfortunately the dev-images are bookworm-based, and there's currently no `component/php83` for `bookworm-wikimedia`... [14:52:07] (03CR) 10Jforrester: "Should we land this?" [integration/config] - 10https://gerrit.wikimedia.org/r/1182115 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [15:00:40] (03CR) 10Hashar: "I forgot to inform @krinkle@fastmail.com that I wrote that script based on the commit message he wrote at Ia268663f87efbf5f6d4e37a4a4ffff2" [integration/config] - 10https://gerrit.wikimedia.org/r/1182115 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [15:02:00] 10Continuous-Integration-Config, 10Math, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Allow control over which extra extensions are installed (Math REL1_43 jobs exceed 60min timeout) - https://phabricator.wikimedia.org/T389998#11158060 (10Reedy) xref {T182472} too.. [15:04:26] # Only run patch coverage if a PHP file was changed on master. [15:04:26] - name: ^.*-phpunit-coverage-patch [15:04:26] branch: master [15:04:28] magic [15:20:21] 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 06Reader Experience Team, 13Patch-For-Review: ReadingLists tests not run in jenkins wmf-quibble-core-vendor-mysql-php81 - https://phabricator.wikimedia.org/T403560#11158237 (10hashar) > For ReadingLists, I think we mainly care about chan... [15:22:27] (03open) 10dancy: staging.tfvars: buildkitd_tag = "wmf-v0.24.0" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/504 (https://phabricator.wikimedia.org/T403625) [15:22:30] (03update) 10dancy: staging.tfvars: buildkitd_tag = "wmf-v0.24.0" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/504 (https://phabricator.wikimedia.org/T403625) [15:23:52] (03merge) 10dancy: staging.tfvars: buildkitd_tag = "wmf-v0.24.0" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/504 (https://phabricator.wikimedia.org/T403625) [15:25:03] maintenance-disconnect-full-disks build 735283 integration-agent-docker-1062 (/: 24%, /srv: 98%, /var/lib/docker: 28%): OFFLINE due to disk space [15:28:48] !log Upgraded gitlab-cloud-runners (staging) buildkitd to v0.24.0 [15:28:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:30:03] maintenance-disconnect-full-disks build 735284 integration-agent-docker-1045 (/: 33%, /srv: 98%, /var/lib/docker: 30%): OFFLINE due to disk space [15:30:03] maintenance-disconnect-full-disks build 735284 integration-agent-docker-1062 (/: 24%, /srv: 85%, /var/lib/docker: 28%): RECOVERY disk space OK [15:34:32] (03open) 10dancy: prod.tfvars: buildkitd_tag = "wmf-v0.24.0" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/505 (https://phabricator.wikimedia.org/T403625) [15:34:34] (03update) 10dancy: prod.tfvars: buildkitd_tag = "wmf-v0.24.0" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/505 (https://phabricator.wikimedia.org/T403625) [15:35:03] maintenance-disconnect-full-disks build 735285 integration-agent-docker-1045 (/: 33%, /srv: 16%, /var/lib/docker: 29%): RECOVERY disk space OK [15:35:41] (03merge) 10dancy: prod.tfvars: buildkitd_tag = "wmf-v0.24.0" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/505 (https://phabricator.wikimedia.org/T403625) [15:38:11] 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 06Reader Experience Team, 13Patch-For-Review: ReadingLists tests not run in jenkins wmf-quibble-core-vendor-mysql-php81 - https://phabricator.wikimedia.org/T403560#11158409 (10hashar) An alternative would be to express the dependencies b... [15:38:26] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Deploy zuul executor on executor VM - https://phabricator.wikimedia.org/T403847#11158414 (10LSobanski) 05Openβ†’03In progress p:05Triageβ†’03High [15:50:24] !log Upgraded gitlab-cloud-runners (production) buildkitd to v0.24.0 [15:50:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:06:34] (03open) 10dancy: README.md: Remove reference to removed file [repos/releng/buildkit] (wmf/build) - 10https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/71 [16:06:37] (03update) 10dancy: README.md: Remove reference to removed file [repos/releng/buildkit] (wmf/build) - 10https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/71 [16:09:20] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Deploy zuul executor on executor VM - https://phabricator.wikimedia.org/T403847#11158614 (10hashar) >>! In T403847#11153712, @Dzahn wrote: > executor.manage_ansible > > ` > It... [16:09:47] (03merge) 10dancy: Remove .gitreview [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/998 [16:12:34] (03update) 10dancy: README.md: Remove reference to removed file [repos/releng/buildkit] (wmf/build) - 10https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/71 [16:33:37] 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 06Reader Experience Team, 13Patch-For-Review: ReadingLists tests not run in jenkins wmf-quibble-core-vendor-mysql-php81 - https://phabricator.wikimedia.org/T403560#11158791 (10Jdlrobson-WMF) [17:25:46] (03update) 10dancy: Python builder: Always use a virtualenv [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/83 (https://phabricator.wikimedia.org/T357548) [17:31:49] (03merge) 10dancy: README.md: Remove reference to removed file [repos/releng/buildkit] (wmf/build) - 10https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/71 [17:32:49] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: buildkit v0.24.0 released - https://phabricator.wikimedia.org/T403625#11159078 (10dancy) Deployed to gitlab-cloud-runners, wmcs, and trusted runners. [17:33:03] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: buildkit v0.24.0 released - https://phabricator.wikimedia.org/T403625#11159082 (10dancy) 05Openβ†’03Resolved p:05Triageβ†’03Medium [17:43:03] (03PS2) 10Umherirrender: build: Updating mediawiki/mediawiki-codesniffer to 48.0.0 [integration/docroot] - 10https://gerrit.wikimedia.org/r/1185446 (owner: 10Libraryupgrader) [17:43:07] (03CR) 10Umherirrender: build: Updating mediawiki/mediawiki-codesniffer to 48.0.0 (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/1185446 (owner: 10Libraryupgrader) [17:54:49] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Deploy zuul executor on executor VM - https://phabricator.wikimedia.org/T403847#11159166 (10Dzahn) Aha! Thanks for pointing that out. Good. [18:48:49] (03CR) 10Hashar: [C:03+2] build: Updating mediawiki/mediawiki-codesniffer to 48.0.0 (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/1185446 (owner: 10Libraryupgrader) [19:04:04] 10Beta-Cluster-Infrastructure, 10RESTBase, 07Beta-Cluster-reproducible, 13Patch-For-Review: HyperSwitch/errors/not found (404) on beta cluster: There was an issue displaying this preview - https://phabricator.wikimedia.org/T402206#11159415 (10Jdlrobson-WMF) Any ideas on the timeline for this? I am a little... [19:07:38] (03PS1) 10Umherirrender: Zuul: [../extensions/AutoModerator] Add CommunityConfiguration for phan [integration/config] - 10https://gerrit.wikimedia.org/r/1186034 [19:10:29] 10Beta-Cluster-Infrastructure, 10Automoderator, 10Moderator-Tools-Team (Kanban), 07Wikimedia-production-error: AutoModerator WikiPageConfig::getConfigData failed to load config from wiki: {error} - https://phabricator.wikimedia.org/T403756#11159446 (10Scardenasmolinar) This looks to be an error in the `Aut... [19:13:09] (03CR) 10Hashar: [C:03+2] Zuul: [../extensions/AutoModerator] Add CommunityConfiguration for phan [integration/config] - 10https://gerrit.wikimedia.org/r/1186034 (owner: 10Umherirrender) [19:14:46] (03Merged) 10jenkins-bot: Zuul: [../extensions/AutoModerator] Add CommunityConfiguration for phan [integration/config] - 10https://gerrit.wikimedia.org/r/1186034 (owner: 10Umherirrender) [19:21:10] !log integration-castor05: deleting some old branches from /srv/castor/castor-mw-ext-and-skins/ [19:21:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:22:01] 10Beta-Cluster-Infrastructure, 10RESTBase, 07Beta-Cluster-reproducible, 13Patch-For-Review: HyperSwitch/errors/not found (404) on beta cluster: There was an issue displaying this preview - https://phabricator.wikimedia.org/T402206#11159512 (10Krinkle) My idea/patch here is a long-term improvement to solve... [19:23:11] !log integration-castor05: deleting php74 caches [19:23:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:27:05] 10Beta-Cluster-Infrastructure, 10Automoderator, 10Moderator-Tools-Team (Kanban), 07Wikimedia-production-error: AutoModerator WikiPageConfig::getConfigData failed to load config from wiki: {error} - https://phabricator.wikimedia.org/T403756#11159521 (10Scardenasmolinar) The `Talk page message` field has bee... [19:31:53] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Deploy zuul executor on executor VM - https://phabricator.wikimedia.org/T403847#11159575 (10thcipriani) >>! In T403847#11153698, @Dzahn wrote: > We need an `/etc/zuul/zuul.conf`... [19:36:41] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Deploy zuul executor on executor VM - https://phabricator.wikimedia.org/T403847#11159623 (10Dzahn) Yes, let me try the simplest option first and ensure the same config file is de... [19:46:53] 10Beta-Cluster-Infrastructure, 10Automoderator, 10Moderator-Tools-Team (Kanban), 07Wikimedia-production-error: AutoModerator WikiPageConfig::getConfigData failed to load config from wiki: {error} - https://phabricator.wikimedia.org/T403756#11159657 (10Krinkle) [20:03:42] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 13Patch-For-Review: Deploy zuul executor on executor VM - https://phabricator.wikimedia.org/T403847#11159738 (10Dzahn) Next step to debug is: ` 2025-09-08 20:01:28,869 ERROR... [20:09:10] !log Unblock 201.141.104.0/22 (T403996) [20:09:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:20:24] (03PS1) 10RLazarus: helm-linter: Bump for envoy 1.29.12 [integration/config] - 10https://gerrit.wikimedia.org/r/1186056 (https://phabricator.wikimedia.org/T403663) [20:20:26] (03PS1) 10RLazarus: jjb: Update to helm-linter:0.7.4 to pick up envoy-future 1.29.12 [integration/config] - 10https://gerrit.wikimedia.org/r/1186057 (https://phabricator.wikimedia.org/T403663) [20:25:32] ahoy releng - anyone in Americas timezones able to review those two? ^ happy to wait for h.ashar tomorrow if need be :) [21:00:14] rzl: Do they have to be in a US TZ? :P [21:00:43] 10Beta-Cluster-Infrastructure, 10RESTBase, 07Beta-Cluster-reproducible, 13Patch-For-Review: HyperSwitch/errors/not found (404) on beta cluster: There was an issue displaying this preview - https://phabricator.wikimedia.org/T402206#11160082 (10Tgr) >>! In T402206#11159512, @Krinkle wrote: > I could not find... [21:01:42] Reedy: no, and I would never say anything out loud about someone else's work-life balance! :D [21:01:55] heh [21:02:05] (03CR) 10Reedy: [C:03+2] helm-linter: Bump for envoy 1.29.12 [integration/config] - 10https://gerrit.wikimedia.org/r/1186056 (https://phabricator.wikimedia.org/T403663) (owner: 10RLazarus) [21:02:11] let's see if I can get it to build the docker image [21:02:19] thanks <3 [21:02:46] that's the hard part ;D [21:03:53] (03Merged) 10jenkins-bot: helm-linter: Bump for envoy 1.29.12 [integration/config] - 10https://gerrit.wikimedia.org/r/1186056 (https://phabricator.wikimedia.org/T403663) (owner: 10RLazarus) [21:04:57] !log Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1186056 T403663 [21:05:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:05:00] T403663: Upgrade Envoy to v1.29.12 - https://phabricator.wikimedia.org/T403663 [21:05:08] now we wait [21:13:55] rzl: image is pushed.. not shown on https://docker-registry.wikimedia.org/releng/helm-linter/tags/ (yay, caching with weird bugs) [21:14:36] (03CR) 10Reedy: [C:03+2] jjb: Update to helm-linter:0.7.4 to pick up envoy-future 1.29.12 [integration/config] - 10https://gerrit.wikimedia.org/r/1186057 (https://phabricator.wikimedia.org/T403663) (owner: 10RLazarus) [21:15:12] Reedy: ah yeah that's got me too -- it's a static HTML page regenerated on a schedule, rather than a live view [21:15:27] i can get two different versions from the cache, seemingly [21:15:30] just for added confusing [21:15:31] I forget how fast the update is but it's long enough to notice [21:15:34] oh fun! [21:16:12] (03Merged) 10jenkins-bot: jjb: Update to helm-linter:0.7.4 to pick up envoy-future 1.29.12 [integration/config] - 10https://gerrit.wikimedia.org/r/1186057 (https://phabricator.wikimedia.org/T403663) (owner: 10RLazarus) [21:16:13] yeah... And it also doesn't tell you what TZ it is either [21:16:49] 06Release-Engineering-Team, 06SRE, 07Puppet: docker-registry "Last updated at" text hiding under scrollbar - https://phabricator.wikimedia.org/T404008#11160187 (10Reedy) [21:17:42] 06Release-Engineering-Team, 06SRE: docker-registry "Last updated at" text hiding under scrollbar - https://phabricator.wikimedia.org/T404008#11160190 (10Reedy) [21:17:46] 06Release-Engineering-Team, 06SRE: docker-registry "Last updated at" time should specify TZ - https://phabricator.wikimedia.org/T404010#11160191 (10Reedy) [21:17:50] 06Release-Engineering-Team, 06SRE: docker-registry will show different last updated TZ as you refresh the page... - https://phabricator.wikimedia.org/T404011#11160192 (10Reedy) [21:21:46] INFO:jenkins_jobs.cli.subcommand.update:Number of jobs updated: 1 [21:21:52] rzl: Got a patch to recheck? [21:22:40] !log deployed updated helm-lint job https://gerrit.wikimedia.org/r/1186057 T403663 [21:22:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:22:43] T403663: Upgrade Envoy to v1.29.12 - https://phabricator.wikimedia.org/T403663 [21:26:30] !log Unblock 143.179.0.0/16 (T404012) [21:26:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:31:04] Reedy: ah sorry, away from my desk -- I can try it in a little bit, or honestly any helm-lint job is fine [21:58:02] (03CR) 10Jforrester: Zuul: [../extensions/AutoModerator] Add CommunityConfiguration for phan (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1186034 (owner: 10Umherirrender) [22:15:48] (03open) 10dancy: Revert buildkit to wmf-v0.22.0 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/506 (https://phabricator.wikimedia.org/T403625) [22:15:49] (03update) 10dancy: Revert buildkit to wmf-v0.22.0 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/506 (https://phabricator.wikimedia.org/T403625) [22:15:59] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work, 13Patch-For-Review: buildkit v0.24.0 released - https://phabricator.wikimedia.org/T403625#11160442 (10dancy) 05Resolvedβ†’03Open [22:16:52] (03merge) 10dancy: Revert buildkit to wmf-v0.22.0 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/506 (https://phabricator.wikimedia.org/T403625) [22:20:33] (03CR) 10Daimona Eaytoy: build: Updating mediawiki/mediawiki-codesniffer to 48.0.0 (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/1185446 (owner: 10Libraryupgrader) [22:34:49] !log Revert gitlab-cloud-runners buildkit to v0.20.0 [22:34:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:36:13] Reedy: belatedly, looking good, thanks! [22:36:25] yay [22:48:15] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Deploy zuul executor on executor VM - https://phabricator.wikimedia.org/T403847#11160535 (10thcipriani) >>! In T403847#11159738, @Dzahn wrote: > Next step to debug is: > > > `... [22:51:46] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#11160549 (10HFan-WMF) Hi there, I'm the new product manager on the Reader Experience team, and would love to be able to create new milestones and subprojects f... [22:51:57] 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 06Reader Experience Team, 13Patch-For-Review: ReadingLists tests not run in jenkins wmf-quibble-core-vendor-mysql-php81 - https://phabricator.wikimedia.org/T403560#11160551 (10Jdlrobson-WMF) Thanks everyone for sharing your thoughts and... [23:09:16] 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 06Reader Experience Team, 13Patch-For-Review: ReadingLists tests not run in jenkins wmf-quibble-core-vendor-mysql-php81 - https://phabricator.wikimedia.org/T403560#11160600 (10bd808) >>! In T403560#11160551, @Jdlrobson-WMF wrote: > 3. Wh...