[03:16:20] * hashar screams something about gitlab [03:18:57] inflatador: you should have used `dch -i -c dockerfiles/elasticsearch/changelog` to create an entry [03:19:37] that one is wrong (date is off and the version increase went from s1 to s3) [03:19:47] the previous changelog entry from august also had a wrong date [03:21:00] 10Gerrit, 10Release-Engineering-Team: "Puppet Compiler undefined" check shown for changes on non-puppet changes - https://phabricator.wikimedia.org/T353302 (10Krinkle) [03:21:32] 10Gerrit, 10Release-Engineering-Team: "Puppet Compiler undefined" check shown for changes on non-puppet changes - https://phabricator.wikimedia.org/T353302 (10Krinkle) [03:24:31] inflatador: https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/56 :) [03:24:34] and I will build them [03:28:17] !log Built image docker-registry.wikimedia.org/dev/cirrus-elasticsearch:7.10.2-s2 # T353270 [03:28:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:28:21] T353270: Update relforge elasticsearch instance extra plugin - https://phabricator.wikimedia.org/T353270 [03:38:03] (03CR) 10Hashar: [C: 03+2] Add a banner for the 2023 developer survey [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/974166 (https://phabricator.wikimedia.org/T351109) (owner: 10Hashar) [03:38:39] (03Merged) 10jenkins-bot: Add a banner for the 2023 developer survey [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/974166 (https://phabricator.wikimedia.org/T351109) (owner: 10Hashar) [03:47:15] 10Gerrit, 10Release-Engineering-Team (Quid Pro Crow πŸ¦ƒ), 10Patch-For-Review: Add MoTD to gerrit - https://phabricator.wikimedia.org/T351109 (10hashar) 05Openβ†’03Resolved a:03hashar The Javascript plugin attaches to the [[ https://gerrit.wikimedia.org/r/Documentation/pg-plugin-endpoints.html#_banner | ban... [06:27:55] (03CR) 10Hashar: [C: 03+2] "That got delayed due to oddities with pip. https://gerrit.wikimedia.org/r/c/integration/quibble/+/963863 made the suite uses a virtualenv " [integration/quibble] - 10https://gerrit.wikimedia.org/r/958495 (https://phabricator.wikimedia.org/T345093) (owner: 10Hashar) [06:50:22] (03Merged) 10jenkins-bot: Migrate from setup.py to pyproject.toml (PEP 517) [integration/quibble] - 10https://gerrit.wikimedia.org/r/958495 (https://phabricator.wikimedia.org/T345093) (owner: 10Hashar) [07:23:43] (03PS1) 10Hashar: release: Quibble 1.6.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/982641 (https://phabricator.wikimedia.org/T345093) [07:25:04] (03PS1) 10Hashar: release: Start 1.6.1 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/982643 [07:25:51] (03CR) 10Hashar: "If you feel like reviewing the changelog entries :)" [integration/quibble] - 10https://gerrit.wikimedia.org/r/982641 (https://phabricator.wikimedia.org/T345093) (owner: 10Hashar) [07:28:35] (03PS1) 10Stang: zuul: Add MilkyDefer to allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/982646 [07:28:45] (03PS2) 10Stang: zuul: Add MilkyDefer to allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/982646 [07:36:58] (03Abandoned) 10Hashar: zuul: Add MilkyDefer to allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/982646 (owner: 10Stang) [07:46:40] 10GitLab: GitLab login 422: email has been taken - https://phabricator.wikimedia.org/T353122 (10MilkyDefer) 05Openβ†’03Resolved a:03MilkyDefer Looks like after I attached an email address and verified it in Wikitech, I regained access to my account again. Presumably the authentication part returned a null em... [07:50:45] (03Abandoned) 10Hashar: Add code coverage template for extensions and try to add one for GlobalCssJs [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/154757 (owner: 10Legoktm) [07:54:18] (03PS3) 10Hashar: ci-fullrun-extensions: Run npm install in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/760948 (owner: 10Kosta Harlan) [07:56:27] 10Release-Engineering-Team, 10Traffic, 10collaboration-services: CI on gitlab for eBPF / networking heavy projects - https://phabricator.wikimedia.org/T353279 (10LSobanski) [08:04:14] (03CR) 10Hashar: [C: 03+2] "We do not run npm install for extensions anymore due to I6f17792181bf6b5191a6c347337ae0ba806c7625 but I guess it does not hurt to always r" [integration/quibble] - 10https://gerrit.wikimedia.org/r/760948 (owner: 10Kosta Harlan) [08:06:40] (03Abandoned) 10Hashar: build: Skip npm-test for ci-fullrun scripts [integration/quibble] - 10https://gerrit.wikimedia.org/r/902665 (https://phabricator.wikimedia.org/T332386) (owner: 10Kosta Harlan) [08:24:09] 10Release-Engineering-Team, 10MediaWiki-extensions-StopForumSpam, 10ci-test-error: api.php?format=json (FormatJson.php) Allowed memory size fatal error in quibble-vendor-mysql-php74-selenium-docker - https://phabricator.wikimedia.org/T353001 (10hashar) Note that the test suite / CI should most probably not f... [08:25:21] 10Continuous-Integration-Config, 10Quibble, 10Browser-Tests, 10User-zeljkofilipin: Support for varying server configuration per browser test - https://phabricator.wikimedia.org/T248304 (10hashar) 05Openβ†’03Declined After four years, I am boldly declined it. That can always be reopened if needed. [08:45:52] 10Continuous-Integration-Config, 10Quibble, 10Browser-Tests, 10User-zeljkofilipin: Support for varying server configuration per browser test - https://phabricator.wikimedia.org/T248304 (10kostajh) Noting that we might need to do something like this for #temporary_accounts work, both for Selenium and API-Te... [08:47:31] 10Gerrit: Gerrit UI shows PCC data on non-puppet.git patches that had experimental builds - https://phabricator.wikimedia.org/T353181 (10hashar) [08:47:37] 10Gerrit, 10Release-Engineering-Team: "Puppet Compiler undefined" check shown for changes on non-puppet changes - https://phabricator.wikimedia.org/T353302 (10hashar) [08:47:47] 10Gerrit: Gerrit UI shows PCC data on non-puppet.git patches that had experimental builds - https://phabricator.wikimedia.org/T353181 (10hashar) [09:01:26] 10Release-Engineering-Team, 10Traffic, 10collaboration-services: CI on gitlab for eBPF / networking heavy projects - https://phabricator.wikimedia.org/T353279 (10Vgutierrez) we currently perform manual tests on developer machines (far from optimal). So if we can spawn our own runner we could run docker conta... [11:28:01] 10Release-Engineering-Team, 10Traffic, 10collaboration-services, 10Patch-For-Review: CI on gitlab for eBPF / networking heavy projects - https://phabricator.wikimedia.org/T353279 (10CodeReviewBot) vgutierrez opened https://gitlab.wikimedia.org/repos/sre/tcp-mss-clamper/-/merge_requests/10 e2e: Provide e2e... [11:32:03] 10Release-Engineering-Team, 10Traffic, 10collaboration-services, 10Patch-For-Review: CI on gitlab for eBPF / networking heavy projects - https://phabricator.wikimedia.org/T353279 (10Vgutierrez) @thcipriani being able to run privileged containers seems to be enough, at least for basic eBPF tests (not sure a... [12:25:15] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Create repository with configuration files for common CI use cases - https://phabricator.wikimedia.org/T352273 (10KBach) First version of the repository is now ready (https://gitlab.wikimedia.org/repos/technical-documentation/deployment-p... [12:26:13] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Create repository with configuration files for common CI use cases - https://phabricator.wikimedia.org/T352273 (10KBach) [12:54:04] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review: Deal with tox 4 upgrading - https://phabricator.wikimedia.org/T345695 (10hashar) [12:54:11] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Jenkins: Tox verbose outputs has poor contrast in Jenkins console output - https://phabricator.wikimedia.org/T347241 (10hashar) 05Openβ†’03Resolved a:03hashar Worked around by removing tox verbose messages (which are written in l... [13:53:51] (03CR) 10Jforrester: [C: 03+1] "Looks great!" [integration/quibble] - 10https://gerrit.wikimedia.org/r/982641 (https://phabricator.wikimedia.org/T345093) (owner: 10Hashar) [14:47:30] (03CR) 10Hashar: [C: 03+2] release: Quibble 1.6.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/982641 (https://phabricator.wikimedia.org/T345093) (owner: 10Hashar) [14:48:59] (03CR) 10Hashar: [C: 04-1] "Actually the parallel npm install got rolled back previously since it messes up with the way Jenkins processes the console output ( T30305" [integration/quibble] - 10https://gerrit.wikimedia.org/r/760948 (owner: 10Kosta Harlan) [14:49:08] (03CR) 10Hashar: [C: 03+2] release: Start 1.6.1 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/982643 (owner: 10Hashar) [14:50:57] James_F: looks like a data provider could be used for the poolcounter tests at https://gerrit.wikimedia.org/r/c/mediawiki/core/+/979082 :) [14:51:10] but I am not sure whether that is nitpicking :D [14:51:18] hashar: I just want the damn thing landed so we can backport. [14:51:24] heheh [14:51:37] hashar: Because the previous, non-working version was already backported to release branches, and they're getting pushed in a few days. :-( [14:52:25] let me see whether I can sprint it [14:52:48] Eh. [14:53:38] James_F: I will do it [14:53:42] should take just a few minutes [14:56:48] Argh. [14:56:51] As a follow-up, FFS. [14:57:24] Writing the test suite in a nicer way isn't a justification to hold up a merge. [15:04:32] 10MediaWiki-Releasing, 10MW-1.41-release: Write release announcement for 1.41.0-rc.0 - https://phabricator.wikimedia.org/T346928 (10Jdforrester-WMF) [15:09:07] James_F: done: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/979082/6..7/tests/phpunit/integration/includes/poolcounter/PoolCounterConnectionManagerTest.php [15:09:37] Ack. [15:10:22] and for the record, I find it rude that you get rid of my cr -1 after I said I was rewriting the test [15:10:45] I find it rude to C-1 a patch and break its merge after I said it was urgent just for a formatting fix. [15:10:59] at least you flagged thatchange required some tests to be added :-] [15:11:02] (03Merged) 10jenkins-bot: release: Quibble 1.6.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/982641 (https://phabricator.wikimedia.org/T345093) (owner: 10Hashar) [15:11:43] Yeah, I didn't like that the original had no tests, but after it broke production I was particularly unhappy that the fixed version didn't have tests either. :-( [15:12:16] hehe I hear you! [15:12:17] 10Gitlab-Application-Security-Pipeline, 10Patch-For-Review, 10SecTeam-Processed, 10Security, 10user-sbassett: Improve related gitlab test repository - https://phabricator.wikimedia.org/T348788 (10CodeReviewBot) mmartorana merged https://gitlab.wikimedia.org/repos/security/ci-cd-testing-gitlab-ci-security... [15:12:54] (03Merged) 10jenkins-bot: release: Start 1.6.1 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/982643 (owner: 10Hashar) [15:12:59] sometime I wonder whether I should I get change blocked by the job checking whether coverage has increased [15:13:23] Given that quality of its determination, no. :-) [15:13:25] (03PS1) 10Kosta Harlan: zuul: Add codehealth template to MediaModeration [integration/config] - 10https://gerrit.wikimedia.org/r/982812 [15:13:46] Foo 0% -> 100%, Bar 99.5% -> 99.4%, FAIL [15:14:03] !log Tag Quibble 1.6.0 @ 22f0d119d6ab7866d04ce99e16decfaabf9c21f7 # T345093 T345695 T348434 [15:14:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:14:09] T345093: Update Quibble setuptools check to modern alternative - https://phabricator.wikimedia.org/T345093 [15:14:10] T345695: Deal with tox 4 upgrading - https://phabricator.wikimedia.org/T345695 [15:14:10] T348434: tox v4 recreate test environment sharing the same envdir / env_dir - https://phabricator.wikimedia.org/T348434 [15:14:20] and I think the patch coverage jobs relies on some half faulty spaghetti php code [15:14:25] Yes. [15:14:34] And only tests the files touched. [15:14:43] So often you miss some coverage changes. [15:14:47] which is hidden behind something like 4 or 5 layers of land mined code [15:14:55] * James_F shudders. :-) [15:15:03] it is ranting day [15:15:24] that is going to be worse and worse up until I am in vacations this friday :troll_face: [15:15:36] Not the only one. [15:15:47] ranting? :) [15:15:57] 10Release-Engineering-Team, 10MediaWiki-extensions-StopForumSpam, 10ci-test-error: api.php?format=json (FormatJson.php) Allowed memory size fatal error in quibble-vendor-mysql-php74-selenium-docker - https://phabricator.wikimedia.org/T353001 (10sbassett) >>! In T353001#9402059, @hashar wrote: > Note that the... [15:16:04] Ranting and going on holiday on Friday. [15:16:12] !! [15:20:30] (03PS1) 10Hashar: dockerfiles: update Quibble to 1.6.0 [integration/config] - 10https://gerrit.wikimedia.org/r/982814 (https://phabricator.wikimedia.org/T345093) [15:22:43] hashar thanks for the cleanup and sorry for the trouble [15:23:08] I'll see if dch is available for Mac...I don't see it in macports ;( [15:25:34] If anyone is available to run fab , I'll be even further in your karmic debt! [15:25:44] ^^ against dev-images that is [15:27:18] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10phan-taint-check-plugin, 10Patch-For-Review, and 3 others: phan-taint-check gitlab ci security template should use its own image - https://phabricator.wikimedia.org/T352820 (10CodeReviewBot) mmartorana merged https://gitlab.wikimedia.org/repos/secu... [15:28:44] 10GitLab, 10Wikidata, 10wmde-wikidata-tech, 10Wikidata Dev Team (Quality Tools "Sprint"): [QB] Investigate moving QB from Gerrit to GitLab - https://phabricator.wikimedia.org/T350705 (10ItamarWMDE) [15:34:53] (03PS1) 10Hashar: jjb: switch jobs to Quibble 1.6.0 [integration/config] - 10https://gerrit.wikimedia.org/r/982817 [15:35:00] (03CR) 10Hashar: [C: 03+2] dockerfiles: update Quibble to 1.6.0 [integration/config] - 10https://gerrit.wikimedia.org/r/982814 (https://phabricator.wikimedia.org/T345093) (owner: 10Hashar) [15:35:52] inflatador: my rule of thumb is that if you have a Mac, you are doomed for open source development :) [15:36:15] hashar My thumbs are double-jointed, thanks very much ;P [15:36:52] I am serious, I am not even sure how as an SRE you can do your work with a macbook :D [15:37:24] anyway, running `dch` would be achievable with a docker image (which leads to all the question about running a docker image being so broken on a mac specially with the ARM chips) [15:37:29] https://guide.macports.org/index.html [15:37:36] else dch is a perl script [15:37:41] so maybe that can be run "as is" [15:37:52] (03Merged) 10jenkins-bot: dockerfiles: update Quibble to 1.6.0 [integration/config] - 10https://gerrit.wikimedia.org/r/982814 (https://phabricator.wikimedia.org/T345093) (owner: 10Hashar) [15:40:12] If I really have to do something x86-centric, I just use a VM [15:40:59] https://mac.getutm.app/ is pretty good [15:43:49] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Write tutorials for common CI tasks - https://phabricator.wikimedia.org/T352274 (10TBurmeister) Just a drive-by note, I noticed there's a category on mw.org for this! https://www.mediawiki.org/wiki/Category:Continuous_integration_tutorials [15:48:38] 10Release-Engineering-Team (Radar), 10Anti-Harassment, 10Security-Team, 10iPoid-Service, 10Patch-For-Review: Use Gitlab Security Pipeline for ipoid - https://phabricator.wikimedia.org/T338238 (10CodeReviewBot) mmartorana opened https://gitlab.wikimedia.org/repos/mediawiki/services/ipoid/-/merge_requests/... [15:52:17] 10Release-Engineering-Team (Radar), 10Anti-Harassment, 10Security-Team, 10iPoid-Service, 10Patch-For-Review: Use Gitlab Security Pipeline for ipoid - https://phabricator.wikimedia.org/T338238 (10mmartorana) Hi @kostajh - I have created [[ https://gitlab.wikimedia.org/repos/mediawiki/services/ipoid/-/merg... [15:58:28] (03PS2) 10Hashar: jjb: switch jobs to Quibble 1.6.0 [integration/config] - 10https://gerrit.wikimedia.org/r/982817 [15:58:32] (03PS1) 10Hashar: dockerfiles: upgrade setuptools for Quibble 1.6.0 [integration/config] - 10https://gerrit.wikimedia.org/r/982848 [15:58:40] (03CR) 10Hashar: [C: 03+2] dockerfiles: upgrade setuptools for Quibble 1.6.0 [integration/config] - 10https://gerrit.wikimedia.org/r/982848 (owner: 10Hashar) [15:59:10] 10GitLab, 10Wikidata, 10wmde-wikidata-tech, 10Wikidata Dev Team (Quality Tools "Sprint"): [QB] Investigate moving Query Builder from Gerrit to GitLab - https://phabricator.wikimedia.org/T350705 (10ItamarWMDE) [15:59:59] (03Merged) 10jenkins-bot: dockerfiles: upgrade setuptools for Quibble 1.6.0 [integration/config] - 10https://gerrit.wikimedia.org/r/982848 (owner: 10Hashar) [16:12:21] 10MediaWiki-Releasing, 10MW-1.41-release: Write release announcement for 1.41.0-rc.0 - https://phabricator.wikimedia.org/T346928 (10Jdforrester-WMF) [16:25:34] 10GitLab (Integrations), 10Striker, 10User-bd808: GitLab users with only provider=cas3 identies are not found when Striker attempts to create GitLab repostories - https://phabricator.wikimedia.org/T353176 (10bd808) >>! In T353176#9401632, @Hawkeye7 wrote: > @bd808 I can confirm that it is working now. Thanks... [16:29:39] 10Release-Engineering-Team, 10Fundraising-Backlog, 10MediaWiki-extensions-FundraiserLandingPage, 10donate.wikimedia.org, and 2 others: donate.wikimedia.org landing pages don't have descriptions in search engine results - https://phabricator.wikimedia.org/T254808 (10Pcoombe) Still no change [16:36:54] I will do the Quibble 1.6.0 upgrade later tonight [16:37:34] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10phan-taint-check-plugin, 10Patch-For-Review, and 3 others: phan-taint-check gitlab ci security template should use its own image - https://phabricator.wikimedia.org/T352820 (10CodeReviewBot) sbassett opened https://gitlab.wikimedia.org/repos/securi... [16:42:51] 10Release-Engineering-Team, 10MediaWiki-extensions-StopForumSpam, 10ci-test-error: api.php?format=json (FormatJson.php) Allowed memory size fatal error in quibble-vendor-mysql-php74-selenium-docker - https://phabricator.wikimedia.org/T353001 (10sbassett) 05Resolvedβ†’03Open [17:01:43] 10Release-Engineering-Team, 10MediaWiki-extensions-StopForumSpam, 10Patch-For-Review, 10ci-test-error: api.php?format=json (FormatJson.php) Allowed memory size fatal error in quibble-vendor-mysql-php74-selenium-docker - https://phabricator.wikimedia.org/T353001 (10sbassett) Er, wait we already [[ https://g... [17:02:49] 10MediaWiki-Releasing, 10MW-1.41-notes, 10MW-1.41-release, 10Release: Release MW 1.41.0-rc.0 - https://phabricator.wikimedia.org/T346927 (10Jdforrester-WMF) [17:02:51] 10MediaWiki-Releasing, 10MW-1.41-release: Write release announcement for 1.41.0-rc.0 - https://phabricator.wikimedia.org/T346928 (10Jdforrester-WMF) 05Openβ†’03Resolved a:03Jdforrester-WMF [17:02:57] 10MediaWiki-Releasing, 10MW-1.41-release, 10Release: Release MediaWiki 1.41.0 - https://phabricator.wikimedia.org/T346919 (10Jdforrester-WMF) [17:02:59] 10MediaWiki-Releasing, 10MW-1.41-notes, 10MW-1.41-release, 10Release: Release MW 1.41.0-rc.0 - https://phabricator.wikimedia.org/T346927 (10Jdforrester-WMF) 05Openβ†’03Resolved a:03Jdforrester-WMF [17:42:07] 10Release-Engineering-Team (Priority Backlog πŸ“₯): Experiment with WMCS as a k8s provider for gitlab-cloud-runner cluster - https://phabricator.wikimedia.org/T353356 (10dduvall) [17:43:07] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10cloud-services-team: Experiment with WMCS as a k8s provider for gitlab-cloud-runner cluster - https://phabricator.wikimedia.org/T353356 (10dduvall) [19:41:57] 10GitLab, 10Release-Engineering-Team: `npm ci` not working on trusted runners for https://gitlab.wikimedia.org/repos/data-engineering/superset - https://phabricator.wikimedia.org/T353197 (10dancy) @BTullis Looks like things are working in https://gitlab.wikimedia.org/repos/data-engineering/superset/-/jobs/1754... [19:43:20] seriously [19:43:21] https://gitlab.wikimedia.org/toolforge-repos/jouncebot/-/merge_requests/new?merge_request%5Bsource_branch%5D=humor-optout [19:43:30] that is what gitlab gives me after pushing [19:43:37] AND THAT IS A 404 NOT FOUND [19:44:09] β–ˆ β–ˆ β–ˆ β–ˆ T H A T [19:46:34] I give up [19:47:23] the web UI does tell me I pushed to a humor-optout branch but there is no where to turn that in a merge request :(\ [19:48:11] (03CR) 10Hashar: [C: 03+2] jjb: switch jobs to Quibble 1.6.0 [integration/config] - 10https://gerrit.wikimedia.org/r/982817 (owner: 10Hashar) [19:48:11] are you logged in? [19:48:26] !log Updating jobs to use Quibble 1.6.0 | https://gerrit.wikimedia.org/r/c/integration/config/+/982817 [19:48:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:48:34] taavi: yes [19:48:51] I think I will just `/ignore jouncebot` [19:48:56] and avoid futher touching Gitlab [19:49:10] huh, that link works for me [19:49:17] :-] [19:49:40] maybe it doesn't like that you used your super admin powers to push that branch directly [19:49:53] I know Gitlab gives a 404 when you are not logged in (which I always the case for me) [19:50:01] oh [19:50:08] (03Merged) 10jenkins-bot: jjb: switch jobs to Quibble 1.6.0 [integration/config] - 10https://gerrit.wikimedia.org/r/982817 (owner: 10Hashar) [19:50:28] so as an admin I can push to any repo/branch ? [19:50:40] otherwhise I guess it would have asked me to fork the repo maybe? [19:50:54] apparently, you're not listed on https://gitlab.wikimedia.org/toolforge-repos/jouncebot/-/project_members [19:51:48] * hashar forks / push to fork and see what it does [19:53:36] (β•―Β°β–‘Β°οΌ‰β•―οΈ΅ ┻━┻ [19:53:46] taavi: THAT DID IT! [19:55:27] πŸŽ‰ [20:00:29] taavi: thanks for finding out that admin can push anywhere. I guess I should strip my admin rights ;) [20:00:57] with great power comes great confusion [20:01:49] !log Deleted disabled Jenkins job still using Quibble 1.5.5 and not in JJB: https://integration.wikimedia.org/ci/job/quibble-daily-EntitySchema-vendor-mysql-php74-docker/ [20:01:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:17:12] 10Release-Engineering-Team, 10Traffic, 10collaboration-services, 10Patch-For-Review: CI on gitlab for eBPF / networking heavy projects - https://phabricator.wikimedia.org/T353279 (10Vgutierrez) Docker is definitely not a valid option here since we need to test against several kernels (at least 5.10 and 6.1) [20:29:09] so looks like I can comment `+1` in Gitlab to get a merge request merged by CI [20:29:11] that is nice [20:29:19] https://gitlab.wikimedia.org/repos/releng/gerrit-gitlab-archiver/-/merge_requests/1?commit_id=d1495df139521dfd13fcc03a10a32ecce8757937&view=parallel#note_63656 [20:30:00] jokes aside, I am off! [20:30:58] hey folks, I want to push https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/955015 to enable the feature on mwdebug. I have very little experience deploying. Anyone willing to shadow me to make sure I don't mess it up? [20:31:55] cwhite: have you filed form X34-bis? [20:33:07] cwhite: the easiest would be to add your change to the next backport & config dpleoyment window which starts in half an hour. [20:33:08] https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20231213T2100 [20:33:37] then rebase your change (Gerrit marks it as being in conflict, most probably cause CommonSettings.php got edited meanwhile) [20:34:08] and people doing the deployment window would sync up in #wikimedia-operations [20:34:20] right on, that is easy :) [20:34:30] AS LONG AS YOU FILE FORM X34-BIS [20:34:40] * hashar giggles at bureaucracy [20:34:45] but yeah that is easy [20:34:53] :D [20:34:55] one can see that backport & config window as deployment as a service [20:35:10] merely like calling 1-800-AWS-DEPLOY [20:35:32] operator: "hello how can I help you break your web *cough* deploy today?" [20:37:50] I can't afford AWS, now what? [20:37:58] D: [20:38:05] ;D [20:38:25] I am off to bed! [20:38:37] * cwhite waves [20:55:26] 10Project-Admins: Create project tag for Adiutor - https://phabricator.wikimedia.org/T353380 (10Vikipolimer) [21:03:19] 10Continuous-Integration-Config, 10PHP 8.3 support: Make PHP 8.3 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) - https://phabricator.wikimedia.org/T353362 (10Jdforrester-WMF) [21:06:28] 10Continuous-Integration-Config, 10PHP 8.2 support: Make PHP 8.2 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) - https://phabricator.wikimedia.org/T352085 (10Jdforrester-WMF) [21:06:36] 10Continuous-Integration-Config, 10PHP 8.3 support: Make PHP 8.3 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) - https://phabricator.wikimedia.org/T353362 (10Jdforrester-WMF) [21:06:38] (03PS1) 10Jforrester: Zuul: [mediawiki/core] Make PHP 8.2 and 8.3 voting for master [integration/config] - 10https://gerrit.wikimedia.org/r/982918 (https://phabricator.wikimedia.org/T352085) [21:08:09] (03CR) 10CI reject: [V: 04-1] Zuul: [mediawiki/core] Make PHP 8.2 and 8.3 voting for master [integration/config] - 10https://gerrit.wikimedia.org/r/982918 (https://phabricator.wikimedia.org/T352085) (owner: 10Jforrester) [21:09:13] (03PS2) 10Jforrester: Zuul: [mediawiki/core] Make PHP 8.2 and 8.3 voting for master [integration/config] - 10https://gerrit.wikimedia.org/r/982918 (https://phabricator.wikimedia.org/T352085) [21:11:12] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/core] Make PHP 8.2 and 8.3 voting for master [integration/config] - 10https://gerrit.wikimedia.org/r/982918 (https://phabricator.wikimedia.org/T352085) (owner: 10Jforrester) [21:12:06] ooh [21:12:16] Core-only for now. [21:12:25] But finally we're living in… 2022. [21:12:50] Huge thanks to Umherirrender for making so many fixes. [21:14:39] (03Merged) 10jenkins-bot: Zuul: [mediawiki/core] Make PHP 8.2 and 8.3 voting for master [integration/config] - 10https://gerrit.wikimedia.org/r/982918 (https://phabricator.wikimedia.org/T352085) (owner: 10Jforrester) [21:15:42] with all those PHP versions running, I guess we migth want to add some more Jenkins instances [21:16:09] hashar: In a few days we'll kill the PHP 7.3 jobs. [21:16:20] \o/ [21:16:22] But that only saves a few job runs a week (REL1_35-only). [21:16:41] I forgot to ask WMCS whether we had capacity for more [21:16:48] :-( [21:17:15] and there was a suggestion to use disk less instances to potentially speeds up disk io [21:17:26] ie have everything hold in tmpfs which means in RAM [21:17:31] We can move the phan code to run inside the `composertest` jobs? That's an easy-ish win? [21:17:41] Right. That'd be cool if it's easier. [21:17:47] then [21:17:53] I think I will move the build to java next year [21:17:55] * hashar grins [21:18:08] (I'm intentionally not deploying Zuul config until 982901 lands.) [21:18:15] Oh no. [21:18:20] * James_F hides from hashar . [21:18:23] java is good! [21:19:00] I think adding more instances is certainly doable, it might be just about asking [21:19:14] and possibly maybe buncho f tests can be marked @group Standalone [21:19:26] I also have some task to move Wikibase selenium tests out of the main job [21:19:44] anyway that is lot of things [21:19:59] Or just stop running the selenium tests at all. ;-) [21:20:00] (03CR) 10Hashar: [C: 03+2] zuul: Add codehealth template to MediaModeration [integration/config] - 10https://gerrit.wikimedia.org/r/982812 (owner: 10Kosta Harlan) [21:20:49] (03PS2) 10Hashar: zuul: Add codehealth template to MediaModeration [integration/config] - 10https://gerrit.wikimedia.org/r/982812 (https://phabricator.wikimedia.org/T321837) (owner: 10Kosta Harlan) [21:20:50] !log Zuul: [mediawiki/core] Make PHP 8.2 and 8.3 voting for master for T352085 and T353362 [21:20:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:20:55] T352085: Make PHP 8.2 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) - https://phabricator.wikimedia.org/T352085 [21:20:55] T353362: Make PHP 8.3 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) - https://phabricator.wikimedia.org/T353362 [21:21:03] 10Continuous-Integration-Config, 10PHP 8.3 support, 10Patch-For-Review: Make PHP 8.3 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) - https://phabricator.wikimedia.org/T353362 (10Novem_Linguae) [21:21:09] (03CR) 10Hashar: [C: 03+2] zuul: Add codehealth template to MediaModeration [integration/config] - 10https://gerrit.wikimedia.org/r/982812 (https://phabricator.wikimedia.org/T321837) (owner: 10Kosta Harlan) [21:22:20] (03Merged) 10jenkins-bot: zuul: Add codehealth template to MediaModeration [integration/config] - 10https://gerrit.wikimedia.org/r/982812 (https://phabricator.wikimedia.org/T321837) (owner: 10Kosta Harlan) [21:23:02] !log Reload Zuul for Add codehealth template to MediaModeration # T321837 [21:23:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:23:05] T321837: Repositories integrated into Codehealth Pipeline (Production) - https://phabricator.wikimedia.org/T321837 [21:24:07] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for ssastry - https://phabricator.wikimedia.org/T353383 (10ssastry) [21:25:16] hashar: if you're not about to go to bed, there's no deployers and a lot of patches [21:25:26] (I know it's like 10:30pm for you, sorry) [21:26:15] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for ssastry - https://phabricator.wikimedia.org/T353383 (10ssastry) [21:26:38] RhinosF1: yeah it is too late :-\ [21:27:02] hashar: thought so, know if there's any other relengers around? [21:27:06] And sleep well :) [21:27:24] brennen: dduvall: dduvall: jeena: may one of you run the backport window ? https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20231213T2100 [21:28:12] or I guess they should level up [21:28:16] and be able to deploy [21:28:23] anyway I am off for real. [21:29:26] yes I can [21:29:55] thanks jeena [21:29:59] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for ssastry - https://phabricator.wikimedia.org/T353383 (10ssastry) [21:31:00] thcipriani: I'm noticing these later windows seem to be having less people show despite the number listed [21:31:26] yep, that's a pattern [21:31:52] they're best effort and the number of deployers is dwindling it seems [21:34:27] thcipriani: might be worth asking those who don't deploy ever to take themselves off [21:34:47] I'm not sure I've seen RoanKattouw do a window in months [21:40:30] that's true, not everyone listed on the deployers list is available for all windows, sometimes most windows, that's always been self-managed: if you have availability, add yourself, if not then don't [21:41:17] hence the claiming the window as a deployer at the start, to make it clear who's running the window [21:43:35] I think the backport training was my original solution to the dwindling deployers problem, but that hasn't been a glowing success. I have some ideas why that is, so iterating from there will be the next step. [21:45:22] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for ssastry - https://phabricator.wikimedia.org/T353383 (10thcipriani) p:05Triageβ†’03Medium Howdy @ssastry β€”I sent you an invite to a hangout during the backport window tomorrow, hope that still works for you. See you then! [21:45:52] eliminating the need for deployers under many circumstances would be the ideal outcome here, I think. [21:54:54] 10Continuous-Integration-Config, 10PHP 8.3 support: Make PHP 8.3 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) - https://phabricator.wikimedia.org/T353362 (10Novem_Linguae) [21:56:14] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for ssastry - https://phabricator.wikimedia.org/T353383 (10ssastry) [21:59:30] thcipriani: I'm intrigued and scared by the idea of eliminating the need for deployers [21:59:35] How does that look? [21:59:45] 10Continuous-Integration-Config, 10ci-test-error: apitests do not run across repositories - https://phabricator.wikimedia.org/T352887 (10Jdforrester-WMF) The api-tests were very intentionally only applied to core and not extensions in 26fb84df00 due to the cost of running extra tests for all code and the MW te... [22:00:35] (03CR) 10Jforrester: "check experimental" [integration/docroot] - 10https://gerrit.wikimedia.org/r/979358 (owner: 10Hashar) [22:04:06] RhinosF1: we are not 100% sure, but one thing to consider is a web UI that does what running `scap backport` does when someone fills out a form and hits a button. [22:04:46] bd808: sounds fancy but how open should deploying to prod be? [22:05:06] we can still have authentication and authorization [22:05:12] Oh yes [22:05:14] RhinosF1: lots of unknowns there (it's a nascent idea), but deployers in 80% of cases are doing very little since scap backport came into being. So if we were able to provide a similar interface to the one we provide in -operations during deployment (e.g., build a queue of people who are around, person-by-person to mwdebug, person-by-person live) that's pretty much the whole idea. [22:05:16] Nobody's working on it, to be clear, still in idea phase, but iit might solve some problems. [22:05:30] But are we talking a group same size as deployers? Or wider? [22:05:51] that depends on how confident we become in the automation [22:06:05] Scap backport is already a fairly low curve to learn. It's the debugging and being prepared to deal with when stuff blows up that's more important now [22:06:18] why not everyone with +2 in mediawiki/core.git? It's worth thinking about. [22:06:28] It's worth thinking about, yes [22:06:38] The web UI is interesting [22:06:45] I like the idea [22:06:59] I just wonder how prepared people will be for when things don't work [22:07:24] When things work perfectly, a web UI around scap backport is a fairly simple concept [22:07:54] Last week at the Developer Experience offsite I asked a "what scares us" question about something similar to this to start thinking about what technical and social barriers we can see today [22:08:49] there are certainly a number of known issues, likely more currently unknown issues, and then social concerns to think though [22:10:20] but I think we are ready to start asking about these next level concepts so we can start moving towards full continuous integration. We may certainly decide not to go all the way to every patch getting live deployed to enwiki, but we should have the option I think. [22:10:48] As a concept, I like it [22:13:14] I'm more worried about it becoming so easy that people are complacent and when things do go wrong, (which let's assume they do and at some point in the future all wikis go offline because of a bad deployment for some reason), are people going to be familiar and have the access to debug and fix that at pace [22:13:20] And communicate etc [22:14:16] I already think with some of our newer deployers that there's a reluctance to be brave and say no or stop when unsure [22:15:05] I can think of a few examples where the confidence to deploy a patch has been lower than it should have been or it's taken time to call deploying off [22:24:49] RhinosF1: it is a reasonable concern to voice, but I also think it will be something that we can avoid. We need to improve many things to get to a place where continuous delivery is possible. Some of that work will be in monitoring and alerting. Other parts will be in making rollback safe and simple. [22:25:07] we will build better guardrails at the same time we build faster cars :) [22:25:28] I like your last sentence [22:26:02] I like the concept and the idea of continuous delivery [22:26:22] I'm sure they'll be a lot to work out as the move towards it happens [22:26:43] 100% Stay tuned for future discussions :) [22:27:00] I will [22:28:24] And if you have spare time, think about what scares you about the idea or the possible implementation. there will come a time, hopefully not too far away, where that is a question we ask of a lot of folks. [22:28:55] Will make some coherent thoughts [22:43:08] RhinosF1: It's important to provide easy rollback and stop deployment methods, but otherwise I think many of the current deployers (releng) have less visibility into what could go wrong and how to fix it than the people who wrote the code, so it could actually be better for the developer to be able to deploy themself, and if they are uncertain about their work, then maybe it's not ready to be deployed. [22:54:08] ^ [22:55:16] acknowledging the trick parts, i think i view broader access to deployment as potentially a way to wind up with better-distributed knowledge of some of the realities of production. [22:59:34] 'tricky parts, that is [23:02:17] I think that jeena's point is well known within the Release Engineering team, but probably a surprise to a lot of folks outside the team. We can trace the detachment of RelEng from being a peer team of the MediaWiki team back to the 2015 reorg of doom. [23:18:08] (03PS1) 10Jforrester: Zuul: Migrate repos from 74-81 to 75-83 composer if passing [integration/config] - 10https://gerrit.wikimedia.org/r/982934 [23:25:26] (03PS1) 10Jforrester: Zuul: [mediawiki/libs/less.php] Make PHP 8.3 voting [integration/config] - 10https://gerrit.wikimedia.org/r/982935 [23:25:35] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/libs/less.php] Make PHP 8.3 voting [integration/config] - 10https://gerrit.wikimedia.org/r/982935 (owner: 10Jforrester) [23:25:42] (03CR) 10Jforrester: [C: 03+2] Zuul: Migrate repos from 74-81 to 75-83 composer if passing [integration/config] - 10https://gerrit.wikimedia.org/r/982934 (owner: 10Jforrester) [23:27:03] (03Merged) 10jenkins-bot: Zuul: Migrate repos from 74-81 to 75-83 composer if passing [integration/config] - 10https://gerrit.wikimedia.org/r/982934 (owner: 10Jforrester) [23:27:22] !log Zuul: Migrate repos from 74-81 to 75-83 composer if passing [23:27:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:27:27] (03Merged) 10jenkins-bot: Zuul: [mediawiki/libs/less.php] Make PHP 8.3 voting [integration/config] - 10https://gerrit.wikimedia.org/r/982935 (owner: 10Jforrester) [23:27:32] !log Zuul: [mediawiki/libs/less.php] Make PHP 8.3 voting [23:27:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL