[00:02:47] 10GitLab (Project Migration), 10Striker, 10Tools, 10Goal, 10User-bd808: Replace Diffusion integration with Gitlab integration in Striker (toolsadmin) - https://phabricator.wikimedia.org/T296893 (10bd808) [00:02:53] 10GitLab (Project Migration), 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Epic, and 2 others: Migrate active repositories in Phabricator Differential to GitLab - https://phabricator.wikimedia.org/T191182 (10bd808) [00:03:50] 10GitLab (Project Migration), 10Striker, 10Tools, 10Developer-Advocacy (Jul-Sep 2022), and 2 others: Replace Diffusion integration with Gitlab integration in Striker (toolsadmin) - https://phabricator.wikimedia.org/T296893 (10bd808) [00:29:31] 10Beta-Cluster-Infrastructure: deployment-ms-fe03 puppet failure - https://phabricator.wikimedia.org/T316845 (10Zabe) [00:38:12] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Data-Persistence, 10serviceops, 10serviceops-collab: sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Dzahn) >>! In T315713#8190804, @Marostegui wrote: > Is there a way in puppet that any new host... [00:38:31] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Data-Persistence, 10serviceops, 10serviceops-collab: sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Dzahn) a:03Dzahn [00:38:49] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10serviceops, 10serviceops-collab: sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Dzahn) [00:39:24] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10serviceops, 10serviceops-collab: sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Dzahn) removing data-persistence again. You did answer the question and we know what to do. Thanks! [01:36:08] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10PHP 8.0 support, 10ci-test-error: CI job mediawiki-quibble-composer-mysql-php80-docker on mediawiki/core gate-and-submit is flaky failing with Segmentation fault - https://phabricator.wikimedia.org/T315167 (10tstarling) > `#0 0x000055b6dd6... [01:49:11] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10ci-test-error: CI job mediawiki-quibble-composer-mysql-php80-docker on mediawiki/core gate-and-submit is flaky failing with Segmentation fault - https://phabricator.wikimedia.org/T315167 (10tstarling) The issue was introduced in PHP 5.3, it i... [01:57:57] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: deployment-ms-fe03 puppet failure - https://phabricator.wikimedia.org/T316845 (10Zabe) @MatthewVernon Hey, I tried migrating deployment-prep to https://gerrit.wikimedia.org/r/c/operations/puppet/+/769941, but I failed. Could you help me out with my patch? [07:40:57] (03CR) 10Hashar: [C: 03+2] jjb: Switch quibble jobs to 1.4.6 [integration/config] - 10https://gerrit.wikimedia.org/r/828611 (https://phabricator.wikimedia.org/T305525) (owner: 10Jforrester) [07:43:16] (03Merged) 10jenkins-bot: jjb: Switch quibble jobs to 1.4.6 [integration/config] - 10https://gerrit.wikimedia.org/r/828611 (https://phabricator.wikimedia.org/T305525) (owner: 10Jforrester) [07:43:51] !log Updating Jenkins jobs for Quibble 1.4.5 > 1.4.6 + php 7.4 update | T305525 | T314586 | T316601 | https://gerrit.wikimedia.org/r/c/integration/config/+/828611 [07:43:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:43:56] T314586: Enable Wikibase CI wikis to link to themselves - https://phabricator.wikimedia.org/T314586 [07:43:56] T316601: PHP Warning: Erroneous data format for unserializing 'Wikimedia\Rdbms\MySQLPrimaryPos' - https://phabricator.wikimedia.org/T316601 [07:43:56] T305525: Experiment with using pnpm in CI to speed up npm install process - https://phabricator.wikimedia.org/T305525 [08:05:15] hi qchris ;) [08:05:30] Hi hashar :-) [08:38:10] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Upstream: Jobs started failing on https://releases-jenkins.wikimedia.org on 2022-08-21 - https://phabricator.wikimedia.org/T315897 (10hashar) [08:43:33] (03CR) 10Jaime Nuche: [C: 04-1] "Most of the changes are good, but there's a couple of issues related to permissions and what lock file is being used." [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [09:26:44] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641 (10Zabe) [09:36:11] !log shutoff deployment-urldownloader02 [09:36:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:47:44] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Upstream: Jobs started failing on https://releases-jenkins.wikimedia.org on 2022-08-21 - https://phabricator.wikimedia.org/T315897 (10hashar) I have amended the task description with a repro case which can be run via https://... [09:56:29] AH I have found a fix for TheresNoTime account breaking the branch cut job! [09:56:31] https://phabricator.wikimedia.org/T315897#8205451 !! [09:57:05] I am pretty sure it is a bug in Jenkins core itself only catching UsernameNotFoundException authentication exception [09:57:19] ignoring other causes such as DisabledException [09:57:45] I wrote a hotfix for the git plugin, I will look at writing a test covering the fix if I ever understand how to use mockito [10:01:10] * TheresNoTime can break things by just existing [10:19:52] TheresNoTime: to be fair it is almost certainly an issue with Jenkins issue which only handles unknown user exceptions [10:20:01] my half written code https://github.com/jenkinsci/git-plugin/commit/86edb439f0b3b0cd65197308870410ece66ea2c0 :D [10:28:15] 10Phabricator: Merge multiple Phab accounts made in error into single ITS account - https://phabricator.wikimedia.org/T316721 (10Aklapper) 05Open→03Resolved Account @Vpoundstone disabled. Regarding shortcuts, there aren't many, according to the upstream docs: https://secure.phabricator.com/book/phabricator/... [11:25:43] hmm [11:26:04] I managed to learn the basis of Mockito and write a test in less than an hour https://github.com/hashar/git-plugin/commit/bd0d9cd588d7ba493a0bdc6ad49c1c05e1d997de#diff-afcdb22a338c19846f11fec2341467f449a0e64b5244165f5926bd30fd033edc ! [11:47:21] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Upstream: Jobs started failing on https://releases-jenkins.wikimedia.org on 2022-08-21 - https://phabricator.wikimedia.org/T315897 (10hashar) #upstream pull request https://github.com/jenkinsci/git-plugin/pull/1322 I have co... [11:47:55] hmm done [11:47:57] lunch time! [11:48:21] jnuche: I have learned about mockito today :] https://github.com/hashar/git-plugin/commit/bd0d9cd588d7ba493a0bdc6ad49c1c05e1d997de#diff-afcdb22a338c19846f11fec2341467f449a0e64b5244165f5926bd30fd033edc [11:50:11] mockito, ha, that takes me back :) [11:50:49] also, don't use that name when in polite company among Spaniards... [11:54:53] is that a slang/offensive word in spanish? :-\ [11:57:12] it's not slang, but it sounds exactly the same as the diminutive for the word for "snot" [11:57:54] I will try to remember about that [11:58:03] (of course you can still say it around Spanish speakers, I was just joking :D) [12:48:43] 10Release-Engineering-Team, 10Infrastructure-Foundations, 10SRE-tools: Investigate sharing releng common python code to pywmflib - https://phabricator.wikimedia.org/T316757 (10Volans) @hashar thanks for the comprehensive summary of our IRC chat. I don't mind either direction, create a separate library for y... [12:59:30] (03CR) 10Jaime Nuche: [C: 03+2] Initial mw container deployment stuff [tools/train-dev] - 10https://gerrit.wikimedia.org/r/789669 (https://phabricator.wikimedia.org/T299648) (owner: 10Ahmon Dancy) [13:00:24] (03Merged) 10jenkins-bot: Initial mw container deployment stuff [tools/train-dev] - 10https://gerrit.wikimedia.org/r/789669 (https://phabricator.wikimedia.org/T299648) (owner: 10Ahmon Dancy) [13:10:41] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10ci-test-error: CI job mediawiki-quibble-composer-mysql-php80-docker on mediawiki/core gate-and-submit is flaky failing with Segmentation fault - https://phabricator.wikimedia.org/T315167 (10hashar) I find it quite amazing you had a patch read... [13:14:14] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10ci-test-error, 10php-segfault: CI job mediawiki-quibble-composer-mysql-php80-docker on mediawiki/core gate-and-submit is flaky failing with Segmentation fault - https://phabricator.wikimedia.org/T315167 (10hashar) >>! In T315167#8204954, @t... [13:14:41] 10Continuous-Integration-Infrastructure, 10RelEng-Archive-FY201718-Q1, 10MediaWiki-Core-Tests, 10MediaWiki-extensions-WikibaseClient, and 5 others: Job mediawiki-extensions-php55 frequently fails due to "Segmentation fault" - https://phabricator.wikimedia.org/T142158 (10hashar) [13:14:49] 10Continuous-Integration-Infrastructure, 10Quibble, 10Patch-For-Review, 10php-segfault: MediaWiki phpunit tests SIGSEGV under php5.5 on Jessie - https://phabricator.wikimedia.org/T192432 (10hashar) [13:34:50] 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Scap, 10SRE, 10Python3-Porting: git-fat replacement/removal - https://phabricator.wikimedia.org/T279509 (10demon) [15:18:52] (03PS1) 10Mglaser: Add dependency for BlueSpiceCountThings [integration/config] - 10https://gerrit.wikimedia.org/r/829027 [15:34:55] 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Scap, 10Wikidata-Query-Service: wdqs: replace git-fat with git-lfs - https://phabricator.wikimedia.org/T316876 (10demon) [15:35:19] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Upstream: Jobs started failing on https://releases-jenkins.wikimedia.org on 2022-08-21 - https://phabricator.wikimedia.org/T315897 (10dancy) I installed https://people.wikimedia.org/~hashar/T315897/git-4.11.5-96-g13a61a06-PR1... [15:40:10] 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Scap, 10Discovery-Search: wikimedia/discovery/analytics: replace git-fat with git-lfs - https://phabricator.wikimedia.org/T316877 (10demon) [15:41:51] 10Gerrit, 10Release-Engineering-Team (Bonus Level 🕹ī¸): gerrit: stop using git-fat - https://phabricator.wikimedia.org/T316878 (10demon) [15:52:34] 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Scap, 10SRE, 10Python3-Porting: git-fat replacement/removal - https://phabricator.wikimedia.org/T279509 (10EBernhardson) additional affected projects: * search/MjoLniR/deploy * search/airflow [16:07:48] 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Scap, 10Discovery-Search: wikimedia/discovery/analytics: replace git-fat with git-lfs - https://phabricator.wikimedia.org/T316877 (10EBernhardson) I can't explain why the find command above doesn't find them, but the above list is missing a number of things: `... [16:16:58] 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Scap, 10Discovery-Search: wikimedia/discovery/analytics: replace git-fat with git-lfs - https://phabricator.wikimedia.org/T316877 (10EBernhardson) `find . -name '*.whl' -or -name '*.jar' -ls -exec cat {} \;` this should instead be (i didn't expect this either):... [16:20:58] 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Scap, 10Discovery-Search: wikimedia/discovery/analytics: replace git-fat with git-lfs - https://phabricator.wikimedia.org/T316877 (10EBernhardson) [18:23:47] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Upstream, 10ci-test-error, 10php-segfault: CI job mediawiki-quibble-composer-mysql-php80-docker on mediawiki/core gate-and-submit is flaky failing with Segmentation fault - https://phabricator.wikimedia.org/T315167 (10Umherirrender) The c... [18:35:45] (03PS5) 10Jeena Huneidi: Replace Lock with TimeoutLock [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) [18:40:22] (03CR) 10Jeena Huneidi: "Thanks for the reviews!" [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [18:42:35] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Patch-For-Review: Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641 (10Zabe) [19:04:07] 10GitLab (Project Migration), 10Release-Engineering-Team, 10Fundraising Tech - Chaos Crew: Create new GitLab project group: Fundraising-Tech - https://phabricator.wikimedia.org/T316695 (10Dzahn) I created https://gitlab.wikimedia.org/repos/fundraising-tech as a new subgroup under the /repos/ top-group. So... [19:08:57] 10GitLab (Project Migration), 10Release-Engineering-Team, 10Fundraising Tech - Chaos Crew: Create new GitLab project group: Fundraising-Tech - https://phabricator.wikimedia.org/T316695 (10Dzahn) You can compare group members from the Gerrit link to the new link here: https://gitlab.wikimedia.org/groups/repo... [19:12:21] 10GitLab (Project Migration), 10Release-Engineering-Team, 10Fundraising Tech - Chaos Crew, 10serviceops, 10serviceops-collab: Create new GitLab project group: Fundraising-Tech - https://phabricator.wikimedia.org/T316695 (10Dzahn) 05Open→03In progress a:03Dzahn [19:12:42] 10GitLab (Project Migration), 10Release-Engineering-Team, 10Fundraising Tech - Chaos Crew, 10serviceops, 10serviceops-collab: Create new GitLab project group: Fundraising-Tech - https://phabricator.wikimedia.org/T316695 (10Dzahn) [19:17:25] 10GitLab (Project Migration), 10Release-Engineering-Team, 10Fundraising Tech - Chaos Crew, 10serviceops, 10serviceops-collab: Create new GitLab project group: Fundraising-Tech - https://phabricator.wikimedia.org/T316695 (10jgleeson) thanks @Dzahn ! [19:21:19] 10GitLab (Project Migration), 10Release-Engineering-Team, 10Fundraising Tech - Chaos Crew, 10serviceops, 10serviceops-collab: Create new GitLab project group: Fundraising-Tech - https://phabricator.wikimedia.org/T316695 (10Dzahn) 05In progress→03Resolved Yep. cheers. Feel free to reopen if there are... [19:42:20] 10Project-Admins: Request for project MediaWiki-extensions-SemanticQueryInterface - https://phabricator.wikimedia.org/T138603 (10Vedmaka) @Aklapper yup, sorry! absolutely, I have just shut the https://github.com/vedmaka/SemanticQueryInterface/issues down [19:59:22] 10Project-Admins: Create project tag for MediaWiki-extensions-BlockInactive - https://phabricator.wikimedia.org/T316904 (10Vedmaka) [20:07:33] 10Project-Admins: Create project tag for MediaWiki-extensions-BlockInactive - https://phabricator.wikimedia.org/T316904 (10Peachey88) 05Open→03Resolved Hi, requested public project #mediawiki-extensions-blockinactive has been created: https://phabricator.wikimedia.org/project/view/6132/ (In case you need to... [20:09:03] 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Scap, 10Discovery-Search: wikimedia/discovery/analytics: replace git-fat with git-lfs - https://phabricator.wikimedia.org/T316877 (10demon) Weird! Thanks for spotting that. [20:20:52] <^demon> ebernhardson: I found the shortened `find` command we wanted..... find . \( -name "*.jar" -o -name "*.whl" \) -print -exec cat {} \; [20:21:09] ^demon: still odd, but better :) [20:21:22] <^demon> (fun fact: the space between \( and -name is important! [20:21:40] after a few dozen years, still getting surprised by find [20:23:53] <^demon> So, since I caught you.... what's the build/deployment story around search/mjolnir, search/airflow and wikimedia/discovery/analytics? How often do we build & deploy these? Are they done by hand or is there a job? [20:24:39] ^demon: the first two are pretty stable and rarely change, the third has all the python dependencies for things we run inside the yarn cluster and changes a bit more often [20:25:42] ^demon: the python bits have a couple scripts, make_wheels.sh and upload_wheels.py go over the requirements.txt files and put all the wheels into a directory, then upload_wheels.py sync's them into archiva [20:26:04] ^demon: then on scap deploy virtualenv's get built from the requirements.txt + wheels. It should all really be replaced with a pypi mirror [20:28:14] * bd808 is happy to have moved his prod python things to containers which allowed dropping local repos full of wheels [20:28:36] i asked them to run containers in yarn, apparently the functionality exists but it's not so simple to just turn on :P [20:28:57] if i can get another container runtime that will give me a TB of memory for a few hours at a time... [20:33:06] <^demon> So, we want to move off of git-fat. But moving to git-lfs means we don't actually need archiva -- we'd just be storing the large files on the gerrit node itself [20:33:43] <^demon> (I'm open to solutions that don't involve git-fat or git-lfs or really git at all :)) [20:34:13] using archiva here is a complete hack anyways, we tell archiva that these are zip files (which they are!) and it's happy enough. We used archiva because it already existed [20:34:51] * ^demon nods [20:34:55] <^demon> I remember this all too well [20:48:40] 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Scap, 10Discovery-Search: wikimedia/discovery/analytics: replace git-fat with git-lfs - https://phabricator.wikimedia.org/T316877 (10hashar) `find` is indeed tricky from time to time, well done on figuring out the issue. I have an alternative to list files that... [20:48:50] I have seen `find` and had to seek a oneliner ;D [20:51:30] ^demon: about git lfs, I found out the Gerrit plugin has support for S3 storage https://gerrit.wikimedia.org/r/plugins/lfs/Documentation/config.md#lfs-s3-backend which me be interesting [20:51:55] saves us from having to replicate lfs files between Gerrit and its replica (maybe the replicatoin plugin handles that but I haven't checked) [20:52:04] anyway, it is a different story really [20:52:34] I am happy to see git-fat replacement has activities \o/ [20:52:39] <^demon> Yeah I already saw that. I tried to see if it would work with non-S3 object stores but I don't believe it will. [20:54:16] ^demon: OpenStack Swift has an S3 emulation layer, so maybe that can be used [20:54:40] but I guess that would be something to investigate late [20:54:41] r [20:54:47] anyway I am off to bed. Kudos! [20:57:46] <^demon> That said, we already handle replication of data between the primary and replicas so it's no biggie to do that for lfs data too. And lfs kinda sucks, so I'm exploring non-git options where we can [20:57:52] <^demon> Gnite hashar [22:03:13] 10Scap, 10MW-on-K8s, 10MediaWiki-Internationalization, 10Patch-For-Review: Investigate unexpected l10n cache rebuilds - https://phabricator.wikimedia.org/T316714 (10dancy) [22:04:56] 10Scap, 10MW-on-K8s, 10MediaWiki-Internationalization, 10Patch-For-Review: Investigate unexpected l10n cache rebuilds - https://phabricator.wikimedia.org/T316714 (10dancy) [22:06:13] 10Scap, 10MW-on-K8s, 10MediaWiki-Internationalization, 10Patch-For-Review: Investigate unexpected l10n cache rebuilds - https://phabricator.wikimedia.org/T316714 (10dancy) [22:10:50] 10Scap, 10MW-on-K8s, 10MediaWiki-Internationalization, 10Patch-For-Review: Investigate unexpected l10n cache rebuilds - https://phabricator.wikimedia.org/T316714 (10dancy) [22:15:17] 10Scap, 10MW-on-K8s, 10MediaWiki-Internationalization, 10Patch-For-Review: Investigate unexpected l10n cache rebuilds - https://phabricator.wikimedia.org/T316714 (10dancy)