[05:58:29] (03PS1) 10Hashar: Remove php 8.1 gating jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1020477 (https://phabricator.wikimedia.org/T320818) [05:58:44] 10Continuous-Integration-Config, 13Patch-For-Review, 07PHP 8.1 support: Configure CI to test the Wikimedia production gate code against PHP 8.1 as well as PHP 7.4 - https://phabricator.wikimedia.org/T320818#9721298 (10hashar) 05Resolved→03Open [06:00:42] (03CR) 10CI reject: [V:04-1] Remove php 8.1 gating jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1020477 (https://phabricator.wikimedia.org/T320818) (owner: 10Hashar) [06:02:00] (03PS2) 10Hashar: Remove php 8.1 gating jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1020477 (https://phabricator.wikimedia.org/T320818) [06:05:55] (03CR) 10Hashar: [C:03+2] Remove php 8.1 gating jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1020477 (https://phabricator.wikimedia.org/T320818) (owner: 10Hashar) [06:07:30] (03Merged) 10jenkins-bot: Remove php 8.1 gating jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1020477 (https://phabricator.wikimedia.org/T320818) (owner: 10Hashar) [06:09:21] (03CR) 10Hashar: "That did not get deployed and I have did it this morning." [integration/config] - 10https://gerrit.wikimedia.org/r/1020359 (owner: 10Jforrester) [06:10:59] (03CR) 10Hashar: [C:04-2] "We need to keep them for as long as Wikimedia is still using php 7.4!" [integration/config] - 10https://gerrit.wikimedia.org/r/1020362 (owner: 10Jforrester) [06:12:49] (03CR) 10Hashar: [C:04-2] "As long as Wikimedia primarily runs php 7.4, we should run it in `test` and give the immediate feedback to developers." [integration/config] - 10https://gerrit.wikimedia.org/r/1020363 (owner: 10Jforrester) [06:14:53] (03CR) 10Hashar: [C:03+1] "This can be cherry picked and deployed yes :) It does not affect Wikimedia production which runs neither Sqlite or Postgre." [integration/config] - 10https://gerrit.wikimedia.org/r/1020364 (owner: 10Jforrester) [06:18:49] 06Release-Engineering-Team, 06Infrastructure-Foundations, 06serviceops, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9721307 (10hashar) >>! In T362518#9719270, @Jdforrester-WMF wrote: > This has also broken building CI images. Will have to migrate them to bulls... [07:01:16] 06Release-Engineering-Team, 06Infrastructure-Foundations, 06serviceops, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9721419 (10MoritzMuehlenhoff) >>! In T362518#9719270, @Jdforrester-WMF wrote: > This has also broken building CI images. Will have to migrate th... [08:59:09] 06Project-Admins: Create project tag for New-Engagement-Experiments - https://phabricator.wikimedia.org/T362721#9721700 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #new-engagement-experiments has been created: https://phabricator.wikimedia.org/project/view/7107/ (In case you n... [09:13:23] 10Diffusion, 10Gerrit, 10Phabricator: Diffusion fails to push into Gerrit - https://phabricator.wikimedia.org/T362524#9721781 (10Aklapper) Switched the three repos from using K19 for an ssh:// URI to using K18 for an https:// URI. Let's see if that helps, if not I may just switch this off completely [10:00:31] 06Release-Engineering-Team: Run a query to pull the number of Phabricator Tasks reported by community, resolved by WMF - https://phabricator.wikimedia.org/T362406#9721854 (10Aklapper) 05Open→03Resolved I posted the DB query output (464 times: Phab task URL, title, assignee, author) in public P60752 (whic... [10:01:34] 06Release-Engineering-Team: Run a query to pull the number of Phabricator Tasks reported by community, resolved by WMF - https://phabricator.wikimedia.org/T362406#9721857 (10Aklapper) For the records, SQL query that was used: `mysql:phstats@m3-slave.eqiad.wmnet [phabricator_maniphest]> SELECT DISTINCT CONCA... [10:16:04] 10Diffusion, 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Clean up Diffusion's repo mirroring HTTP 403 failures in /var/log/phd/daemons.log on phab1004 - https://phabricator.wikimedia.org/T362021#9721908 (10Aklapper) ` aklapper@phab1004:~$ grep -r "Error while updating the" /var/log/phd/da... [10:29:37] 10Diffusion, 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Clean up Diffusion's repo mirroring HTTP 403 failures in /var/log/phd/daemons.log on phab1004 - https://phabricator.wikimedia.org/T362021#9721932 (10Aklapper) * https://phabricator.wikimedia.org/diffusion/NRDK fails because https://... [12:21:20] 10Gerrit (Gerrit 3.8): Gerrit commit message formatting does not handle angle-bracketed URLs well, adds extra semicolon - https://phabricator.wikimedia.org/T358762#9722259 (10hashar) 05Open→03Resolved a:03hashar The issue has been fixed by upgrading to Gerrit 3.8 on Monday 15th :) [12:23:16] 10Gerrit (Gerrit 3.8): Clicking "Add reviewer" button on Gerrit focuses the comment field, not the reviewers field - https://phabricator.wikimedia.org/T355535#9722263 (10hashar) 05Open→03Resolved a:03hashar It looks like the issue has been resolved in Gerrit 3.8. I can no more reproduce after upgrading... [12:39:28] 10Gerrit, 07Upstream: Gerrit does not send email when a new patchset is created using the web interface - https://phabricator.wikimedia.org/T343471#9722276 (10Paladox) Does this still not work (gerrit is now on 3.8)? [12:46:27] 10Gerrit, 07Upstream: Gerrit does not send email when a new patchset is created using the web interface - https://phabricator.wikimedia.org/T343471#9722290 (10Jdforrester-WMF) >>! In T343471#9722276, @Paladox wrote: > Does this still not work (gerrit is now on 3.8)? AIUI the web editor in gerrit is broken sin... [12:47:30] 10Gerrit, 07Upstream: Gerrit does not send email when a new patchset is created using the web interface - https://phabricator.wikimedia.org/T343471#9722292 (10Paladox) >>! In T343471#9722290, @Jdforrester-WMF wrote: >>>! In T343471#9722276, @Paladox wrote: >> Does this still not work (gerrit is now on 3.8)? >... [12:52:50] (03CR) 10Jforrester: "This is tagged against the wrong task?" [integration/config] - 10https://gerrit.wikimedia.org/r/1020477 (https://phabricator.wikimedia.org/T320818) (owner: 10Hashar) [12:55:02] (03CR) 10Jforrester: [C:03+2] "Oh, oops, thank you. Thought I had done so." [integration/config] - 10https://gerrit.wikimedia.org/r/1020359 (owner: 10Jforrester) [12:55:50] (03CR) 10Jforrester: "Why? These are the *selenium* jobs, which don't particularly depend on the PHP flavour, as I said in the commit message." [integration/config] - 10https://gerrit.wikimedia.org/r/1020362 (owner: 10Jforrester) [12:57:01] (03CR) 10Jforrester: "Why? The `test` pipeline gets run much more often, and works-in-8.1-but-not-7.4 issues are incredibly rare. It's better to run the faster " [integration/config] - 10https://gerrit.wikimedia.org/r/1020363 (owner: 10Jforrester) [13:14:12] 06Release-Engineering-Team, 06Infrastructure-Foundations, 06serviceops, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9722373 (10Jdforrester-WMF) >>! In T362518#9721307, @hashar wrote: >>>! In T362518#9719270, @Jdforrester-WMF wrote: >> This has also broken buil... [13:18:25] 10Gerrit, 07Upstream: Gerrit does not send email when a new patchset is created using the web interface - https://phabricator.wikimedia.org/T343471#9722381 (10hashar) The web editor breakage got filed as {T362545} and that is cache related. [13:52:52] 10Gerrit, 06Release-Engineering-Team: Gerrit "Checks" permalink to Zuul also matches other changes - https://phabricator.wikimedia.org/T358253#9722513 (10hashar) a:03hashar [13:53:02] (03PS1) 10Hashar: wm-zuul-status: filter based solely on change number [software/gerrit] (deploy/wmf/stable-3.8) - 10https://gerrit.wikimedia.org/r/1020840 (https://phabricator.wikimedia.org/T358253) [13:53:49] (03CR) 10Hashar: "It was a bit long to reach the task you filed back in February, but that is the implementation/fix :)" [software/gerrit] (deploy/wmf/stable-3.8) - 10https://gerrit.wikimedia.org/r/1020840 (https://phabricator.wikimedia.org/T358253) (owner: 10Hashar) [13:54:13] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: Gerrit "Checks" permalink to Zuul also matches other changes - https://phabricator.wikimedia.org/T358253#9722529 (10hashar) p:05Triage→03Low [14:02:59] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s, 06serviceops: Find a way to stage updated PHP packages on wikikube - https://phabricator.wikimedia.org/T362628#9722547 (10MoritzMuehlenhoff) This isn't just limited to updating PHP, but also extends to the full OS stack underneath (libs used by PHP etc). Whe... [14:21:39] 10Release-Engineering-Team (Now this 🫠), 10Scap, 06Data-Engineering, 06Data-Platform-SRE, and 2 others: analytics/refinery: Stop using git-fat - https://phabricator.wikimedia.org/T328472#9722622 (10BTullis) 05Open→03Resolved Thanks so much @dancy and @hashar and everyone else who has helped. I beli... [14:22:30] 10Release-Engineering-Team (Now this 🫠), 10Scap, 06serviceops-radar, 07git-lfs, 07Python3-Porting: git-fat replacement/removal - https://phabricator.wikimedia.org/T279509#9722630 (10BTullis) [14:22:31] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 06Data-Platform-SRE, 10Wikidata, and 2 others: wdqs: replace git-fat with git-lfs - https://phabricator.wikimedia.org/T316876#9722628 (10BTullis) [14:24:10] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Wikidata, 10Wikidata-Query-Service, and 2 others: wdqs: replace git-fat with git-lfs - https://phabricator.wikimedia.org/T316876#9722636 (10BTullis) [14:36:02] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Wikidata, 10Wikidata-Query-Service, and 2 others: wdqs: replace git-fat with git-lfs - https://phabricator.wikimedia.org/T316876#9722681 (10BTullis) I think I'm right in saying that we don't need to complete the archiva migration before switching t... [14:40:49] 10Release-Engineering-Team (Now this 🫠), 06cloud-services-team, 06collaboration-services, 10VPS-project-devtools, and 2 others: Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9722698 (10Dzahn) {F47170916} ^ afraid it's not stable yet. seems down again. [15:07:06] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s, 06serviceops: Find a way to stage updated PHP packages on wikikube - https://phabricator.wikimedia.org/T362628#9722794 (10akosiaris) > * We have staging base images and staging service images updated daily based on what is in Debian and apt.wikimedia.org In... [15:54:03] 10Release-Engineering-Team (Priority Backlog 📥), 10wikimedia.biterg.io: Enable domain matching in wikimedia.biterg.io configuration - https://phabricator.wikimedia.org/T327785#9723269 (10Aklapper) 05Stalled→03Open Future Hatstall version has been deployed two months ago, so we could have another discussion... [15:56:43] 06Release-Engineering-Team: Run a query to pull the number of Phabricator Tasks reported by community, resolved by WMF - https://phabricator.wikimedia.org/T362406#9723310 (10ARamirez_WMF) thank you @Aklapper! yes, i will make sure to add this disclaimer to the slide for amanda to present. [16:14:12] 10Gerrit, 06Security-Team, 07Security, 07Upstream: Investigate Gerrit email delay following 3.7.6 upgrade - https://phabricator.wikimedia.org/T355529#9723394 (10hashar) Upstream has released new versions of Gerrit on April 14th and made the issue I have filed public: https://issues.gerritcodereview.co... [16:33:20] 10Phabricator: Automate quarterly WMF QLS request for Phabricator data (tasks reported by community, resolved by WMF) - https://phabricator.wikimedia.org/T362804 (10Aklapper) 03NEW [16:33:42] 10Phabricator: Automate quarterly WMF QLS request for Phabricator data (tasks reported by community, resolved by WMF) - https://phabricator.wikimedia.org/T362804#9723493 (10Aklapper) p:05Triage→03Low a:03Aklapper [16:39:32] hashar: do you know who is handling the putty key revoking? There is someone asking about it on #mediawiki [17:16:22] bawolff: taavi / moritz and I removed a bunch early this morning [17:17:30] ah I see you found the task :) [17:18:02] I think the person figured it out. one of the users who was affected didn't know what he was supposed to do, but i think he figured it out [17:18:33] yeah my fault :/ [17:18:41] I guess I should have emailed them [17:22:58] bawolff: I have checked the accounts in Gerrit, they barely had any activity nor anything recent [17:23:12] so I guess they had an issue doing a ssh to toolforge/WMCS [17:23:22] and eventually got it resolved by generating a new key maybe? [17:23:46] we shall see tomorrow when Europeans are back [17:23:52] I am off for dinner [17:24:40] fwiw, I manually emailed the users whose WMCS keys I disabled. and that would certainly not have sent off any additional automated emails [18:20:21] I just got an email informing me that my SSH key has been deleted on Gerrit. [18:25:03] maxh: A bunch got disabled after the putty vuln. If possible, best to generate a new key and then add it to gerrit [18:26:00] Should I look into that some more? I assumed it didn't apply to me since I don't use putty. [18:28:07] It probably doesn't if you have never used putty [18:28:18] although im not an expert on this vuln [18:28:38] In that case, it was probably just diabled as a precaution [18:29:00] Maybe an admin would undisable it in that case? I'm not sure (/me looks at taavi ) [18:29:04] Yeah, fair. It is a P521 key. [18:31:48] Apparently I can just add it back myself. [18:44:03] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review, 05Release, 05Train Deployments: 1.43.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T361395#9724010 (10dancy) [18:58:04] Train is blocked on https://phabricator.wikimedia.org/T362817 [19:22:59] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Wikidata, 10Wikidata-Query-Service, and 2 others: wdqs: replace git-fat with git-lfs - https://phabricator.wikimedia.org/T316876#9724120 (10dancy) >>! In T316876#9722681, @BTullis wrote: > I think I'm right in saying that we don't need to complete... [19:35:23] (03PS3) 10Ahmon Dancy: Edit Repo Config [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1020740 [19:35:54] (03PS4) 10Ahmon Dancy: Enable lfs for wikidata/query/deploy [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1020740 (https://phabricator.wikimedia.org/T316876) [19:36:22] (03CR) 10Ahmon Dancy: [V:03+2 C:03+2] Enable lfs for wikidata/query/deploy [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1020740 (https://phabricator.wikimedia.org/T316876) (owner: 10Ahmon Dancy) [19:42:34] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Wikidata, 10Wikidata-Query-Service, and 3 others: wdqs: replace git-fat with git-lfs - https://phabricator.wikimedia.org/T316876#9724191 (10dancy) [20:02:20] 10Continuous-Integration-Infrastructure, 10doc.wikimedia.org, 10Doxygen, 10MediaWiki-General, and 2 others: mediawiki-core-doxygen fails on REL1_42: You are using an unsupported PHP version (PHP 7.4.33) - https://phabricator.wikimedia.org/T362385#9724231 (10Jdforrester-WMF) 05Open→03Resolved >>! In... [20:23:13] 10Gerrit, 07Upstream: Gerrit does not send email when a new patchset is created using the web interface - https://phabricator.wikimedia.org/T343471#9724323 (10matmarex) Just tried it, still not working – I edited https://gerrit.wikimedia.org/r/c/VisualEditor/VisualEditor/+/774399, I have my preferences set up... [21:08:10] 10Continuous-Integration-Infrastructure, 07Zuul: zuul-merger fails when repository names overlaps - https://phabricator.wikimedia.org/T157818#9724459 (10Dzahn) [21:08:13] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9724460 (10Dzahn) [21:24:11] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9724498 (10Dzahn) [21:41:47] 06Release-Engineering-Team, 10Scap, 06serviceops: scap should optionally display helmfile diffs for review - https://phabricator.wikimedia.org/T362717#9724523 (10Scott_French) Thanks, Reuven! I think it should be feasible to do something like that, yes. From a quick glance, it looks like the helm-diff plugi... [22:31:51] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9724668 (10Dzahn) kind of weird: contint1002 and contint2002 both have rsyncd running and rsync snippets once created by puppet but I can't find th... [22:33:31] 10Release-Engineering-Team (Now this 🫠), 10Scap, 10MediaWiki-libs-Mime, 10MediaWiki-Platform-Team (Radar), and 2 others: Get rid of the /srv/mediawiki/php symbolic link - https://phabricator.wikimedia.org/T359643#9724669 (10CodeReviewBot) dancy opened https://gitlab.wikimedia.org/repos/releng/train-dev/-/m... [22:34:10] 10Release-Engineering-Team (Now this 🫠), 10Scap, 10MediaWiki-libs-Mime, 10MediaWiki-Platform-Team (Radar), and 2 others: Get rid of the /srv/mediawiki/php symbolic link - https://phabricator.wikimedia.org/T359643#9724670 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/train-dev/-/m... [22:37:57] 06Release-Engineering-Team, 10Scap, 06serviceops: scap should optionally display helmfile diffs for review - https://phabricator.wikimedia.org/T362717#9724675 (10Scott_French) Hmmm ... I just realized that "ignoring" the image tag change (in the image-build case) when comparing the initial and pre-apply roun... [22:39:25] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9724676 (10Dzahn) see https://gerrit.wikimedia.org/r/c/operations/puppet/+/587491/5/modules/profile/manifests/ci/data_rsync.pp and https://gerrit... [22:42:42] 06Release-Engineering-Team, 10Scap, 06serviceops: scap should optionally display helmfile diffs for review - https://phabricator.wikimedia.org/T362717#9724679 (10RLazarus) That sounds reasonable! Note for the future that `helm diff` has a `--suppress-output-line-regex` which does exactly what you'd like it t... [22:49:38] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9724704 (10Dzahn) data pathes and sizes: existing primary server: ` root@contint2002:/# du -hs /var/lib/jenkins 2.2G /var/lib/jenkins root@cont... [23:29:27] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9724768 (10Dzahn) [23:42:58] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9724791 (10Dzahn) bugs in runbook: - there is no contint.discovery.wmnet name - it's contint.wikimedia.org - there is also a "gearman_server" set... [23:50:16] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9724808 (10Dzahn) [23:50:30] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9724809 (10Dzahn) @hashar topic branch: https://gerrit.wikimedia.org/r/q/topic:%22contint-bullseye%22 [23:51:09] 06Project-Admins, 06Movement-Insights: Create data-problem Phabricator tag - https://phabricator.wikimedia.org/T362839#9724810 (10nshahquinn-wmf) [23:52:29] 06Project-Admins, 06Movement-Insights: Create data-problem Phabricator tag - https://phabricator.wikimedia.org/T362839#9724815 (10nshahquinn-wmf) I don't want this created yet. First, I'm looking for feedback on the idea (primarily from Foundation folks involved in the data governance work mentioned in the des...