[01:53:47] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for adi.darkblue - https://phabricator.wikimedia.org/T434266 (10adi.darkblue) 03NEW [01:59:24] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for adi.darkblue - https://phabricator.wikimedia.org/T434266#12193855 (10Pharos) I endorse, we're both part of the Wikimedia NYC chapter, and collaborating on the Wikinewsie tool: https://phabricator.wikimedia.org/pro... [06:59:38] (03CR) 10Hashar: [C:03+2] release: Quibble 1.19.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1321961 (https://phabricator.wikimedia.org/T432934) (owner: 10Hashar) [06:59:56] (03CR) 10Hashar: [C:03+2] release: Start new version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1321962 (owner: 10Hashar) [07:14:07] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for adi.darkblue - https://phabricator.wikimedia.org/T434266#12194046 (10brennen) 05Openβ†’03Resolved a:03brennen This should have been handled by the addition to Trusted-Contributors, but possibly the approve... [07:14:16] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for adi.darkblue - https://phabricator.wikimedia.org/T434266#12194049 (10brennen) [07:18:51] (03Merged) 10jenkins-bot: release: Quibble 1.19.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1321961 (https://phabricator.wikimedia.org/T432934) (owner: 10Hashar) [07:18:56] (03Merged) 10jenkins-bot: release: Start new version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1321962 (owner: 10Hashar) [07:25:36] !log Tag Quibble 1.19.0 @ a8a84ed1c0adb34688ce22066ebde8a2d584a7d4 # T432934 T432943 T427922 [07:25:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:25:42] T432934: Make quibble report stage outcomes - https://phabricator.wikimedia.org/T432934 [07:25:43] T432943: Collect status and run time for npm installs in Quibble - https://phabricator.wikimedia.org/T432943 [07:25:43] T427922: Speedup the mwext-phpunit-coverage-publish castor job - https://phabricator.wikimedia.org/T427922 [08:06:36] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T430833#12194221 (10jnuche) 05Openβ†’03Resolved [08:20:19] (03PS1) 10Hashar: dockerfiles: update Quibble to 1.19.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1322660 [08:20:52] (03CR) 10Hashar: [C:03+2] dockerfiles: update Quibble to 1.19.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1322660 (owner: 10Hashar) [08:25:34] (03Merged) 10jenkins-bot: dockerfiles: update Quibble to 1.19.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1322660 (owner: 10Hashar) [08:48:59] (03PS1) 10Hashar: Update Quibble jobs to 1.19.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1322673 (https://phabricator.wikimedia.org/T432934) [09:06:18] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12194484 (10ABran-WMF) [09:09:21] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12194488 (10ABran-WMF) Thanks @ssingh for the rollout, I tested the backends today with: {P95940} ` === gitlab-replica-a.wikimedia.org === PASS direct... [09:10:40] (03PS2) 10Mpostoronca: zuul: Add EventBus and EventLogging deps for WikimediaAntiAbuse [integration/config] - 10https://gerrit.wikimedia.org/r/1321972 (https://phabricator.wikimedia.org/T432848) [09:13:21] (03CR) 10Mszwarc: [C:03+1] zuul: Add EventBus and EventLogging deps for WikimediaAntiAbuse [integration/config] - 10https://gerrit.wikimedia.org/r/1321972 (https://phabricator.wikimedia.org/T432848) (owner: 10Mpostoronca) [09:13:56] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, and 2 others: Get traces/logs from ATS on Gerrit CI errors - https://phabricator.wikimedia.org/T430901#12194523 (10ABran-WMF) p:05Highβ†’03Medium [09:15:03] maintenance-disconnect-full-disks build 831005 integration-agent-docker-1092 (/: 29%, /srv: 95%, /var/lib/docker: 34%): OFFLINE due to disk space [09:19:22] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, and 2 others: Gerrit: network errors in CI - https://phabricator.wikimedia.org/T429749#12194544 (10ABran-WMF) 05Stalledβ†’03Resolved The network error rate is now consistently lower than 1%, this... [09:20:03] maintenance-disconnect-full-disks build 831006 integration-agent-docker-1092 (/: 29%, /srv: 80%, /var/lib/docker: 34%): RECOVERY disk space OK [09:30:03] maintenance-disconnect-full-disks build 831008 integration-agent-docker-1092 (/: 29%, /srv: 100%, /var/lib/docker: 35%): OFFLINE due to disk space [09:35:03] maintenance-disconnect-full-disks build 831009 integration-agent-docker-1092 (/: 29%, /srv: 39%, /var/lib/docker: 34%): RECOVERY disk space OK [10:21:08] 10Phabricator Antivandalism Extension, 06Release-Engineering-Team (Doing 😎): Delete the "master" branch of the Phabricator Antivandalism extension repo - https://phabricator.wikimedia.org/T413890#12194759 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper `master` branch deleted via https://gerrit.wikimedia... [11:15:36] 10Phabricator: [QA] Create a Test Plan template in Phabricator - https://phabricator.wikimedia.org/T425375#12194935 (10Aklapper) 05In progressβ†’03Resolved Closing per last comment; author is not available anymore. [11:38:36] 10Phabricator: Remove disabled accounts from project / sub-projects - https://phabricator.wikimedia.org/T434289 (10OWresch-WMF) 03NEW [11:55:30] 10Phabricator: Remove disabled accounts from project / sub-projects - https://phabricator.wikimedia.org/T434289#12195032 (10Peachey88) 05Openβ†’03Resolved a:03Peachey88 Done, As projects don't have any members once they have subprojects created (they get moved to the created subproject), the person had t... [12:16:47] (03PS2) 10Jforrester: jjb: Skip npm install for mwext PHPUnit coverage jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1321626 (https://phabricator.wikimedia.org/T427922) (owner: 10Mhurd) [13:29:35] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Priority Backlog πŸ“₯): Disable Publishing for all source repos (to hopefully fix T143162 for good) - https://phabricator.wikimedia.org/T407967#12195231 (10Aklapper) I believe that https://gerrit.wikimedia.org/r/c/labs/striker/+/521367 and especially https... [13:29:48] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Priority Backlog πŸ“₯): Disable Publishing for all source repos (to hopefully fix T143162 for good) - https://phabricator.wikimedia.org/T407967#12195233 (10Aklapper) β†’14Duplicate dup:03T143162 [13:29:53] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 07Developer Productivity: Reduce task notification noise/frequency of changes to associated open patchsets - https://phabricator.wikimedia.org/T143162#12195235 (10Aklapper) [13:31:41] 10Diffusion, 10Phabricator (Upstream), 07TestMe, 07Upstream: Commits in a repository mirrored on Diffusion update unrelated tasks - https://phabricator.wikimedia.org/T260405#12195256 (10Aklapper) 05Openβ†’03Resolved Closing per T407967#12195231 and the previous comment here. Not sure it's solved in t... [13:42:38] (03PS1) 10Fomafix: Zuul: [mediawiki/extensions/HotCat] Add BetaFeatures dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1322768 [13:49:26] 06Release-Engineering-Team (Radar), 10Scap, 10MediaWiki-Internationalization, 13Patch-For-Review, 07Wikimedia-Performance-recommendation: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740#12195329 (10Aklapper) 05Stalledβ†’03Open [13:56:10] do we have data on (extensions)? repos where CI hasn't passed in a veeery long time? [13:56:41] like, occasionally I just stumble upon repos where all the activity is automated patches where they've either been force-merged or just sitting open for months at a time [13:57:22] and it would be cool if we could somehow detect those in a more structured way and maybe consider if those projects are still alive or if we should consider archiving them [13:58:22] 10Gerrit: Can't use 2nd SSH key - https://phabricator.wikimedia.org/T426139#12195339 (10Aklapper) 05Openβ†’03Stalled Sorry for the late answer. I'm not sure what to make out of this: > ` > debug2: we sent a publickey packet, wait for reply > debug3: receive packet: type 51 > debug1: Authentications that can co... [14:15:18] taavi: https://releng-data.wmcloud.org/-/dashboards/ci-by-repo-and-job [14:15:27] taavi: probably in there somewhere [14:15:55] One of them checks if a job passed and failed on the same sha1: https://releng-data.wmcloud.org/-/dashboards/ci-by-repo-and-job/top-flaky-jobs-30d [14:16:09] That means it has pass status and repo and sha1 at least [14:16:51] I guess you'd want to look for something with zero passing on one of the key jobs like PHPUnit and only failures [14:17:35] That will ofc return low activity repos where nothing was merged and the only patch proposed failed, but it's a start and you can tweak thresholds to your liking probably [14:27:15] 06Project-Admins, 10MediaWiki-Special-pages, 06Product Safety and Integrity: Split user-muting features (incl. Special:Mute) into its own Phabricator project (e.g. MediaWiki-Mute? / MediaWiki-Special-Mute?) - https://phabricator.wikimedia.org/T409674#12195424 (10Aklapper) Makes sense to me but I'd love to se... [14:31:40] !log integration: updating Quibble jobs to Quibble 1.19.0 # T432934 T432943 T427922 [14:31:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:31:46] T432934: Make quibble report stage outcomes - https://phabricator.wikimedia.org/T432934 [14:31:46] T432943: Collect status and run time for npm installs in Quibble - https://phabricator.wikimedia.org/T432943 [14:31:47] T427922: Speedup the mwext-phpunit-coverage-publish castor job - https://phabricator.wikimedia.org/T427922 [14:41:18] (03CR) 10Hashar: [C:03+2] Update Quibble jobs to 1.19.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1322673 (https://phabricator.wikimedia.org/T432934) (owner: 10Hashar) [14:41:37] (03PS2) 10Hashar: jjb: update Quibble jobs to 1.19.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1322673 (https://phabricator.wikimedia.org/T432934) [14:41:49] (03CR) 10Hashar: [C:03+2] jjb: update Quibble jobs to 1.19.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1322673 (https://phabricator.wikimedia.org/T432934) (owner: 10Hashar) [14:44:25] (03Merged) 10jenkins-bot: jjb: update Quibble jobs to 1.19.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1322673 (https://phabricator.wikimedia.org/T432934) (owner: 10Hashar) [14:55:37] (03CR) 10Hashar: [C:03+2] "Note: it is possible BetaFeatures has tests depending on other extensions. If that is the case, the tests would have to be marked skipped" [integration/config] - 10https://gerrit.wikimedia.org/r/1322768 (owner: 10Fomafix) [15:03:09] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/extensions/HotCat] Add BetaFeatures dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1322768 (owner: 10Fomafix) [15:03:28] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/HotCat] Add BetaFeatures dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1322768 (owner: 10Fomafix) [15:04:44] hashar: any easy investigation we in fr-tech should do re https://phabricator.wikimedia.org/T434024 ? [15:05:03] also, happy friday! [15:05:14] hmm [15:05:17] no clue! :) [15:05:23] hah, sweet :) [15:05:42] I'll bring it in DeveloperExperience on Slack cause I think I have seen something passing recently about build taking more time [15:05:45] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/HotCat] Add BetaFeatures dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1322768 (owner: 10Fomafix) [15:06:05] hashar: thank you [15:07:05] good thing, Zuul reports the duration of jobs to Prometheus and we thus have some graph https://grafana.wikimedia.org/d/ccaceff0-9ca6-4c89-9c4b-9138003439bf/zuul-job-prometheus?orgId=1&from=now-90d&to=now&timezone=utc&var-pipeline=$__all&var-job=wikimedia-fundraising-civicrm-bookworm&var-status=$__all&viewPanel=panel-5 [15:08:43] 10Phabricator Antivandalism Extension, 06Release-Engineering-Team (Doing 😎): Delete the "master" branch of the Phabricator Antivandalism extension repo - https://phabricator.wikimedia.org/T413890#12195577 (10Pppery) https://github.com/wikimedia/phabricator-antivandalism still has it. [15:11:09] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12195583 (10hashar) i think @mhurd andΒ @peter noticed the MediaW... [15:11:44] (03CR) 10Hashar: [C:03+2] "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/1322768 (owner: 10Fomafix) [15:12:47] greg-g: I don't think anything has changed, but really I can be wrong. I gotta dig [15:20:50] hashar: really appreciate it. I know it's hard to find direct causes for "has started to take longer "recently"" :) [15:22:37] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12195624 (10hashar) And the changes we made to the `docker-regis... [15:23:49] greg-g: yeah, though I am pretty sure I have seen something about the jobs being slower recently [15:24:21] * greg-g nods [15:30:41] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12195653 (10hashar) And from a Quibble / MediaWiki job, the dail... [15:36:16] 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team (Doing 😎), 10Projects-Cleanup: Archive GitLab "api-platform/catalog-to-wiki" repo and "api-platform" namespace - https://phabricator.wikimedia.org/T431560#12195668 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Archived https://g... [15:36:38] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12195674 (10Lars) Thanks @hashar. I did some work to speed up ou... [15:57:15] (03update) 10dancy: Release 4.278.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1243 [15:57:16] (03open) 10dancy: Release 4.278.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1243 [15:59:46] (03merge) 10dancy: Release 4.278.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1243 [16:13:50] !log integration: deleted integration-agent-T422258 agent # T422258 [16:13:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:13:53] T422258: Puppet fail to create volume group for ephemeral disk space when it is sda (instead of sdb) - https://phabricator.wikimedia.org/T422258 [16:33:19] 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 05FY252627 Reading Lists (Graduating from Beta), 07OKR-Work, and 2 others: ReadingLists tests not run in jenkins gate and submit job - https://phabricator.wikimedia.org/T403560#12196012 (10ldelench_wmf) [16:42:17] 10Phabricator, 06Release-Engineering-Team (Doing 😎), 06ServiceOps, 07Essential-Work: Rule to tag new bugs with `Serviceops new` instead of `serviceops` tag - https://phabricator.wikimedia.org/T414777#12196024 (10Aklapper) Sure sure, I've updated H476 accordingly now to also include #serviceops-radar in... [17:31:47] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12196149 (10Andrew) a minor thing that I did on the 24th: https... [17:37:05] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12196170 (10hashar) I went to look at [[ https://grafana.wikimed... [17:45:41] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12196202 (10Lars) @Andrew Interesting, our first timeout was abo... [18:19:18] I take it this codesniffer security issue dependency thing is in the process of being fixed by Libraryupgrader and I just need to be patient? [18:20:32] Lcawte: libup is indeed working on it but you can bump codesniffer manually on that repo if you're blocked by it [18:22:13] Not vital patches by any means - is there a task/dashboard/something to track it (other than gerrit feed) so I know when to push the rechecks again? I was assuming it was going alphabetically but it seems maybe not? [18:25:50] Lcawte: T434187 i believe [18:25:50] T434187: CI runs for repos with codesniffer 51 fail due to PKSA-rdkp-vv9z-mjkg - https://phabricator.wikimedia.org/T434187 [18:25:58] alphabetically-ish [18:26:19] there's no good dashboard for the libup push queue sadly [18:33:16] ty [18:37:38] 10Gerrit: Can't use 2nd SSH key - https://phabricator.wikimedia.org/T426139#12196331 (10Mroik) Nope, it's still giving me the same issue for the 2nd key. I even tried to remove it and re-add it but the issue is still there. [19:12:38] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, and 2 others: cloudcephosd1043 drives are very very busy - https://phabricator.wikimedia.org/T434334 (10Andrew) 03NEW [19:15:34] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, and 2 others: cloudcephosd1043 drives are very very busy - https://phabricator.wikimedia.org/T434334#12196432 (10Andrew) p:05Triageβ†’03High a:03Andrew This started happening not... [19:35:18] 10Gerrit: Can't use 2nd SSH key - https://phabricator.wikimedia.org/T426139#12196466 (10Aklapper) 05Stalledβ†’03Open Hmm, sorry. I'd recommend to also bring this up in a support forum to maybe gain more traction: https://www.mediawiki.org/wiki/Communication [19:46:53] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎), 10Striker, and 2 others: Striker: Always set all IO to none and hidden for internal URIs when creating Diffusion repositories - https://phabricator.wikimedia.org/T409341#12196487 (10Aklapper) [20:08:03] 06Project-Admins, 06Data-Persistence (work done), 07Developer Productivity: Fold Phab tags for optimizer-bug and Slow-DB-Query tag under Wikimedia-database-issue - https://phabricator.wikimedia.org/T305639#12196569 (10Aklapper) Opinions welcome. [22:14:41] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12196806 (10Lars) Thanks @Andrew and @hashar. I //think// it's b... [22:19:20] Project mediawiki-core-doxygen build #22133: 04FAILURE in 1 min 17 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/22133/ [22:21:06] (03open) 10jforrester: releases: Bump REL1_46 codesniffer from 50.0.0 to 51.0.1 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/144 (https://phabricator.wikimedia.org/T434187) [22:41:09] 10Gerrit: Can't use 2nd SSH key - https://phabricator.wikimedia.org/T426139#12196852 (10Mroik) After searching `APACHE-SSHD-2.12.0` I found that it is `mina-sshd`, a Java library for the ssh protocol. Specifically regarding my ssh key (which is hardware backed) I found [[ https://github.com/apache/mina-sshd/issu... [23:06:09] (03merge) 10taavi: releases: Bump REL1_46 codesniffer from 50.0.0 to 51.0.1 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/144 (https://phabricator.wikimedia.org/T434187) (owner: 10jforrester) [23:34:18] Yippee, build fixed! [23:34:18] Project mediawiki-core-doxygen build #22134: 09FIXED in 16 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/22134/