[00:11:12] 10Release-Engineering-Team (Social Piranhas 🐟), 10Striker, 10Patch-For-Review: Striker-created Diffusion mirrors of GitLab repos are empty (due to master vs main branch name mismatch) - https://phabricator.wikimedia.org/T348131 (10Aklapper) Thanks! Note to myself: I found at least [one toolforge-repos projec... [00:50:56] 10Release-Engineering-Team, 10RESTBase, 10serviceops: RESTBase scap deployment failed - https://phabricator.wikimedia.org/T349318 (10thcipriani) I figured out how it's possible for a server to get into this state. **Timeline** 1. Successful scap deployment 2. Remove `/etc/nagios/nrpe.d/check_endpoints_rest... [01:50:23] 10Beta-Cluster-Infrastructure, 10Thumbor, 10serviceops, 10Beta-Cluster-reproducible: deployment-prep needs a Thumbor instance - https://phabricator.wikimedia.org/T344605 (10AlexisJazz) I want to test a security issue but can't do it properly because beta cluster has no thumbnailer and I have no sysop bit e... [02:12:10] !log Add "Logs generated by test" console section to Jenkins [02:12:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:42:23] 10Release-Engineering-Team, 10Tech-Docs-Team, 10Documentation: Deployment pipeline (GitLab/Kokkuri/Blubber) documentation improvements - https://phabricator.wikimedia.org/T342317 (10KBach) [08:54:40] 10Beta-Cluster-Infrastructure, 10serviceops: Unable to upload files on Beta Commons - https://phabricator.wikimedia.org/T340908 (10hashar) I merely did the troubleshooting at T340908#8985768 , then I don't know anything about how memcached is setup or how to drive it through Puppet. I'd guessed @jijiki based o... [08:55:45] 10GitLab (CI & Job Runners), 10collaboration-services, 10serviceops: Standardize Debian package builds on GitLab CI - https://phabricator.wikimedia.org/T304491 (10MatthewVernon) Cool, I'm glad it's working for you :) I'm not wedded to the `.*-wikimedia.*` naming restriction, but I think some restriction on... [10:25:20] 10Release-Engineering-Team (Social Piranhas 🐟), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.42.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T348354 (10Lucas_Werkmeister_WMDE) >>! In T348354#9253290, @Lucas_Werkmeister_WMDE wrote: > * **Change**: Two Wikibase c... [10:58:47] 10Release-Engineering-Team (Social Piranhas 🐟), 10Striker: Striker-created Diffusion mirrors of GitLab repos are empty (due to master vs main branch name mismatch) - https://phabricator.wikimedia.org/T348131 (10Aklapper) 05Openβ†’03Resolved [11:05:19] I’m not sure I’m interpreting https://integration.wikimedia.org/ci/job/mwgate-node16-docker/79248/console correctly, but the _cacache on integration-agent-docker-1040 might need manual fixing? [11:05:25] (if this is the persistent kind of error) [11:08:30] unrelated: I just noticed the channel topic says β€œPhab β†’ Phorge migration in progress”, is that still true? ^^ [11:09:43] nope, that is a filthy lie by now!!!!1 [11:10:07] :D [11:16:06] grmbl, my IRC logs don’t say what the old topic was, only what it was changed to ^^ [11:16:16] (I was wondering if it used to be β€œup” or β€œall ok” or whatever) [11:16:27] want to just change it to something? [11:43:18] 10Release-Engineering-Team, 10RESTBase, 10serviceops: RESTBase scap deployment failed - https://phabricator.wikimedia.org/T349318 (10hnowlan) Looks like https://gerrit.wikimedia.org/r/c/operations/puppet/+/961003 was the source of the check disappearing, my bad. [11:51:25] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Social Piranhas 🐟), 10Patch-For-Review: docpub: allow publishing for failed pipelines - https://phabricator.wikimedia.org/T349309 (10CodeReviewBot) jnuche opened https://gitlab.wikimedia.org/repos/releng/docpub/-/merge_requests/12 allow publishing for... [12:07:05] Lucas_WMDE: normally it would be "Up" [12:14:07] 10Release-Engineering-Team (Priority Backlog πŸ“₯): Decrease number of open Phab tickets with assignee field set for more than two years (aka cookie licking) (Q3/2023 edition) - https://phabricator.wikimedia.org/T332676 (10Aklapper) Reminder email sent to 117 task assignees about their 221 open assigned tasks. [12:25:11] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Social Piranhas 🐟), 10Patch-For-Review: docpub: allow publishing for failed pipelines - https://phabricator.wikimedia.org/T349309 (10CodeReviewBot) jnuche opened https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/43 jenkins-rel:... [13:35:53] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for mabualruz - https://phabricator.wikimedia.org/T345186 (10Mabualruz) @thcipriani Thanks for the update I can attend any window maybe to switch it around I might attend the Evening window this time. I am free any day from 23... [14:07:37] 10WikimediaDebug, 10MediaWiki-Platform-Team, 10Observability-Tracing, 10MW-1.41-notes (1.41.0-wmf.27; 2023-09-19): Add support for request tracing to WikimediaDebug browser extension - https://phabricator.wikimedia.org/T340573 (10pmiazga) After a conversation with @CDanis, we clarified that this ticket is... [14:16:13] 10WikimediaDebug, 10MediaWiki-Platform-Team, 10Observability-Tracing, 10MW-1.41-notes (1.41.0-wmf.27; 2023-09-19): Add support for request tracing to WikimediaDebug browser extension - https://phabricator.wikimedia.org/T340573 (10pmiazga) [14:26:53] 10WikimediaDebug, 10MediaWiki-Platform-Team, 10Observability-Tracing, 10MW-1.41-notes (1.41.0-wmf.27; 2023-09-19): Add support for request tracing to WikimediaDebug browser extension - https://phabricator.wikimedia.org/T340573 (10pmiazga) @CDanis FYI: when the extension is enabled for the current request -... [14:27:16] 10WikimediaDebug, 10MediaWiki-Platform-Team, 10Observability-Tracing, 10MW-1.41-notes (1.41.0-wmf.27; 2023-09-19): Add support for request tracing to WikimediaDebug browser extension - https://phabricator.wikimedia.org/T340573 (10pmiazga) p:05Triageβ†’03Medium [14:30:04] 10WikimediaDebug, 10MediaWiki-Platform-Team, 10Observability-Tracing, 10MW-1.41-notes (1.41.0-wmf.27; 2023-09-19): Add support for request tracing to WikimediaDebug browser extension - https://phabricator.wikimedia.org/T340573 (10pmiazga) [14:58:43] 10WikimediaDebug, 10MediaWiki-Platform-Team, 10Observability-Tracing, 10MW-1.41-notes (1.41.0-wmf.27; 2023-09-19): Add support for request tracing to WikimediaDebug browser extension - https://phabricator.wikimedia.org/T340573 (10pmiazga) [16:17:13] 10GitLab, 10Release-Engineering-Team: Gerrit has some helpful diff views that it would be nice to have in Gitlab - https://phabricator.wikimedia.org/T349394 (10Tchanders) [16:22:26] 10Gitlab-Application-Security-Pipeline, 10Patch-For-Review, 10SecTeam-Processed, 10Security, 10user-sbassett: Improve related gitlab test repository - https://phabricator.wikimedia.org/T348788 (10CodeReviewBot) sbassett opened https://gitlab.wikimedia.org/repos/security/secteam-boilerplate-fork/-/merge_r... [17:35:45] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Patch-For-Review, and 2 others: Address issues within certain Gitlab CI security templates - https://phabricator.wikimedia.org/T338034 (10CodeReviewBot) sbassett merged https://gitlab.wikimedia.org/repos/security/gitlab-ci-s... [17:36:02] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Patch-For-Review, and 2 others: Address issues within certain Gitlab CI security templates - https://phabricator.wikimedia.org/T338034 (10CodeReviewBot) sbassett opened https://gitlab.wikimedia.org/repos/security/gitlab-ci-s... [18:01:51] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Patch-For-Review, and 2 others: Address issues within certain Gitlab CI security templates - https://phabricator.wikimedia.org/T338034 (10CodeReviewBot) sbassett opened https://gitlab.wikimedia.org/repos/security/gitlab-ci-s... [18:02:41] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Patch-For-Review, and 2 others: Address issues within certain Gitlab CI security templates - https://phabricator.wikimedia.org/T338034 (10CodeReviewBot) sbassett merged https://gitlab.wikimedia.org/repos/security/gitlab-ci-s... [18:27:51] dancy: I am guessing you are missing CR+2 on performance/fresnel repo? Or should we poke t.imo about the changes I made? ( ref: https://gerrit.wikimedia.org/r/q/project:performance/fresnel+is:open+label:code-review%252B1%252Cdancy ) [18:28:32] I do have +2 permission. I was going to leave the merges to you. [18:30:31] ah [18:30:34] then you can CR+2 them :) [18:30:43] okeedokes! [18:30:49] (03CR) 10Ahmon Dancy: [C: 03+2] build: dedupe npm dependencies [performance/fresnel] - 10https://gerrit.wikimedia.org/r/967137 (owner: 10Hashar) [18:30:58] (03CR) 10Ahmon Dancy: [C: 03+2] build: Update jsdoc-to-markdown to 8.0.0 [performance/fresnel] - 10https://gerrit.wikimedia.org/r/966933 (owner: 10Hashar) [18:31:01] (03CR) 10Ahmon Dancy: [C: 03+2] build: Update Qunit from 2.15.0 to 2.20.0 [performance/fresnel] - 10https://gerrit.wikimedia.org/r/967139 (owner: 10Hashar) [18:31:29] Puppetteer I have no idea [18:31:36] (03Merged) 10jenkins-bot: build: dedupe npm dependencies [performance/fresnel] - 10https://gerrit.wikimedia.org/r/967137 (owner: 10Hashar) [18:31:38] gotta ask t.imo about it since that upgrades Chrome [18:31:45] (03Merged) 10jenkins-bot: build: Update jsdoc-to-markdown to 8.0.0 [performance/fresnel] - 10https://gerrit.wikimedia.org/r/966933 (owner: 10Hashar) [18:31:53] (03Merged) 10jenkins-bot: build: Update Qunit from 2.15.0 to 2.20.0 [performance/fresnel] - 10https://gerrit.wikimedia.org/r/967139 (owner: 10Hashar) [18:34:34] rimraf well [18:34:47] I am not sure why a package is required to recursively delete files from the file system [18:35:42] ah https://boneskull.com/recursive-directory-removal-in-node-js/ [18:38:20] recursive If true, perform a recursive directory removal. In recursive mode, operations are retried on failure. Default: false. Deprecated. [18:38:26] it is of course marked deprecated [19:12:23] 10Release-Engineering-Team (Social Piranhas 🐟), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.42.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T348354 (10thcipriani) 05Openβ†’03Resolved Optimistically closing this out since we're on all groups. Keeping an eye... [19:12:30] 10Release-Engineering-Team (GitLab V: Event Horizon πŸŒ„), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T330208 (10thcipriani) [19:12:53] 10Release-Engineering-Team (Social Piranhas 🐟), 10Release, 10Train Deployments: 1.42.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T348355 (10thcipriani) [19:12:58] 10Release-Engineering-Team (GitLab V: Event Horizon πŸŒ„), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T330208 (10thcipriani) [19:13:15] well that's what I get going off my memory of what version we're on [19:14:59] (03PS1) 10Hashar: build: remove rimraf, require NodeJS 14.14.0 [performance/fresnel] - 10https://gerrit.wikimedia.org/r/967518 [19:35:29] (03CR) 10Hashar: "One less dependency :)" [performance/fresnel] - 10https://gerrit.wikimedia.org/r/967518 (owner: 10Hashar) [19:42:16] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Patch-For-Review, and 2 others: Address issues within certain Gitlab CI security templates - https://phabricator.wikimedia.org/T338034 (10CodeReviewBot) sbassett opened https://gitlab.wikimedia.org/repos/security/gitlab-ci-s... [19:42:52] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Patch-For-Review, and 2 others: Address issues within certain Gitlab CI security templates - https://phabricator.wikimedia.org/T338034 (10CodeReviewBot) sbassett merged https://gitlab.wikimedia.org/repos/security/gitlab-ci-s... [19:50:04] 10Gitlab-Application-Security-Pipeline, 10Patch-For-Review, 10SecTeam-Processed, 10Security, 10user-sbassett: Improve related gitlab test repository - https://phabricator.wikimedia.org/T348788 (10CodeReviewBot) sbassett merged https://gitlab.wikimedia.org/repos/security/secteam-boilerplate-fork/-/merge_r... [20:05:53] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Patch-For-Review, and 2 others: Address issues within certain Gitlab CI security templates - https://phabricator.wikimedia.org/T338034 (10CodeReviewBot) sbassett opened https://gitlab.wikimedia.org/repos/security/gitlab-ci-s... [20:07:17] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Patch-For-Review, and 2 others: Address issues within certain Gitlab CI security templates - https://phabricator.wikimedia.org/T338034 (10CodeReviewBot) sbassett merged https://gitlab.wikimedia.org/repos/security/gitlab-ci-s... [20:12:34] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Patch-For-Review, and 2 others: Address issues within certain Gitlab CI security templates - https://phabricator.wikimedia.org/T338034 (10CodeReviewBot) sbassett opened https://gitlab.wikimedia.org/repos/security/gitlab-ci-s... [20:13:15] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Patch-For-Review, and 2 others: Address issues within certain Gitlab CI security templates - https://phabricator.wikimedia.org/T338034 (10CodeReviewBot) sbassett merged https://gitlab.wikimedia.org/repos/security/gitlab-ci-s... [21:18:48] (03CR) 10Ahmon Dancy: [C: 03+2] "Excellent!" [performance/fresnel] - 10https://gerrit.wikimedia.org/r/967518 (owner: 10Hashar) [21:19:29] (03Merged) 10jenkins-bot: build: remove rimraf, require NodeJS 14.14.0 [performance/fresnel] - 10https://gerrit.wikimedia.org/r/967518 (owner: 10Hashar) [23:49:29] 10Release-Engineering-Team (Radar), 10DBA, 10MediaWiki-libs-Rdbms, 10Observability-Logging, 10Wikimedia-Logstash: Transaction profiler logs full query which is truncated by logstash - https://phabricator.wikimedia.org/T349140 (10Krinkle) I suspect this may've been caused or amplified by the now-resolved...