[03:01:47] Project mwcore-phpunit-coverage-master build #1710: 04STILL FAILING in 1 min 45 sec: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1710/ [06:48:01] 10Project-Admins, 10Wikimedia-Small-Projects-User-Group: Create project tags for WikiSP-Projects and WikiSP-Millestone - https://phabricator.wikimedia.org/T293170 (10Aklapper) On an unrelated note, https://phabricator.wikimedia.org/project/profile/2808/ was renamed from an ACL group to #WikiSP-Office, which vi... [06:49:11] 10Project-Admins, 10Wikimedia-Small-Projects-User-Group: Create project tags for WikiSP-Projects and WikiSP-Milestone - https://phabricator.wikimedia.org/T293170 (10Aklapper) [06:58:45] 10Beta-Cluster-Infrastructure: Certificate for *.beta.wmflabs.org has expired (July 2020) - https://phabricator.wikimedia.org/T257968 (10Aklapper) [06:58:56] 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic, 10HTTPS: Beta cluster certificates have expired (September 2020) - https://phabricator.wikimedia.org/T262806 (10Aklapper) [06:59:18] 10Beta-Cluster-Infrastructure: upload.wikimedia.beta.wmflabs.org certificate expired (October 2021) - https://phabricator.wikimedia.org/T293070 (10Aklapper) [07:18:22] (03PS1) 10Zoranzoki21: Archive the URNames extension [integration/config] - 10https://gerrit.wikimedia.org/r/731323 (https://phabricator.wikimedia.org/T293562) [07:56:51] 10Continuous-Integration-Config, 10Fresnel, 10Performance-Team: All quibble-fresnel jobs crash before starting: "Could not find browser" - https://phabricator.wikimedia.org/T292684 (10hashar) Congratulations Timo, and thank you for fixing it. [07:59:04] (03CR) 10Hashar: [C: 03+2] Archive the URNames extension [integration/config] - 10https://gerrit.wikimedia.org/r/731323 (https://phabricator.wikimedia.org/T293562) (owner: 10Zoranzoki21) [08:00:55] (03Merged) 10jenkins-bot: Archive the URNames extension [integration/config] - 10https://gerrit.wikimedia.org/r/731323 (https://phabricator.wikimedia.org/T293562) (owner: 10Zoranzoki21) [08:07:29] (03CR) 10Hashar: "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/731323 (https://phabricator.wikimedia.org/T293562) (owner: 10Zoranzoki21) [08:27:11] (03PS2) 10Hashar: Zuul: [mediawiki/extensions/MobileFrontendContentProvider] Enable CI [integration/config] - 10https://gerrit.wikimedia.org/r/731122 (owner: 10Zabe) [08:27:57] (03CR) 10Hashar: [C: 03+2] "Amended to require MobileFrontend (it is listed in extension.json as a requirement)." [integration/config] - 10https://gerrit.wikimedia.org/r/731122 (owner: 10Zabe) [08:29:37] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/MobileFrontendContentProvider] Enable CI [integration/config] - 10https://gerrit.wikimedia.org/r/731122 (owner: 10Zabe) [08:40:23] morning kostajh, get the latest mwcli update? ;D [08:40:27] its getting ALL of the commands [08:43:38] (03CR) 10Hashar: "Deployed, I give a recheck on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MobileFrontendContentProvider/+/730445" [integration/config] - 10https://gerrit.wikimedia.org/r/731122 (owner: 10Zabe) [09:00:05] addshore: just updated :) [09:00:33] <3 [09:02:22] although I think I need T293086 before I would start using mw more regularly [09:02:23] T293086: macOS host environment management - https://phabricator.wikimedia.org/T293086 [09:17:30] 10Phabricator: Puppet failure on phabricator-prod-1001.devtools.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T293604 (10hashar) [09:25:15] (03CR) 10Kosta Harlan: [C: 03+2] Replace setuptools_scm with setuptools-git-versioning [integration/quibble] - 10https://gerrit.wikimedia.org/r/730646 (https://phabricator.wikimedia.org/T292772) (owner: 10Hashar) [09:30:31] 10Phabricator: Puppet failure on phabricator-prod-1001.devtools.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T293604 (10hashar) The sole lead I have would be change in `ensure_packages` behavior. stdlib has been updated roughly at the same time from 7.0.1 to 8.1.0. Done by https://gerrit.wikimedia.... [09:41:24] !log lucaswerkmeister-wmde@deployment-mwmaint01:~$ mwscript createAndPromote.php commonswiki --sysop --force 'Lucas Werkmeister (WMDE)' # T248676 [09:41:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:41:26] T248676: Literal HTML
  • shown in case of an error while saving (here: page protected) - https://phabricator.wikimedia.org/T248676 [09:45:05] (03Merged) 10jenkins-bot: Replace setuptools_scm with setuptools-git-versioning [integration/quibble] - 10https://gerrit.wikimedia.org/r/730646 (https://phabricator.wikimedia.org/T292772) (owner: 10Hashar) [09:45:39] (03CR) 10Kosta Harlan: [C: 03+2] Support multiple workers in PHP 7.4+ web server [integration/quibble] - 10https://gerrit.wikimedia.org/r/730547 (https://phabricator.wikimedia.org/T259456) (owner: 10Hashar) [09:46:12] I broke quibble on my machine bha, mysql fails :D [09:46:41] yet another rabbit hole [09:51:08] hashar: oof [09:51:26] hashar: I rely on `--db-is-external` [09:51:28] might be related to a newer version of mariadb I have with bullseye [09:51:46] it fails to connect to mysql over a socket when doing the GRANT query [09:51:53] I will dig into it eventually [10:04:50] 10Phabricator: Puppet failure on phabricator-prod-1001.devtools.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T293604 (10dcaro) The issue (I think) is that the ensure_packages is running before the package declaration, and thus creates a new package resource, then the package declaration (in package... [10:05:52] (03Merged) 10jenkins-bot: Support multiple workers in PHP 7.4+ web server [integration/quibble] - 10https://gerrit.wikimedia.org/r/730547 (https://phabricator.wikimedia.org/T259456) (owner: 10Hashar) [10:41:51] found it, `mysql_install_db` now defaults to create a grant for the invoking user (no more `root` by default) [10:42:25] unless one passes `--auth-root-authentication-method=normal` (which is legacy) [10:43:11] will try to craft a patch eventually [10:47:59] 10Beta-Cluster-Infrastructure: Unable to undelete [[File:ACDC test file 1.pdf]] on Beta Wikimedia Commons - https://phabricator.wikimedia.org/T293558 (10dom_walden) @LucasWerkmeister I think it is because it has so many archived revisions that it cannot undelete. Might be same issue as T257298. Sorry for deleti... [12:36:16] 10Phabricator, 10Patch-For-Review: Puppet failure on phabricator-prod-1001.devtools.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T293604 (10dcaro) Yep, that made the trick! \o/ ` dcaro@phabricator-prod-1001:~$ sudo run-puppet-agent Info: Using configured environment 'production' Info: Retrieving... [12:46:20] 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic, 10Epic, 10HTTPS: [epic] The SSL certificate for Beta cluster domains fails to properly renew & deploy - https://phabricator.wikimedia.org/T293585 (10AlexisJazz) [13:07:44] (03PS1) 10Hashar: Try to connect to MySQL as normal user [integration/quibble] - 10https://gerrit.wikimedia.org/r/731437 [13:10:55] (03CR) 10jerkins-bot: [V: 04-1] Try to connect to MySQL as normal user [integration/quibble] - 10https://gerrit.wikimedia.org/r/731437 (owner: 10Hashar) [13:11:15] (03PS1) 10Hashar: Fix MySQL user creation on Debian Bullseye [integration/quibble] - 10https://gerrit.wikimedia.org/r/731438 [13:11:45] (03Abandoned) 10Hashar: Try to connect to MySQL as normal user [integration/quibble] - 10https://gerrit.wikimedia.org/r/731437 (owner: 10Hashar) [13:17:31] (03CR) 10Kosta Harlan: [C: 03+2] Fix MySQL user creation on Debian Bullseye [integration/quibble] - 10https://gerrit.wikimedia.org/r/731438 (owner: 10Hashar) [13:17:55] kostajh: I drafted something way more complicated until I have read the `mysql_install_db` doc ;) [13:23:34] 10Beta-Cluster-Infrastructure: upload.wikimedia.beta.wmflabs.org certificate expired (October 2021) - https://phabricator.wikimedia.org/T293070 (10Lucas_Werkmeister_WMDE) Yeah, in production deleted files just produce a 404, e.g. [this upload.wikimedia.org URL](https://upload.wikimedia.org/wikipedia/commons/9/94... [13:24:15] 10Beta-Cluster-Infrastructure: Unable to undelete [[File:ACDC test file 1.pdf]] on Beta Wikimedia Commons - https://phabricator.wikimedia.org/T293558 (10Daimona) >>! In T293558#7435575, @dom_walden wrote: > @LucasWerkmeister I think it is because it has so many archived revisions (10818) that it cannot undelete.... [13:31:17] 10Project-Admins: Create project tag for Safety Survey of WMF Trust and Safety Tools Team - https://phabricator.wikimedia.org/T292081 (10Aklapper) a:03Aklapper [13:31:59] 10Project-Admins: Create project tag for Safety Survey of WMF Trust and Safety Tools Team - https://phabricator.wikimedia.org/T292081 (10Aklapper) 05Open→03Resolved I went ahead and called this #WMF-Safety-Survey to be slightly more specific (though I've wondered if this should be a subproject of #trust_and_... [13:32:15] 10Project-Admins: Create "Radar" milestones for #Research and #Machine-Learning-Team project tags - https://phabricator.wikimedia.org/T283538 (10Aklapper) Hi, any news / decision here? [13:33:14] (03Merged) 10jenkins-bot: Fix MySQL user creation on Debian Bullseye [integration/quibble] - 10https://gerrit.wikimedia.org/r/731438 (owner: 10Hashar) [13:37:18] 10Continuous-Integration-Config, 10Toolforge, 10User-dcaro, 10cloud-services-team (Kanban): Set up CI for cloud/toolforge/buildpacks repository - https://phabricator.wikimedia.org/T265685 (10dcaro) [14:18:46] 10Continuous-Integration-Config, 10Toolforge Build Service, 10User-dcaro, 10cloud-services-team (Kanban): Set up CI for cloud/toolforge/buildpacks repository - https://phabricator.wikimedia.org/T265685 (10dcaro) [14:36:38] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/WikiLambda] Temporarily switch quibble to composer [integration/config] - 10https://gerrit.wikimedia.org/r/731741 (https://phabricator.wikimedia.org/T290274) [14:41:06] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/WikiLambda] Temporarily switch quibble to composer [integration/config] - 10https://gerrit.wikimedia.org/r/731741 (https://phabricator.wikimedia.org/T290274) (owner: 10Jforrester) [14:42:42] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikiLambda] Temporarily switch quibble to composer [integration/config] - 10https://gerrit.wikimedia.org/r/731741 (https://phabricator.wikimedia.org/T290274) (owner: 10Jforrester) [14:43:06] !log Zuul: [mediawiki/extensions/WikiLambda] Temporarily switch quibble to composer T290274 [14:43:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:43:09] T290274: Add any necessary dependencies for WikiLambda to production vendor once they've passed Security review - https://phabricator.wikimedia.org/T290274 [14:44:33] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T281168 (10dancy) 05Open→03Resolved [15:12:18] Project mwcore-phpunit-coverage-master build #1711: 04STILL FAILING in 1 min 40 sec: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1711/ [15:49:42] (03PS26) 10Ahmon Dancy: Replace simple git server with Gerrit [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 (owner: 10Dduvall) [15:50:04] (03CR) 10Ahmon Dancy: Replace simple git server with Gerrit (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 (owner: 10Dduvall) [16:05:19] ahoyhoy! I have a repo that already has a long history that was developed in a teammate's personal github that we now need to get into our own gerrit repo. What's the best way to import a (somewhat) long history that's already in master? Grant temporary permissions to push directly to master or is there something more graceful? [16:09:12] hnowlan: does the gerrit repo have anything in? [16:09:23] Or exist? [16:10:29] Spookreeeno: it exists, and only has a .gitreview file in it atm [16:11:27] qchris: maybe you can point out what you do ^ [16:12:05] hnowlan: no idea what's normally done but qchris normally imports them on creation if you ask [16:14:49] cool, thanks! [16:19:20] zuul seems very backlogged somehow :/ [16:25:00] 10Project-Admins, 10Wikimedia-Small-Projects-User-Group: Create project tags for WikiSP-Projects and WikiSP-Milestone - https://phabricator.wikimedia.org/T293170 (10Galahad) >>! In T293170#7434870, @Aklapper wrote: > On an unrelated note, https://phabricator.wikimedia.org/project/profile/2808/ was renamed from... [16:26:36] !log killed one beta-scap-sync-world job that seemed stuck for a long time [16:26:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:26:47] majavah: let’s see if ^ helps [16:27:31] (the job name used to be beta-scap-eqiad in https://w.wiki/J5$ but I guess it got renamed because I’m not seeing any job by that name now) [16:44:54] I can’t tell if it’s recovering or not, hopefully it is [17:14:20] (03CR) 10Ahmon Dancy: [C: 03+2] Replace simple git server with Gerrit [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 (owner: 10Dduvall) [17:14:31] (03CR) 10Ahmon Dancy: [C: 03+2] "Tested on Linux and Mac OS" [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 (owner: 10Dduvall) [17:14:45] (03Merged) 10jenkins-bot: Replace simple git server with Gerrit [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 (owner: 10Dduvall) [17:49:04] hnowlan: I'd just cherry-pick the .gitreview commit on top of what's the current main/master of your teammate's github repo, and push that as new master onto gerrit. It's probably easiest to temporarily grant force push to do that (as you suggested). [17:49:45] If the gerrit repo has seen some use, simply merge that in. [17:50:12] If you want to, I can try to do that for you. [18:03:13] 10Beta-Cluster-Infrastructure: Unable to undelete [[File:ACDC test file 1.pdf]] on Beta Wikimedia Commons - https://phabricator.wikimedia.org/T293558 (10LucasWerkmeister) I thought the number of revisions might be a problem, but I hoped that the undelete with just one timestamp (revision) and file ID might work... [18:16:32] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.38.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T281169 (10Jdlrobson) [18:18:50] 10Beta-Cluster-Infrastructure: Unable to undelete [[File:ACDC test file 1.pdf]] on Beta Wikimedia Commons - https://phabricator.wikimedia.org/T293558 (10LucasWerkmeister) I’ve uploaded a new version of the file, but PDF thumbnailing seems to be broken on Beta Commons (presumably unrelated), which makes the bold... [18:22:04] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.38.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T281169 (10Jdlrobson) @thcipriani I've missed a few train log triages, but we did say that newly introduced bugs that show up on the mw-client-errors dashboard or mw-clien... [18:26:43] 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic, 10Epic, 10HTTPS: [epic] The SSL certificate for Beta cluster domains fails to properly renew & deploy - https://phabricator.wikimedia.org/T293585 (10Dzahn) p:05Triage→03High [18:30:29] mutante: I wouldn't call it high. It's been an issue about a year now [18:30:37] That's just a tracking task [18:33:54] Spookreeeno: This quickly gets into 2 general phabricator discussions.. the first "should tracking tasks even exist" (I like them, others don't, who should I assign that to now etc) and "should priority reflect wishes or if someone is actually going to work on it" (I am in the camp that says it should reflect wishes, broken cert _should_ be high independent of how likely it is we can actually [18:34:00] fix all important things. Others incl. Andre will tell you though it should reflect how likely it is someone is going to work on it.. so then it's like Prio: low or lowest. [18:34:40] also I am not fully convinced priority field changes much but it's part of clinic duty to do that so I do it [18:35:26] let me change it back to Medium because "everything is medium" unless someone disagrees is very common [18:36:08] mutante: tbh I don't massively see the point in that tracking task, it doesn't list anything to actually work on beyond restart stuff every 90 days. I guess if someone actually cared about beta then it could be high but sadly no WMF team will dedicate time to it. [18:36:12] also the underlying issue here is who runs beta [18:36:20] I mean the question who does [18:36:27] QTE are officially the owners but they don't actually do anything [18:36:45] yea, you already found out [18:37:00] I've seen the task for the stewardship [18:38:08] I meant yes to that first statement, not the second, but looking it up [18:38:54] 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic, 10Epic, 10HTTPS: [epic] The SSL certificate for Beta cluster domains fails to properly renew & deploy - https://phabricator.wikimedia.org/T293585 (10Dzahn) p:05High→03Medium [18:40:43] Spookreeeno: I am not personally against tracking tasks but others are. I see not much but a little value in it. Seeing all the subtasks in one place while those can be assigned to different people and teams with their own addtiontal tags. [18:42:17] I also don't see a need to assign it directly to a person though. I don't think the main task is to find a single owner for each ticket. If anything they should be assigned on team level, not personal level. [18:43:19] I am noticing it does not have the QTE tag though that you just mentioned. I wasn't fully aware of that. [18:43:49] 10Beta-Cluster-Infrastructure, 10Quality-and-Test-Engineering-Team (QTE), 10SRE, 10Traffic, and 2 others: [epic] The SSL certificate for Beta cluster domains fails to properly renew & deploy - https://phabricator.wikimedia.org/T293585 (10Dzahn) [18:44:16] added that.. shrug [18:45:56] It's not really an "epic", but I'm tempted to swap it to #tracking-neverending (what it practically is) [18:46:05] https://phabricator.wikimedia.org/T215217#6665452 [18:46:17] mutante: ^ [18:46:48] majavah: based on what I can guess, it's going to be never ending [18:46:53] Spookreeeno: thanks! ack [18:47:45] aha, so it was planned but then was stalled and is actually open. *nod* [18:49:18] Yeah... [18:50:59] 10Release-Engineering-Team (Doing), 10Security-Team, 10ContentSecurityPolicy, 10GitLab (Administration, Settings & Policy), and 3 others: Define a Content Security Policy for GitLab - https://phabricator.wikimedia.org/T285363 (10hashar) On chromium I get: `counterexample Unrecognized Content-Security-Polic... [19:03:25] (03PS1) 10Ahmon Dancy: setup-certs: Add check for weak signature alg [tools/train-dev] - 10https://gerrit.wikimedia.org/r/731801 [19:03:27] (03PS1) 10Ahmon Dancy: setup-certs: Specify SHA256 signature algorithm [tools/train-dev] - 10https://gerrit.wikimedia.org/r/731802 [19:04:59] (03CR) 10Ahmon Dancy: [C: 03+2] setup-certs: Add check for weak signature alg [tools/train-dev] - 10https://gerrit.wikimedia.org/r/731801 (owner: 10Ahmon Dancy) [19:05:01] (03CR) 10Ahmon Dancy: [C: 03+2] setup-certs: Specify SHA256 signature algorithm [tools/train-dev] - 10https://gerrit.wikimedia.org/r/731802 (owner: 10Ahmon Dancy) [19:05:34] (03Merged) 10jenkins-bot: setup-certs: Add check for weak signature alg [tools/train-dev] - 10https://gerrit.wikimedia.org/r/731801 (owner: 10Ahmon Dancy) [19:05:56] (03Merged) 10jenkins-bot: setup-certs: Specify SHA256 signature algorithm [tools/train-dev] - 10https://gerrit.wikimedia.org/r/731802 (owner: 10Ahmon Dancy) [19:26:04] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.38.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T281169 (10brennen) I have to admit that I think we've been dropping the ball on triaging client errors. Or at least I personally have, during my last several trains. On... [19:28:31] addshore: \o how do I specify a different path for "MediaWiki code not detected in ~/dev/git/gerrit/mediawik" ? [19:29:21] are you in the wizard bit? or have already set it up and want to specify a different path now? [19:30:09] addshore: I just made `~/src/mwdd` as a clone of mwcore, and want to test out the mwcli docker environment with mutagen :) [19:30:23] gotcha, checkout mw dev env [19:30:30] but I don't know how to tell `mwcli` that this is my directory [19:30:32] BUT, be aware you need to put in the full path into there [19:30:47] the bug tracking that bit is https://phabricator.wikimedia.org/T284227 [19:30:57] and then doing `mediawiki create` again should load the new path [19:31:08] ah, ok [19:31:34] I might abstract the env file away from users via some other config to make that ~ bit easier to deal with [19:33:26] kostajh: am I right in thinking that as long as the volumes are not mounted from the mac host it should be speedy? [19:33:36] as in, just defining a volume is fine, as long as it isnt mounted files from disk? [19:35:31] addshore: it will be faster, but I don't know about speedy. that's what I want to check out again [19:38:07] addshore: is it possible none of the prompts are working due to the promptui library replacement? [19:38:30] D: I hope not, they work for me, at least the boolean ones do, let me check the others! [19:38:45] I want to figure out how to setup CI for them too [19:39:32] hmm, the prompts for "clone vector" or "run composer update" don't seem to do anything. [19:39:50] for vector, see T293689 [19:39:51] T293689: Vector skin is not cloned - https://phabricator.wikimedia.org/T293689 [19:40:30] hmm, but you have changed other things too right? [19:40:32] or? [19:40:36] lemme try it lcaolly [19:41:11] no, i just updated mediawiki.yml in the repo [19:42:25] works for me https://usercontent.irccloud-cdn.com/file/cWtuXmCd/image.png [19:42:36] hmm [19:42:50] addshore: have you seen this message before? `Error: No such volume: mwcli-mwdd-default_mediawiki-dot-composer` [19:43:14] where do you see that [19:43:25] that volume should be created when you do the `mediawiki create` [19:43:39] I ran `mwdev docker mediawiki destroy` and then `mwdev docker mediawiki create` [19:44:25] aaah right, that is while it is trying to destroy the volume, and you must not have run create since that volume was added [19:44:29] so that should just be noise [19:44:41] it wont happen again [19:45:05] ah, indeed [19:46:35] ok, I think I have enough to start hacking around with mutagen. I see that even though I'm using the aliased `mwdev` command, `mediawiki create` is copying `mediawiki.yml` from I guess the global install I have of `mw`? The old `mediawki.yml` shows up in the directory for `mwdev docker where`, not the new one with mutagen support [19:47:52] you need to modify the mediawiki.yml in the mwcli repo [19:47:56] and then `make build` [19:48:00] and then make sure you use your built binary [19:48:17] if you ever accidently run `mw` again, it'll overwrite your files with that was packaged with that version [19:49:10] I think we can probably add support without too many changes and maybe just 1 more env var [19:53:00] hrm, now I'm getting `go: inconsistent vendoring in /Users/kostajh/go/src/gitlab.wikimedia.org/releng/cli` with `make build` [19:53:39] `go mod vendor` [19:55:52] yeah, still failing... ".go: permission denied" [19:56:09] `rm -rf ./vendor` and try again [19:56:25] either old versions of go, or our old makefile did some bad things [19:56:35] you might need to sudo the rm [19:56:39] `sudo rm -rf vendor` worked [19:56:43] :) [19:56:48] (sorry, clearly I have no idea what I'm doing here!) [19:56:59] no worries, lots changed between go 1.13 and 1.16 :D [19:57:27] aha, now the prompts work [19:57:45] aaaah epic, i guess the library mgiht not even have been loaded properly [20:05:18] 10Release-Engineering-Team, 10SRE Observability: Alert RelEng when mw-client-error editing dashboard shows errors at a rate of over 1000 errors in a 12 hr period - https://phabricator.wikimedia.org/T293694 (10thcipriani) [20:05:44] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.38.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T281169 (10thcipriani) >>! In T281169#7438077, @brennen wrote: > I have to admit that I think we've been dropping the ball on triaging client errors. Or at least I persona... [20:06:31] 10Phabricator, 10Release-Engineering-Team (Doing), 10Security-Team: Grant access to OIT-LDAP Diffusion repo to contractor Danielattevelt - https://phabricator.wikimedia.org/T287124 (10sbassett) [20:15:05] kostajh: oh so there is a mutagen composer command in the new versions of mutagen? [20:15:21] *grabs the beta instead of whatever release he already got* [20:16:05] PROBLEM - SSH on gerrit2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [20:18:47] still https://phabricator.wikimedia.org/T283582 ... [20:19:07] kostajh: im starting to think more advanced templating might be needed for the files for usecases like this [20:20:07] " the Dell iDRAC sub-system is known to have as much processing power as a casio watch" lol [20:21:46] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10netops, 10ops-codfw: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ) - https://phabricator.wikimedia.org/T283582 (10Dzahn) @Papaul Could we schedule a firmware upgrade for gerrit2001 due to this issue? (not high prio) [20:23:49] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10netops, 10ops-codfw: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ) - https://phabricator.wikimedia.org/T283582 (10Papaul) @Dzahn sure we can. [20:30:24] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10netops, 10ops-codfw: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ) - https://phabricator.wikimedia.org/T283582 (10Dzahn) @cmooney Thank you very much for all the debugging effort you put into this and thanks @Papaul for confirming... [20:30:34] :( docker-compose doesnt let you do "${MEDIAWIKI_VOLUMES_CODE_MUTAGEN:-${MEDIAWIKI_VOLUMES_CODE}}:/var/www/html/w:cached" [20:31:14] ACKNOWLEDGEMENT - SSH on gerrit2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds daniel_zahn https://phabricator.wikimedia.org/T283582 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [20:32:49] kostajh: this is what I was thinking, but not super successful https://gitlab.wikimedia.org/releng/cli/-/commit/71ed4c256914b1d5ffc0c113962abdcea667264d [20:33:09] might be easier to use the new version, and switch all calls from docker-compose to use mutage composer [20:33:11] *compose [20:33:34] Especially if regular docker-compose ignores the `x-mutagen:` definitions in the docker-compose files, that would be awesome [20:35:02] 10Release-Engineering-Team, 10GitLab (Auth & Access), 10User-brennen: New GitLab users being marked as "External" - https://phabricator.wikimedia.org/T293696 (10brennen) [20:43:28] 10Release-Engineering-Team, 10GitLab (Auth & Access), 10User-brennen: New GitLab users being marked as "External" - https://phabricator.wikimedia.org/T293696 (10brennen) cc: @Jelto, @Arnoldokoth - any thoughts on this one? [20:43:51] 10Release-Engineering-Team, 10GitLab (Auth & Access), 10User-brennen: New GitLab users being marked as "External" - https://phabricator.wikimedia.org/T293696 (10brennen) p:05Triage→03High [20:48:39] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10netops, 10ops-codfw: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ) - https://phabricator.wikimedia.org/T283582 (10Papaul) @Dzahn I will go for turning this into a tracking ticket for firmware upgrades with check boxes of affected... [21:06:02] Fatal error: Uncaught Error: Class 'MediaWiki\MediaWikiServices' not found in /srv/mediawiki/php-1.38.0-wmf.4/extensions/TranslationNotifications/scripts/DigestEmailer.php:29 [21:06:22] ^ new fatal, breaks a maintenance job trying to send email [21:09:32] mutante: I'll a phab ticket. Do you have the whole transcript? [21:09:51] oh, I see action in operations. [21:11:12] https://phabricator.wikimedia.org/T293702 [21:11:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10ci-test-error (WMF-deployed Build Failure): TAR_ENTRY_ERROR ENOSPC: no space left on device - https://phabricator.wikimedia.org/T292729 (10hashar) 05Open→03In progress [21:11:59] 10Release-Engineering-Team, 10GitLab (Auth & Access), 10User-brennen: New GitLab users being marked as "External" - https://phabricator.wikimedia.org/T293696 (10brennen) I think this is it: - https://docs.gitlab.com/ee/integration/omniauth.html#configure-omniauth-providers-as-external - [[https://gitlab... [21:12:33] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review, 10ci-test-error (WMF-deployed Build Failure): TAR_ENTRY_ERROR ENOSPC: no space left on device - https://phabricator.wikimedia.org/T292729 (10hashar) I have sent a series of patch to prune the images on a daily basis:... [21:13:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10ci-test-error (WMF-deployed Build Failure): TAR_ENTRY_ERROR ENOSPC: no space left on device - https://phabricator.wikimedia.org/T292729 (10hashar) [21:14:02] 10Gerrit, 10Release-Engineering-Team (Seen): Install rename-project plugin - https://phabricator.wikimedia.org/T201953 (10hashar) a:05hashar→03None [21:17:03] RECOVERY - SSH on gerrit2001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [21:20:30] 10Release-Engineering-Team (Radar), 10Upstream: Upgrade zuul gearman when upstream releases it - https://phabricator.wikimedia.org/T289512 (10hashar) 05Stalled→03Open gear 0.16.0 has been released! On 18/10/2021 17:55, Clark Boylan wrote: > I am happy to announce version 0.16.0 of Gear has been released.... [21:20:41] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Zuul, 10Upstream: Upgrade zuul gearman when upstream releases it - https://phabricator.wikimedia.org/T289512 (10hashar) [21:22:14] 10Release-Engineering-Team, 10GitLab (Auth & Access), 10User-brennen: New GitLab users being marked as "External" - https://phabricator.wikimedia.org/T293696 (10Arnoldokoth) Hey @brennen Yeah, I got the same from reading the docs. With that line in the configuration, all users who sign in through the cas3 p... [21:27:59] 10Release-Engineering-Team, 10GitLab (Auth & Access), 10User-brennen: New GitLab users being marked as "External" - https://phabricator.wikimedia.org/T293696 (10brennen) Thanks for confirming. > "If you decide to remove an OmniAuth provider from the external providers list, you must manually update the user... [21:29:47] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.38.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T281169 (10hashar) Acknowledging that I am indeed running the train this week. My first thought has been it is a bit unfair to block this week train for an issue that star... [21:30:07] 10Release-Engineering-Team, 10GitLab (Auth & Access), 10User-brennen: New GitLab users being marked as "External" - https://phabricator.wikimedia.org/T293696 (10Dzahn) In this case "cas3" refers to https://idp.wikimedia.org which is providing the so while it is an auth provider under our domain and control... [22:27:38] 10Release-Engineering-Team, 10SRE Observability: Alert RelEng when mw-client-error editing dashboard shows errors at a rate of over 1000 errors in a 12 hr period - https://phabricator.wikimedia.org/T293694 (10Jdlrobson) Perhaps https://grafana.wikimedia.org/d/000000566/overview?viewPanel=16&orgId=1&from=now-30... [22:32:32] (03PS8) 10Ahmon Dancy: Access train-dev gerrit server instead of prod gerrit [tools/train-dev] - 10https://gerrit.wikimedia.org/r/723267 [22:33:01] (03CR) 10jerkins-bot: [V: 04-1] Access train-dev gerrit server instead of prod gerrit [tools/train-dev] - 10https://gerrit.wikimedia.org/r/723267 (owner: 10Ahmon Dancy) [22:34:10] (03PS9) 10Ahmon Dancy: Access train-dev gerrit server instead of prod gerrit [tools/train-dev] - 10https://gerrit.wikimedia.org/r/723267 [22:36:07] (03CR) 10Ahmon Dancy: [C: 03+2] Access train-dev gerrit server instead of prod gerrit [tools/train-dev] - 10https://gerrit.wikimedia.org/r/723267 (owner: 10Ahmon Dancy) [22:36:34] (03Merged) 10jenkins-bot: Access train-dev gerrit server instead of prod gerrit [tools/train-dev] - 10https://gerrit.wikimedia.org/r/723267 (owner: 10Ahmon Dancy) [23:10:39] 10Beta-Cluster-Infrastructure, 10VisualEditor: [betalabs] Loading VE produces "Invalid response from server" error - https://phabricator.wikimedia.org/T293716 (10Etonkovidova) [23:24:33] 10Beta-Cluster-Infrastructure, 10Parsoid, 10VisualEditor: [betalabs] Loading VE produces "Invalid response from server" error - https://phabricator.wikimedia.org/T293716 (10Etonkovidova)