[01:28:31] 10Phabricator (Upstream): Upload popup overflows box with a long filename - https://phabricator.wikimedia.org/T316305 (10Reedy) [01:28:40] 10Phabricator (Upstream): Upload popup overflows box with a long filename - https://phabricator.wikimedia.org/T316305 (10Reedy) [01:32:41] 10Phabricator (Upstream): Custom Policy window overflows when task name is too long - https://phabricator.wikimedia.org/T316308 (10Reedy) [05:07:36] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Data-Persistence, 10serviceops, 10serviceops-collab: sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Marostegui) You should be ok: ` root@phab1004:~# telnet m3-master.eqiad.wmnet 3306 Trying 10.64... [06:31:49] 10Continuous-Integration-Config, 10Toolhub, 10Documentation, 10User-Slst2020: Publish docs on doc.wikimedia.org - https://phabricator.wikimedia.org/T305914 (10Slst2020) [06:31:51] 10Continuous-Integration-Config, 10Toolhub, 10Documentation, 10Patch-For-Review, 10User-Slst2020: Set up publish-docs pipeline job - https://phabricator.wikimedia.org/T308632 (10Slst2020) 05In progress→03Open [06:37:43] 10Continuous-Integration-Config, 10Release Pipeline, 10Documentation, 10Patch-For-Review, 10User-dduvall: Provide downstream job for publishing docs from a Jenkins artifact to doc.wikimedia.org - https://phabricator.wikimedia.org/T308591 (10Slst2020) @dduvall Can this task be closed or is anything else n... [08:31:41] 10Diffusion, 10Toolforge: Figure out a git hosting solution for tools/kubernetes - https://phabricator.wikimedia.org/T117071 (10dcaro) 05Open→03Resolved a:03dcaro This task is no longer relevant, we will support creating containers through buildpacks using any publicly accessible git repository (github/b... [08:55:07] 10Continuous-Integration-Config, 10Toolforge Build Service, 10Cloud-Services-Origin-Team, 10Cloud-Services-Worktype-Project, and 2 others: [tbs] Set up CI for cloud/toolforge/buildpacks repository - https://phabricator.wikimedia.org/T265685 (10dcaro) [08:55:57] 10Continuous-Integration-Config, 10Toolforge Build Service, 10Cloud-Services-Origin-Team, 10Cloud-Services-Worktype-Project, and 2 others: [tbs] Set up CI for cloud/toolforge/buildpacks repository - https://phabricator.wikimedia.org/T265685 (10dcaro) [09:49:25] (03CR) 10Hashar: [C: 03+2] "Well done. I have created the jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/826663 (https://phabricator.wikimedia.org/T314063) (owner: 10Ebernhardson) [09:51:17] (03Merged) 10jenkins-bot: Add CI for search/cirrus-streaming-updater [integration/config] - 10https://gerrit.wikimedia.org/r/826663 (https://phabricator.wikimedia.org/T314063) (owner: 10Ebernhardson) [10:00:39] (03CR) 10Hashar: [C: 03+1] Zuul: [mediawiki/core] Make PHP 8.1 voting on REL1_38 and REL1_39 [integration/config] - 10https://gerrit.wikimedia.org/r/826216 (https://phabricator.wikimedia.org/T316080) (owner: 10Jforrester) [10:00:44] (03CR) 10Hashar: [C: 03+1] Zuul: [mediawiki/core] Add a non-vendor php81 job for main branch [integration/config] - 10https://gerrit.wikimedia.org/r/826217 (https://phabricator.wikimedia.org/T316078) (owner: 10Jforrester) [10:35:57] 10Phabricator, 10Traffic: Phabricator was logging out users repeteadly - https://phabricator.wikimedia.org/T316337 (10jcrespo) 05Invalid→03Open [10:46:20] 10Phabricator, 10SRE, 10Traffic, 10Wikimedia-Incident: Phabricator was logging out users repeteadly - https://phabricator.wikimedia.org/T316337 (10jcrespo) [10:47:19] 10Phabricator, 10SRE, 10Traffic, 10Wikimedia-Incident: Phabricator was logging out users repeteadly - https://phabricator.wikimedia.org/T316337 (10Vgutierrez) [10:59:40] 10Phabricator, 10SRE, 10Traffic, 10Wikimedia-Incident: Phabricator was logging out users repeteadly - https://phabricator.wikimedia.org/T316337 (10jcrespo) Preliminary working doc: https://docs.google.com/document/d/1Ka9MQB8OwdzAzJVfZuaIGo5VfnyRNRr_WxLPZ6YFMkE [11:23:14] 10Gerrit: https://your.install.com/application/edit/?title=Platyplus&body=Ornithopter - https://phabricator.wikimedia.org/T316343 (10SethRatcliffe) [11:23:37] 10Gerrit: https://your.install.com/application/edit/?title=Platyplus&body=Ornithopter - https://phabricator.wikimedia.org/T316343 (10SethRatcliffe) https://your.install.com/application/edit/?title=Platyplus&body=Ornithopter [12:31:20] 10Phabricator, 10SRE, 10Traffic, 10Wikimedia-Incident: Phabricator was logging out users repeatedly (2022-08-26) - https://phabricator.wikimedia.org/T316337 (10Aklapper) [14:04:32] (03CR) 10Hashar: Show CI status while waiting for change to merge (032 comments) [tools/release] - 10https://gerrit.wikimedia.org/r/826675 (https://phabricator.wikimedia.org/T315452) (owner: 10Hashar) [14:05:02] (03PS4) 10Hashar: Show CI status while waiting for change to merge [tools/release] - 10https://gerrit.wikimedia.org/r/826675 (https://phabricator.wikimedia.org/T315452) [15:37:19] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Data-Persistence, 10serviceops, 10serviceops-collab: sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Dzahn) Thank you! So I like the part where I don't have to worry about the grants, that's nice... [17:48:25] 10Diffusion, 10Gerrit, 10serviceops, 10serviceops-collab, and 2 others: Gerrit replication to codfw (gerrit-replica.wikimedia.org) stopped working after Gerrit 3.4.5 upgrade - https://phabricator.wikimedia.org/T315942 (10Dzahn) SSHd and ssh client settings have been adjusted to use ecdh-sha2-nistp521 as ke... [17:55:26] (03CR) 10Chad: [V: 03+2 C: 03+2] Edit Repo Config [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/826619 (owner: 10Chad) [18:06:34] <^demon> Can't submit that, that's annoying ^ [18:07:25] ^demon: hmmm.. submitted [18:07:31] but let's fix it too? [18:08:35] <^demon> I can't figure out why I can't -- Admins should have Submit on refs/meta/* [18:09:10] ^demon: you were the one solving gerrit issues not so long ago :) [18:09:34] maybe https://phabricator.wikimedia.org/T234474 [18:09:37] <^demon> Back to my old shenanigans :p [18:09:48] Is he in ldap/gerritadmin mutante ? [18:10:04] because some privs were moved to that ldap group because of issues [18:10:40] hauskatze: indeed, I was about to say https://phabricator.wikimedia.org/T315887 [18:10:59] and that needs https://phabricator.wikimedia.org/T273164 so ... dunno [18:11:05] https://ldap.toolforge.org/group/gerritadmin [18:11:08] LDAP says no [18:11:20] <^demon> Ahhhhh, https://gerrit.wikimedia.org/r/admin/groups/6aec730b12a6e11add70e4f248fcc33b75e47e5c,members -- we no longer have admin group like before? [18:11:31] no because something happened [18:12:55] * ^demon sits down for story time [18:14:35] (03CR) 10Dzahn: "let's just add some more detail what is being edited (next time)" [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/826619 (owner: 10Chad) [18:18:20] no drink, no story [18:29:56] <^demon> Um I checked and all I have in my apartment is this already opened bottle of water or the unchilled tequila I have on the shelf. I can squeeze a lime into the bottle of water if you'd like, or garnish the warm tequila. Your call :) [18:31:14] * Platonides suggest to send the Fedex guy from a nearby store rather than from demon's house [18:31:47] it would be funny to see the message "You have a package sent from hell", though [18:48:55] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10leila) Hi. I'd like to join the admin group. It will allow me to create milestones for the [[ https://phabricator.wikimedia.org/project/profile/45/ | Research board/project ]]. Thanks! [18:53:36] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Ladsgroup) @leila @vyuen and @Evelien_WMDE: Done //Usual disclaimer: Please follow the [guidelines](https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects#Creatin... [19:28:06] 10Release-Engineering-Team (Radar), 10Research, 10User-brennen: Define Metrics for Change Failure Percentage - https://phabricator.wikimedia.org/T289567 (10leila) [19:53:21] <^demon> So, after spending the past few days digging, here's where we are with Git LFS support.... (1) It's enabled for a few repositories, but nobody is actively using it (2) I was at least able to get it to respond to a request to lock/unlock a file that I was tracking (confirmed with lockfiles on-disk) (3) I have _not_ been able to actually get the file pushed to disk on `gerrit1001:/var/lib/gerrit2/review_site/data/lfs/*` yet [19:53:21] <^demon> (4) Having to set `lfs.url` locally seems to be a lie, it auto-discovers the locking mechanism just fine without it [19:53:32] <^demon> I'm increasingly convinced that Git LFS isn't all that great and if we would be better served finding non-Git solutions to the problem [19:53:52] <^demon> thcipriani, mutante ^ [19:59:53] hrm, there's like 19GB of stuff on gerrit in lfs (not that I can tell what it is since it's all "zip archive data" stored by sha) but someone must use it for something [19:59:56] Disregarding for a moment what problem we are solving, but when I see "nobody is actively using it" and I know it adds complexity to our setup.. then I immediately want to remove it [20:00:42] the "adds complexity" part definitely aligns with all my biases :) [20:00:42] could it be design files? [20:00:53] before design moved to scap [20:00:54] <^demon> thcipriani: Where? [20:00:56] or something [20:01:03] <^demon> It's not where it should be? [20:01:16] in puppet we have the lfs path set to /srv/gerrit/plugins/lfs [20:01:21] I remember design wanted a place to store all those large files [20:01:27] <^demon> D'oh. [20:01:31] (maybe/probably not the default place) [20:01:40] <^demon> Here I was assuming it was the default location [20:02:01] rookie mistake ;) [20:02:22] :D [20:02:35] <^demon> In my defense, the locks were! [20:02:47] yeah, /srv/ is our big...what...volume group? whatever. It's where all the space is. [20:03:18] <^demon> Ok, so scratch (1) (2) and (3)..... (4) is just a happy coincidence I think? [20:04:20] <^demon> So basically we just need to find what repos still have a `.gitfat` and/or `.gitattributes` with `filter=fat` and nudge them to use LFS instead? [20:05:29] yeah, there's a whole workflow for archiva, so pointing to some docs would be good, too [20:06:10] like if you were struggling to get it to work with access to the server, what chance does a mortal have? :) [20:06:55] <^demon> Well, I guess I wasn't struggling I was just looking in the wrong place? [20:08:38] <^demon> If the people already using it are ok with it then let's not fix what ain't broken. But for Gerrit....I think there's a better way than using Git LFS or Archiva. [20:09:07] I am a bit nervous about the storage use [20:09:22] like: all the archiva stuff lives somewhere else and I don't know how much of it there is [20:09:59] but the lfs stuff lives on gerrit and I know how much space is there and it's...limited :) [20:12:29] <^demon> Well, the only reference I see to any scap-deployed things using git-fat is WDQS [20:13:16] and maybe ores [20:14:18] <^demon> ORES uses git lfs, from what I can tell? [20:14:45] root@archiva1002:/var/lib/archiva/data# du -hs . [20:14:47] 21G [20:14:50] fwiw [20:14:57] <^demon> thcipriani: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/ores/deploy/+/refs/heads/master/scap/scap.cfg#5 [20:15:33] oh, sorry, you're right, for some reason read "git-lfs" not "git-fat" [20:15:49] we also use it for gerrit itself (last I checked) [20:16:02] git-fat, that is [20:16:15] <^demon> Yeah, that's what I was saying :) [20:16:22] I guess I could believe those are the only two [20:16:46] <^demon> I think Gerrit could be packaged and deployed without LFS or Fat, actually [20:17:45] root@archiva1002:/var/lib/archiva/repositories/releases/com/googlesource# du -hs gerrit/ [20:17:48] 2.4G gerrit/ [20:19:49] available on gerrit1001 /srv: 35G available on gerrit2002 /srv: 530 whopping GB [20:21:06] I have no idea how many of these are still deployed this way (or still deployed at all): https://phabricator.wikimedia.org/P33317 [20:21:58] lots of dusty old files in here :) [20:22:24] <^demon> wdqs definitely is still deployed [20:24:48] archiva also has artifacts for gerrit 2.5.10 and every version that came after that. for a single gerrit version it's just about 75MB. [20:29:41] updated the paste with the last deploy dates of each of those, about half are pretty recent, probably still in-use [20:30:12] only 2 where the last deploy was > 1yr ago [20:32:31] we could garbage collect some of those artifacts. Like I don't even think it's possible to rollback gerrit very far, definitely not to 2.5.10 [21:38:54] 10GitLab (Project Migration), 10Striker, 10Tools, 10Goal, 10User-bd808: Replace Diffusion integration with Gitlab integration in Striker (toolsadmin) - https://phabricator.wikimedia.org/T296893 (10bd808) 05Open→03In progress a:03bd808