[09:54:54] jynus: morning :), I need some help getting a mysql replica working again after a master got out of space, is there any procedure written somewhere? (/me is searching around, but not finding too much) [09:56:45] marostegui: ^ maybe you have an idea too? (for reference, ticket T301951) [09:56:45] T301951: clouddb1002 (toolsdb secondary) replication broken - https://phabricator.wikimedia.org/T301951 [10:05:54] dcaro: you should ask on #data-persistence for a dba, I don't have backups for you [10:06:17] #wikimedia-data-persistence , that is [10:08:29] ack [10:39:33] FYI; I'll briefly disable puppet in eqiad for a few minutes [10:45:53] and back on [12:42:45] FYI I'm going to start moving drmrs to the new routers, I tried to downtime drmrs as much as I could in incing but we might get some noise [13:01:39] nice :) [16:37:08] _joe_: do you have a minute to review https://gerrit.wikimedia.org/r/c/operations/puppet/+/751709 ? Should be short [16:37:31] <_joe_> dcaro: not right now, but I'll try to make room [16:37:52] _joe_: thanks, there's no rush, but sometime this week would be nice (has been around for a bit) [16:38:20] <_joe_> sorry just realized [16:38:25] <_joe_> I was sure I +1'd it [16:39:01] np, I'm here to remind you about it ;) [17:09:07] Hi all, I'm making a new vm via makevm and I see some hosts changing for drmrs: ae0.asw1-b12-drmrs is removed and et-0-0-50.asw1-b12-drmrs and et-0-0-2.cr2-drmrs are added. Could anybody confirm this is fine? [17:10:20] XioNoX: ^^^ [17:11:10] volans: I merged it a while go [17:11:12] ago [17:11:21] razzi: yeah it's fine [17:11:27] ok thanks XioNoX ! [17:39:43] does anyone use any gerrit cli tool to look at comments on a patchset in a terminal? [17:42:00] jhathaway: something other then `git-review -d 763370; git diff HEAD~1` im gussing? [17:42:20] nevermind i see you said comments not commits [17:42:22] * jhathaway goes to try out the command [17:42:58] ah yes, I was hoping to see comments on certain lines [17:43:29] not sure, but hashar may know :) [17:46:21] jhathaway: I don't know of any utilities offhand, but, the data is available in Gerrit's API [17:46:30] :) [17:47:02] https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#list-comments [17:47:38] beware the `)]}'` if you are parsing gerrit API output yourself [17:49:35] (XSSI protection...) [18:11:50] gerrit also stores its meta data in git, so it seems feasible to clone the meta data repo [18:15:57] <_joe_> jhathaway: yes, I once had to dig deep into its bowels, but it's too late in my day to try to recall [18:16:11] :) [18:16:12] <_joe_> there's a whole hierarchy of git refs [19:27:36] moritzm: the docs for bullseye tell me "For OpenStack nodes running nova-compute or cinder-volume, it is strongly advised to add the parameters systemd.unified_cgroup_hierarchy=false and systemd.legacy_systemd_cgroup_controller=false to the kernel command line in order to override the default and restore the old cgroup hierarchy. " [19:27:57] Is there a 'correct' way to alter the kernel command line with puppet, or should I invent a way? [20:07:30] jhathaway: the comments made on patches are not available via git iirc [20:07:41] jhathaway: they are stored in a database on the Gerrit side. But maybe I am wrong [20:08:21] yeah I'm not sure, evidently they switched to notedb, and everything is stored in the notedb git repo [20:08:32] but I haven't tried cloning it and investigating [20:08:34] you could fetch some review metadata though [20:09:03] in my ~/.gitconfig I have: [20:09:05] [alias] [20:09:25] fetchreview = fetch origin refs/notes/review:refs/notes/review [20:09:25] codereview = log --show-notes=review [20:10:32] interesting, what does that show? [21:21:30] jhathaway: sorry i dropped cause I was in meeting [21:21:47] no apolgies needed! [21:21:54] Gerrit stores some review informations as git notes [21:22:05] notes is a magic system in git which I am not familiar with [21:22:15] but that is a way to store metadata along with the rest of the code [21:22:37] and git log --show-notes=review would show the notes after the commit message [21:22:51] you can try it on a small repo (for puppet.git that will be A LOT of data) [21:23:16] then when showing a commit you will see who voted CR+2 for example [21:23:36] but that does not give you the review comments made on the various patches :\ [21:24:18] andrewbogott: yeah, there is, it was recently added for k8s on bullseye since the current kubelet can't deal with cgroups2 either [21:24:53] see profile::kubernetes::node (which only needs/sets systemd.unified_cgroup_hierarchy=false) [21:26:50] moritzm: I see it! https://gerrit.wikimedia.org/r/c/operations/puppet/+/762410 -- thank you [21:27:22] yw :-) [21:36:35] I noticed homer has a deploy repo on Phabricator: https://phabricator.wikimedia.org/diffusion/OSHP/ [21:36:40] is this used? [21:36:59] or has it moved to gerrit/github/gitlab/X [21:40:09] also is anyone here who has "acl*repository-admins" on Phabricator by any chance? [21:41:44] mutante: all homer repos are on gerrit the phab one is a mirror like all the rest [21:45:08] volans: ACK! thx [21:45:26] I need to separate mirrors from non-mirrors next. [21:46:03] or maybe bot, all I want so far is disable pushing via ssh, but let remaining users push via https still [21:50:08] alright, and I got the ACL for it now