[09:10:04] dcaro: re T398780, none of the current workloads are using basically any CPU: https://grafana.wmcloud.org/d/TJuKfnt4z/kubernetes-namespace?orgId=1&from=now-24h&to=now&timezone=browser&var-cluster=P8433460076D33992&var-namespace=tool-toc&viewPanel=panel-1 https://phabricator.wikimedia.org/P78765 [09:10:04] T398780: Request increased build quota for toc Toolforge tool - https://phabricator.wikimedia.org/T398780 [09:11:12] although their jobs.yaml only overrides mem, and uses the default CPU setting https://github.com/kanasimi/wikibot/blob/master/wikitech/toolforge-jobs-toc.yml [09:11:25] huh [09:14:32] that kinda fits the graph yep, I'll as to try setting the cpu limit higher [09:14:38] tanks [10:59:02] seems like PAWS flapped but then came back up [11:19:19] Was it the web workers? [11:19:33] (asking if you checked) [11:20:18] didn't check [12:41:35] ack [14:58:13] dcaro: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1166852 [14:58:28] 👀 [15:02:43] I think it would delete also /var/lib/ceph :/, looking [15:04:42] the patch looks good though [15:05:05] I think it would too, I wonder if maybe ben is only using the boss_ recipes which work differently... [15:05:36] maybe it gets repopulated by their ceph puppet modules or something [15:05:44] (we use different puppet stuff) [15:06:37] well, actually... if it leaves lvm then maybe it won't repartition anything? [15:08:17] I think they might be using ceph-volume * thingies to activate the osd if it can find the lvm [15:09:16] I think it's this stuff https://gerrit.wikimedia.org/g/operations/puppet/+/8086043d77b6e3cd620cc65e548ad5641efa7262/modules/ceph/manifests/osd.pp#103 [15:10:00] we might have to do some of that to the cookbook to get the osd up and running after reimage [15:13:14] yeah [15:13:18] looks like Ben is out today [15:13:44] so if we were to run the existing bootstrap cookbook, what would it do? Nothing, I guess? [15:16:56] let me check [15:18:23] I think it will just ignore those osds, it does a check for partitions (that they will not pass), and then just ignore them [15:18:37] ` return _is_disk() and _does_not_have_partitions() and _its_not_mounted()` [15:18:47] only the ones that pass that are then bootstrapped [15:19:55] I think we don't run `ceph-volume lvm prepare` anywhere, just `create` [16:18:15] andrewbogott: I have a feature request idea for Magnum and I'm trying to guess the best way to report it upstream. Is a Launchpad task generally ok for requesting a net new thing, or are people supposed to write up a full spec blueprint? [16:19:25] I think magnum maybe uses storyboard? But I would start with filling a task rather than writing a blueprint [16:23:18] Lets see if I can make one that is understandable :) [16:25:43] https://docs.openstack.org/magnum/latest/contributor/contributing.html still points to Launchpad so I guess I will start there [16:48:44] https://bugs.launchpad.net/magnum/+bug/2116114 [16:48:54] Why does tools.libraryupgrader@ seem to get so much spam? [16:56:27] Reedy: people scraping emails from Git repos, I'd guess. or it's ended up on some list [16:58:05] The address is in logs like https://libup.wmcloud.org/logs2/1992838 that I am sure have been sucked up by spam bots [16:58:25] `$ git config user.email tools.libraryupgrader@tools.wmflabs.org` [16:59:31] that address being in git history would also be a likely spam vector as t.aavi suggests [17:08:03] * dcaro off [17:08:06] cya tomorrow!