[12:39:24] moritzm: good to merge this? `Muehlenhoff: profile::java: Also add component/jdk on bullseye (c00885b446)` [12:39:46] arturo: ack,please merge along [12:39:58] thanks, merging [13:47:13] jbond: hello [13:47:24] any idea why I would be getting: [13:47:25] [ 2022-03-24T13:46:22 ] WARNING: no nodes found for class: Role::Wikidough [13:47:28] [ 2022-03-24T13:46:22 ] WARNING: No hosts found matching `O:wikidough` unable to do anything [13:47:31] :) [13:50:14] this used to work just fine and I am not sure what changed (most certainly not the role itself) [13:53:39] sukhe: let me check [14:04:49] sukhe: you where missing secrets from the private repo, this ment the job that populates the puppetdb for pcc was failing so when yu run that query it finds nothing as nothing is there. i have fixed both issues now so if you run pcc again it should work [14:06:10] jbond: thank you but I am curious, what secrets were missing? [14:06:16] oh you meant labs/private [14:06:51] yes labs proivate where missing api and password keys for profile::wikidough::webconfig [14:06:57] ahhhh [14:07:16] interesting, yeah, that did change as I was trying to clean it up once [14:07:20] thanks [14:07:30] * sukhe sends jbond some beers [14:08:08] thanks and no problem :) [15:04:56] hi all im adding some addtional checks for ssl cerificate expiries. ill monitor icinga/ops but please ping me if you see anything related [15:05:51] thanks jobo [15:05:55] er wow, fail, sorry jbond [15:06:31] n [15:06:32] np [15:08:11] <_joe_> we should just merge in a single nick [15:08:17] <_joe_> me john and joanna [15:08:42] <_joe_> at least we'd assume any ping we get could be for someone else :D [15:09:09] ~:D hehehe [15:11:35] 😄 [15:12:54] (apologies for the ping jobo :) [15:13:11] but we can all get together and thank jbond anyway [15:23:29] <_joe_> jobo: hey I hope all's going well :) [16:42:05] Can I have some puppet/systemd help, please? https://gerrit.wikimedia.org/r/c/operations/puppet/+/769941/20/modules/swift/manifests/ring_manager.pp#48 defines a systemd::timer::job resource with logging set up (I thought!), but things I log (using pythons syslog output) end up in user.log and if I instead log to stdout, that ends up in /var/log/syslog and not in /var/log/swift_ring_manager/swift_ring_manager.log. What am I doing wrong? [16:43:09] that's a very long question [16:43:35] I mean, I could just decide I don't care and that output in /var/log/syslog is fine [16:45:12] Emperor: tl;dr you will need to set up syslog_identifier on the systemd::timer::job. [16:45:16] i can comment on the cr [16:45:58] jbond: I knew I was missing something obvious, thank you! Here or there is fine (there I will not forget when my IRC client explodes) [16:48:51] ill do it there as its easier to reference later. this is a common issue, and tbh i think we shuld change the defaul behaviour but i havn't looked at all use cases yet, but many timers are currently suffer from this issue and you jusy have an empty /var/log/$timer_name/syslog.log file [16:49:18] Emperor: can you give me an example host where this runs [16:50:56] jbond: currently only running in pontoon ( so ms-fe-01.swift.eqiad1.wikimedia.cloud ), will be running on e.g. ms-fe1009.eqiad.wmnet [16:51:19] thats fine ill check the pontoon instance [16:52:01] jbond: thanks :) [I have tried setting syslog_identifier there just now] [16:57:09] Emperor: sent a comment let me know if that all make senses [17:02:42] I think I understand... [17:08:43] feel free to follow up either here, on the change or via pm with any questions :) [17:10:43] thanks; I left a question on teh change (and have to dash now, sorry, will get an updates later) [17:11:52] ack [18:17:32] I see deneb.codfw.wmnet is at 93% disk usage, mostly coming from /var/lib/docker; any objections for a `docker system prune` ? We could even run it on a systemd timer [18:19:05] ack, sgtm [18:19:26] our icinga disk space checks are already having some --exclude docker stuff [18:19:30] fwiw [18:36:36] razzi: on the CI boxes we run on a daily basis :] See the systemd::timer::job in `modules/profile/manifests/ci/docker.pp`. Might be worth adding similar timers on deneb [18:36:55] hashar: ooh good to know [18:37:41] and maybe those can be moved up to the `docker` class if we want to generalize such pruning [18:38:22] That `docker system prune` freed up 33gb. Use% is down to 78% from 93% [18:39:11] 🗑️ 💪 [18:39:14] nice [18:40:01] now I remember when we converted that to timer, yea, just copy that to the "builder" role/profile [18:50:41] 👍RECOVERY - Disk space on deneb is OK [18:50:56] cool [18:51:49] mutante: hashar, here's what I came up with https://gerrit.wikimedia.org/r/c/operations/puppet/+/773622 [18:54:00] +1 since I see it's like modules/profile/manifests/ci/docker.pp [18:54:17] so if that works it should be fine :) (re: splay option etc) [19:14:52] razzi: I thought about moving those to something like `docker:system:prune` which can then be included both from profile::ci::docker and the deneb profile [19:16:17] also `package_builder` doesn't have any reference to Docker and on the CI instances that do the package builder I don't think we even have docker installed. So probably they should moved from package_builder to a profile [19:16:23] but +1 on the intent! :] [19:17:48] true, it should be in a profile but he was just adding next to existing timers. so consistent and they can be all moved at once [19:18:06] would make that another change on top I guess [20:51:46] mutante: yup no worries :] [21:20:41] hashar: a question, if you're there; the current `systemd::timer::job { 'docker-system-prune-all':` is in a block `if $::realm == 'labs'`; should including a new `docker::prune` also use that conditional? [21:36:51] razzi: https://openstack-browser.toolforge.org/puppetclass/role::builder says the role is used in cloud and prod. but per style guide we should avoid those "realm checks" unless we see no other way. personally I don't see why we should handle it differently in this case. but hashar might have more to say why that was added [21:38:06] I would say ideally just no difference at and all but if we see a need for it then it's nicer to put the difference into hiera (something like: disable-docker-prune: true) and put that only into cloud hiera [21:38:22] (or vice versa)