[11:27:12] I am about to add a "manager mode" to review backups :-) https://gerrit.wikimedia.org/r/c/operations/puppet/+/714040/1/modules/mediabackup/files/readandlist.json [11:33:02] You mean something so easy even I can do it? ;) [11:34:02] not easier, but without permission for deletion, so you don't accidentaly drop 1000 files like it happend to me once [11:34:27] that way I can also use it myself without worries [11:34:36] (in theory it is an account for recoveries) [11:35:28] the integrated UI is not great [11:37:53] we will also be able to share this credentials with NDA people outside of our team, if necessary [12:33:03] sobanski: https://wikitech.wikimedia.org/wiki/Media_storage/Backups#How_to_access_the_web_UI_of_minio [12:41:15] Thanks! Testing it now. [12:42:01] wait, I made a mistake [12:42:31] the read only credentials are at /etc/mediabackup/mediabackups_recovery.conf [12:42:38] updating the wiki [12:44:28] Emperor: how is your gbp-related debian knowledge? [12:44:53] I'm more a dgit person myself, but I have used gbp a bit [12:45:54] ok cool :) so we have https://gerrit.wikimedia.org/r/operations/debs/orchestrator [12:45:59] for creating debian packages for orchestrator [12:46:16] (upstream does technically provide debian packages, but they're awful and to be shunned) [12:46:40] it's been ages since i set this up, and i can't remember how to update to a new upstream release [12:46:53] I get "Not Found" at that URL [12:47:11] * kormat sighs at gerrit [12:47:24] https://gerrit.wikimedia.org/g/operations/debs/orchestrator [12:48:19] jynus: all good, downloading (and deleting -_-) tested [12:51:00] Emperor: Oh! i just (re-)discovered `debian/repack` [12:53:05] I _think_ the usual approach is to fetch from upstream and then use gbp-pq to rebase the patch queue. BICBW. [12:53:21] (I'm quite allergic to using quilt for anything) [12:53:45] cf https://wiki.debian.org/PackagingWithGit#Using_the_upstream_repo-1 [12:54:26] or https://honk.sigxcpu.org/piki/development/debian_packages_in_git/ [12:55:43] oh, yes, I see there's a local repack that maybe just does the work for you [12:56:22] it certainly does _something_ [12:56:26] it created a tarball, which i like [12:57:03] oh, wow [12:57:13] `gbp import-orig ../orchestrator_3.2.6.orig.tar.xz` seems to do all the right things [12:57:26] \o/ [12:57:36] (hence i don't trust it) [12:59:57] The thing I like about dgit is that you just do everything with git operations (including uploads, which are done with dgit push-source) [13:29:18] i'm beginning to hate this: `dpkg-source: error: unrepresentable changes to source` [13:37:48] ok, jumping from 3.2.3 to 3.2.6 was probably a mistake. let's try doing it in one-version increments [13:54:06] i'm declaring defeat [13:54:19] Emperor: i'll get you to try rescue me next week :) [14:06:53] * Emperor thinks they have got systemd to do their will [14:07:40] Emperor: 🤞 [14:08:05] 😨 [14:09:27] on slave1 in kormat's test env prometheus-mysqld-exporter runs only when mariadb is running, and is restarted when mariadb is restarted. So on system boot neither is started; and when you start mariadb, the exporter is also fired up [14:10:02] Emperor: where can i look to see your eldritch config changes? [14:10:48] i'm guessing /etc/systemd/system/mariadb.service.d/depends.conf ? [14:11:37] and _also /etc/systemd/system/prometheus-mysqld-exporter.service.d/depends.conf [14:11:40] you need both [14:12:00] gotcha :) [14:13:47] briefly, because mariadb isn't Enabled, you need the Wants= there so that when you start it, it tries to start the prometheus exporter (but doesn't care if the latter fails); but you need the Requisite= in the prometheus export unit so that when/if mariadb is restarted that also restarts the prometheus exporter [14:14:32] 💡 [14:14:49] what happens if you just stop mariadb (instead of restarting it)? [14:15:08] that will also stop the prometheus exporter [14:15:14] ok cool [14:15:32] I don't think you can arrange for the restart to propagate but a stop not to [14:17:20] similar tricks can probably be achieved on the multi-port setup (but there isn't one in the test env) [14:17:32] and I should now try and puppet up the change properly [14:19:41] how is your git-fu? :) [14:20:41] I like to say that git is reasonable to understand as long as you remember that it is simply a directed acyclic graph ;-) [14:30:21] kormat: how would you like (once I've worked out the puppetry) something to review here? [14:30:39] [this can wait 'til next week, I can make the changes and work out what branch to stick them on later] [14:31:11] Emperor: if you put them on the shared branch (sandbox/kormat/pontoon-mariadb104-test), i can do an informal review, [14:31:25] with a proper review when you're making the changes to the puppet 'production' branch [14:33:13] 👍 [16:27:00] How is require_package('foo') distinct from package{ 'foo': ensure => installed, name => 'foo', } ? Asking (probably for Monday now) because I want to add a notify to a package install [16:27:30] So I think I want to replace a require_package('foo') with a package resource. [16:28:06] [sorry, not used puppet before, kindof feeling my way here] [16:28:44] we have a few macros that do exactly what we want [16:28:56] for example, package resources can only be used once [16:29:17] which is a pain if you use multiple profiles taht use the same package requirements [16:29:43] so you don't get duplicate package definition [16:32:20] Emperor, you want to use ensure_packages() [16:33:04] that's what I was told most recently by the maintainers 0:-) [16:33:32] jynus: can I hang a notify off ensure_packages()? [16:34:19] I this this is what you want: https://wikitech.wikimedia.org/wiki/Puppet_coding#Different_ways_to_install_apt_packages_with_puppet [16:34:41] package is not forbidden, it is just a wrapper [16:36:04] I _think_ I can add a notify via the third arg of ensure_packages [16:36:50] jynus: your link doesn't mention require_package at all... [16:37:11] yes, that is now deprecated, and why I didn't suggest it :-D [16:37:54] sigh, modules/wmflib/README.md neglects to mention that detail. Anyway, this can keep for Monday [16:38:50] but thanks :) [16:39:11] honestly, I had to check the docs, because there are multiple ways to do things [16:39:29] but as an SRE you cannot just delete everything at once [16:39:56] so that is why I ask for reviews for the people that new about this could help me :-) [16:40:05] *knew [16:41:04] I think require_package comes from when ensure_packages wasn't available on a previous version of puppet [16:41:27] Maybe I will put in a CR to add a "DEPRECATED" note to the README :) [16:42:00] context is https://phabricator.wikimedia.org/T266479 [16:42:14] it is not closed, so what we can do is do that patch, and propose it :-) [16:46:08] Emperor, https://gerrit.wikimedia.org/r/c/operations/puppet/+/714079 [17:27:32] TY