[00:29:09] taavi: have you already run the script manually to make sure it works correctly? [06:35:16] puppet has been failing on alert1001 for the last 13h [06:36:00] _joe_: could this be related to: https://gerrit.wikimedia.org/r/c/operations/puppet/+/746801 ? [06:41:42] <_joe_> marostegui: uhm I doubt it, I did re-run puppet after fixing the function [06:41:47] <_joe_> but let me check [06:42:11] <_joe_> yes I can confirm I did run puppet successfully after my fix [06:44:12] <_joe_> so unless someone reverted it, it means there is one check I'm missing and someone added new data that breaks the manifest there [06:44:19] _joe_: I was asking cause it seems that modules/service/manifests/monitor.pp is the one failing and the timing sort of matches [06:44:46] But your change to monitor.pp seems pretty straightforward indeed [06:44:55] <_joe_> 86f53c5357063f1d6714f74a2c7057e5921432eb broke it again [06:46:14] should we revert that and let godog fix it later? [06:46:30] <_joe_> no, let me fix it [06:46:35] oki! [06:46:37] <_joe_> I think I know what the problem is [07:09:09] _joe_: your change fixed it, and puppet runs fine now! [07:09:26] <_joe_> 💪 [07:09:27] Going to try the reimage now then [08:35:47] thanks for the fix _joe_ ! [09:41:34] legoktm: yes [14:51:41] I'm trying to build and upload cergen, with this DIST=buster CERGEN=yes sudo -E cowbuilder --debbuildopts -sa --build ../cergen_0.2.6-1.dsc [14:51:45] though getting an error ATM [14:51:55] Err:2 http://apt.wikimedia.org/wikimedia buster-wikimedia InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9D392D3FFADF18FB [14:53:26] elukey ottomata ^ have you seen this before ? [14:54:09] nope! never built cergen :( [14:55:08] ok thanks [14:57:46] the solution: pass WIKIMEDIA=yes too [15:09:34] hmmm [15:09:37] oh ! good [15:10:34] the onboarding thing I read recently suggested `pdebuild` for building .debs... [15:11:43] Emperor: i wouldn't trust any of them onboarding docs, buddies, etc [15:12:44] everyone knows that git-buildpackage is the way to go, come on [15:15:04] ema: ITYM "dgit sbuild" ;-p [15:19:31] Emperor: wondering if the synopsis of sbuild(1) enters the Guiness world record and if so under which category exactly :) [15:19:40] Emperor: the documented way and (https://wikitech.wikimedia.org/wiki/Debian_Packaging) the method described in the onboarding docs is to use pdbuilder. however we have a few debian developers here (im not one of them so please dont ask me to expand ;)) who have there own work flows as such other tools and workflows will work on deneb and may even be documented [15:19:45] (https://wikitech.wikimedia.org/wiki/Git-buildpackage) :) [15:20:35] yes pdebuild is the way [15:22:04] ema: :) [15:22:19] * Emperor is a big dgit convert, but not about to start imposing it here :) [15:23:08] assuming, of course, that pdebuild can do what you need. *looks sadly at orchestrator package building* [15:23:32] kormat: a certain amount of weeping is an important part of building .debs [15:23:46] (https://wikitech.wikimedia.org/wiki/Orchestrator#Building_orchestrator_packages) [15:28:05] hnowlan: I'd like to update kartotherian's cergen-issued certificate for T297604, any objections and/or things to be aware of ? [15:28:06] T297604: cergen should include the cert's name in SAN too - https://phabricator.wikimedia.org/T297604 [15:29:29] godog: I can't think of a good reason not to... will ping the developers to let them know [15:30:45] hnowlan: ok! thanks, I'll do it now if that's ok ? [15:31:33] I'm ok to wait too btw, it isn't urgent [15:31:50] godog: I'd say go for it [15:32:36] ok! will do [16:01:26] hnowlan: all done, looks good on my end and I'm not seeing anything on fire [16:01:34] godog: great, thanks! [16:02:09] thanks ! [16:04:40] heads up: gonna try an in-place reimage of a restbase host. Cassandra instance data will be unaffected but the OS will be going from stretch to buster. Signs point to it being a safe operation given attempting it on a host removed from the cassandra cluster but who knows [16:12:43] <_joe_> hnowlan: godspeed! [16:15:10] possibly some problems with its mgmt interface, not a great start ;_; [16:20:55] n00b here. I'm working on https://phabricator.wikimedia.org/T298252 and it refers to the "cfssl-based automated pki system" . Is anyone using that in the deployment-prep area already? [16:31:03] ^^ taavi looks like you filed this one, do you know? [16:35:22] <_joe_> inflatador: pretty sure taavi is indeed [16:38:43] I'm happy to be the first if the rest of my team approves, and we'll probably need some hand-holding ;) [16:39:05] there should be a test pki intermediate in deployment-prep [16:47:13] inflatador: yeah that's me, I'll reply on -search [17:02:45] if there are any pandoc users around, I found this handy lua script for writing phabricator's markup language, https://github.com/aaron11496/github-wiki-to-phriction [17:02:59] it's not flawless, but it does a servicable job [17:11:26] <_joe_> jhathaway: the wiki way would be to translate markdown to wikitext, then have restbase call mediawiki to get the html, then have another service convert the html to remarkup [17:11:46] <_joe_> ... and serve that via restbase as well [17:11:57] :) [17:12:30] remarkup doesn't seem to get a lot of love [17:13:21] <_joe_> phabricator too, the company that was developing it folded :/ [17:14:36] unfortunate indeed [17:16:02] <_joe_> FWIW I like phabricator as a task management system [17:17:34] no opinions yet, but its pretty easy to beat Jira, which I used at my last job [17:18:19] <_joe_> heheheh that's what I was comparing it to as well [17:18:50] <_joe_> my main grievance with jira, besides being a big ball of proprietary java, is that it's tuned for enabling bureaucratization [17:21:39] yeah, it always seemed too complex to me, and each org tends to tweak it with strange workflow methods [17:22:04] <_joe_> OTOH, it would allow us to create a wizard for access requests [17:22:17] <_joe_> which are a pain point for our team [17:23:30] true, access requests are nettlesome indeed [17:26:36] looks like something is broken on pki2001/pki1001, debmonitor setup is failing on new hosts [17:29:56] jbond: possibly coincides with your puppet changes at ~15:15 [18:05:34] _joe_: > my main grievance with jira, besides being a big ball of proprietary java, is that it's tuned for enabling bureaucratization [18:05:42] yup that's the problem I see at every org that uses jira [18:05:51] (including my previous job) [18:06:27] All Jira implementations trend asymptomatically towards becoming a buggy/poorly implemented discrete finite automata [18:51:37] jbond: I think this change may be causing puppet tests in CI to fail: https://gerrit.wikimedia.org/r/c/operations/puppet/+/754904 (https://integration.wikimedia.org/ci/job/operations-puppet-tests-buster-docker/38024/console) [20:27:07] I'm seeing some "unable to connect" errors when running a homer diff for https://gerrit.wikimedia.org/r/c/operations/homer/public/+/754028 https://phabricator.wikimedia.org/P18785 [20:51:41] I'll leave that patch as-is for now but it can be reverted if need be [20:52:12] stepping away for a bit [21:26:38] !log rebooting mx1001, to test new kernel [21:26:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:11:13] !log rebooting mx1001 to revert to the old kernel [23:11:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log