[08:23:37] Huh, VO made me do a captcha before I could sign in [10:32:28] heads-up, CI seems to be broken for ops/puppet [10:50:29] https://phabricator.wikimedia.org/P50585 is the error [10:50:32] on contint1002 [10:50:34] trying to see if I can/should fix this [13:33:54] Someone working one something? I just acked `PROBLEM - Host cr1-esams is DOWN: PING CRITICAL - Packet loss = 100%` Though status on Netbox says "Planned" [13:34:14] I think ESAMS networking work is being done all week [13:34:16] yeah this is expected [13:35:13] Got it. Thanks. [13:35:30] Found the email too. :) [14:16:10] anyone from serviceops wants to give a quick review to https://gerrit.wikimedia.org/r/c/operations/puppet/+/949037 ? (touches profile::tlsproxy::envoy) [14:49:41] fabfur I guess we're just unlucky...I see you're reimaging dns3003 and that's blocking my makevm cookbook again ;P [14:49:56] sigh [14:50:01] :| [14:50:05] inflatador sigh is for this step [14:50:16] inflatador: you can just skip it for now [14:50:34] atm I'm not trying to reimage dns3003 [14:50:50] yeah but the host is in authdns_servers so it will be attempted [14:50:50] ah OK, I'm just going off the SAL [14:51:10] inflatador: let us know if it fails? hth [14:52:26] sukhe no, still failing [14:52:37] it's not urgent, so if y'all just wanna hit me up when done that's fine [14:54:45] inflatador: yeah it will fail but what options is it giving you? [14:54:57] can you share those here [14:57:47] sukhe pasted here, let me know if you need more context https://phabricator.wikimedia.org/P50586 [14:58:02] I think it just can't SSH into dns3003 [14:58:06] right [14:58:06] yep [14:58:09] does the skip work? [14:58:13] because if yes, that's totally fine [14:58:15] no, tried it twice [14:58:20] oh I see [14:58:25] ok [14:58:32] give me a second [14:58:40] fabfur: reverting for now [14:58:45] ok [14:58:55] then we can figure out the puppet issue [14:59:04] inflatador: let it sit there for a second, reverting and running agent to remove it [14:59:07] thanks [14:59:25] np, there's no hurry, so if you want to roll fwd, I can wait [14:59:34] no, doesn't make sense to block for everyone [14:59:43] fabfur and I are still still trying to figure out what's wrong with :) [15:04:33] sukhe understood, let me know if/when I can try the cookbook again [15:04:38] inflatador: please try now [15:16:20] inflatador: did it finish? [15:16:52] sorry, I'm in mtg. will try now [15:16:56] np [15:17:27] sukhe still failing [15:17:46] that's pretty weird [15:18:02] try "skip" now [15:18:07] it's probably failing on dns3001/2 [15:31:28] sukhe still failing, still in mtg...if you want to see it live, I've got a tmux under my user (bking) on cumin1001 ...feel free to sudo to me and give it another try if you like [15:35:36] inflatador: fine to run the makevm cookbook? [15:36:08] sukhe Y [15:36:11] ok [15:43:36] inflatador: running [15:43:37] should be fine now [15:43:45] if you get a prompt, let me know [15:46:07] sukhe thanks, it's getting much further along [16:42:50] sukhe I think we're good now, thanks again [16:43:33] sukhe...Oh wait! I have 2 more VMs to provision. Can you wait 45m or so? If not, just ping me when your DNS work is done [16:43:34] inflatador: good to know! sorry for the trouble [16:43:58] inflatador: if you can wrap up in 45m, that works [16:44:25] sukhe OK, I'll get started now. Will hit you up when I'm done [16:44:28] thanks [16:44:34] in theory we shouldn't need to wait [16:44:36] and feel free to hit me up if you need to get started earlier [16:44:48] but given the situation we are right now in keeping the old esams hosts up and the new ones, this is the best [17:06:48] sukhe looks like it's failing again. I'll give it a shot again tomorrow...feel free to finish what you need [17:06:55] sigh [17:07:06] didn't skip work? [17:07:20] sorry about that, we should be in better reliability shortly :) [17:09:12] sukhe not your fault. VM provisioning is tied to DNS updates [17:11:29] inflatador: if you are ok I could try to reimage dns3003 [17:11:34] so we don't overlap [17:17:09] I'm reimaging dns3003, dns changes *may* fail during reimaging [18:25:19] start reimaging dns3004 (dns changes may fail during this time, in case of error skip) [19:44:12] within the puppet repo, does anyone know if there's a way to skip the license header check for a single file? [19:49:17] hmm, good question [19:51:37] inflatador: I think the ignore list is hard coded [19:51:42] what do you need to ignore? [20:29:43] jhathaway it's a text file we use to federate WDQS with other hosts. https://gerrit.wikimedia.org/r/c/operations/puppet/+/949101/ has more context [20:32:59] inflatador: probably worth considering adding txt to the list of ignored extensions, rake_modules/tasks/spdx.rb [20:33:40] or some other extension that indicates the file does not necessarily support SPDX headers [20:34:33] jhathaway thanks, looks like `IGNORE_EXT_PATTERN = /\A\.(?:json|pem|key|csr|crt)\z/i` in line 6 is the place to change? [20:34:46] yup [20:35:21] cool, I'll get a patch up. Not urgent, can cc you or your team if it needs a larger discussion [20:35:54] thanks, jbond put the SPDX stuff together, so he may have an opinion about skipping txt files [20:36:08] so ccing the team would be great [20:36:58] but feel free to add it to IGNORE_FILE if it is blocking you, or ignore the CI check [20:51:26] OK, phab ticket up at https://phabricator.wikimedia.org/T344291 ...again no urgency, will use IGNORE_FILE if needed