[08:04:06] indeed thank you denisse for unbreaking puppet on contin [09:11:22] swfrench-wmf: alternative query you can use: 'R:package = python3-conftool' ;) [14:34:54] volans: thanks for the new spicerack release and introducing CookbookInitSuccess. very helpful! [14:39:33] <3 not yet deployed, I'm releasing it right now... are you spying on me? :D [14:42:42] he's just an avid user of irc highlight keywords [15:13:03] <_joe_> ahahahahah [15:14:11] <_joe_> jelto: FYI I'm about to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1111635 to fix importing packages for conftool [15:22:34] volans: ah, right! thank you :) [15:23:31] cdanis: :) [16:01:59] on-call: nothing to report for EU! [16:46:20] cdanis: only semi-related to my talk about 'static wikitech-static', jobo and I have ambitions to move the existing VM from rackspace to AWS next week (T304688). Are there things running there that won't survive a simple VM snapshot + restore? I'm not 100% clear on what runs on that VM besides the site itself. [16:48:59] andrewbogott: https://wikitech.wikimedia.org/wiki/Wikitech-static#What_is_wikitech-static_running? [16:49:19] oh, nice [16:49:42] for meta-monitoring check with o11y, but it needs the proper config to be able to send emails [16:50:26] for the status page I guess we'll need to change the DNS [16:50:50] yeah [16:51:28] I will start with snap+restore and then worry about dns changes after I see things working [16:51:49] volans: is the meta-monitoring your thing or Chris's? [16:59:02] I set it up long long time ago, nowadays is maintained by o11y [17:00:18] kamila_: after this batch of reimages do you think I could get a 5 minutes window to deploy the new spicerack to cumin1002? :) [17:00:47] sure volans , I can also kill the reimages and redo them after if you prefer [17:00:54] nah [17:00:56] no worries [17:01:16] I just had to restart some anyway, so won't be much of a loss... [17:03:18] volans: go for it [17:03:59] there is stilla reimage and roll-restart ongoing, doh, maybe I just picked the rush hour :D [17:04:03] not yours [17:04:16] eh :D [17:04:28] please keep going I'll check it later on [17:04:32] ok [17:04:48] don't want to disrupt any running cookbook, it shouldn't on deploy but I tend to err on the safe side [17:06:48] makes sense [17:06:57] well, if you're bored, I have some logs for you :D [17:07:05] in a meeting right now [17:07:23] I can later :) [17:07:30] ack, maybe I should write up a bug then [17:13:21] andrewbogott: yeah aside from what volans said, we'll have to move the DNS, that will probably take a little turnaround with legal I think [17:13:49] ok [17:17:41] re: above, the DNS changes for wikimediastatus.net I am assuming? [17:18:03] yes [17:18:25] ok, Traffic can help with that, please feel free to loop us in [17:18:46] there is an ongoing conversation with Legal about ownership of this but aside from that too [17:18:52] brett: ^ [17:19:04] oh interesting, sukhe do you have our Markmonitor account? [17:19:27] `dig ns wikimediastatus.net` :) that's on purpose so it's off-infra [17:19:48] cdanis: yep, I saw the NS. and not right now but we are in the process of owning that [17:19:59] very cool, hadn't heard [17:20:06] thanks! [17:20:19] but at least we can help you get in touch with our Markmonitor point of contact and all for now, and if that doesn't happen soon [17:20:50] cdanis: yeah sorry for the lack of communication, still WIP and hence [17:20:59] no worries :) [17:21:50] brett has his own reasons but when we wanted to move ns2 to anycast (and thus changing the glue records for that), there were a lot of single point of failures on that and that was one of the original conversation starters [17:22:05] brett's reasons are since he owns and maintains ncmonitor and other related stuff now [17:22:29] neat [17:23:18] this brings the meme: how many SREs do you need to change a DNS record? :D [17:23:30] depends on the record [17:23:42] lol, indeed [17:23:58] change?? [17:24:19] yeah super locked records are a thing [17:30:16] My reasons are also because there's just a lot of DNS work that needs to be done regardless of ncmonitor - Legal has too much on their plate to realistically have the time to fix up inconsistent records [17:30:43] volans: 53, natch ;) [17:44:29] brett: makes sense! [18:13:15] what is the postfix equivalent to "exim4 -bt foo@wikimedia.org"? I have used this for > 10 years to test if an email address is valid and if it's delivered to Google or elsewhere, but now exim is gone. [18:15:09] mutante: unfortunately there is no good equivalent :( [18:16:01] hmm, ok. ack. Well then it's not for right this moment.. but longer term I wonder how to debug stuff. [18:16:14] it's fairly common [18:16:28] there is something akin, which sends the result to and email, but it doesn't work that well in practice [18:16:34] *an email [18:16:41] it is a good question [18:17:07] ok, ack. noted. not super urgent but probably will get back to it at some point. thanks [18:17:26] yeah for sure [18:22:33] I can still run exim4 -bt on vrts* but that will tell me it would route whatever to mx-out, even addresses that don't exist, so doesnt have that info [18:24:55] right [18:29:54] -bt does not do actual mail delivery, just a display of how exim would route the email, right? can't we use sendmail for that? [18:30:10] the postfix "sendmail" that is, https://www.postfix.org/sendmail.1.html. -bv? [18:30:25] please disregard if this is a bad idea :] [18:41:32] I tried that after finding it on a stackexchange but there was no sendmail [18:42:17] ah, well, because I did not use sudo.. there is sendmail [18:42:32] but what this does is sends a report to .. does not tell me on the command line [18:42:59] yeah check with jesse once if this is even a good idea! [18:43:11] ack, thx [18:43:57] mutante: where is the stackexchange question equivalent of this? [18:44:19] https://superuser.com/questions/1385652/what-is-the-postfix-equivelant-of-exim-bt-testexample-com [18:44:37] oh interesting [18:44:46] what I personally don't is which sendmail is being referred to here [18:45:38] though [18:45:39] sukhe@mx-in1001:~$ dpkg -L postfix | grep -i sendmail [18:45:39] /usr/sbin/sendmail [18:45:39] /usr/share/man/man1/sendmail.1.gz [18:45:39] /usr/lib/sendmail [18:46:26] [mx-in1001:~] $ sudo which sendmail [18:46:26] /usr/sbin/sendmail [18:47:52] it does kind of work, just that it "spams" everyone on root@ and the mail ends up in spam :) [18:48:09] needs another parameter to mail that info to something not-root [18:48:11] mutante: yeah so it has to be that one, installed by postfix [18:48:23] mutante: yeah! defeats the purpose of silently testing :) [18:48:26] yep:) [18:48:28] to both of that [18:53:14] I tried "-f" to send that report to me.. based on the second answer on https://unix.stackexchange.com/questions/441651/where-is-the-postfix-mail-delivery-status-report but dont think it worked [18:54:27] I ... have another very bad idea [18:54:46] don't do it but edit /etc/aliases, root: , run above command, revert [18:54:47] temp add myself to /etc/aliases :) [18:54:51] [this is a joke] [18:54:53] lol, called it [18:54:54] haha yesss