[17:29:36] hola dancy ! I'm tapping in some folks from my team who are more knowledgeable than i am about this stuff [17:29:49] * dancy waves [17:30:08] This was brought in :https://wikitech.wikimedia.org/wiki/Logstash/Interface [17:30:20] which outlines the interfaces coming into logstash [17:30:51] o/ [17:33:06] Hey Cole. My current goal is to clean up the logging config in scap in beta so that its log messages end up in the right place. The following settings seem to be relevant: [17:33:06] ``` [17:33:06] use_syslog: True [17:33:06] logstash_host: logstash.svc.deployment-prep.eqiad1.wikimedia.cloud:9200 [17:33:06] udp2log_host: deployment-mwlog01.deployment-prep.eqiad1.wikimedia.cloud [17:33:06] ``` [17:33:51] I looked at https://beta-logs.wmcloud.org/ today and I don't see any scap related messages listed, even though scap sync-world runs regularly in beta. [17:34:23] Scap is logging to beta-logs AFAICT: https://beta-logs.wmcloud.org/goto/ee2a115f071d13276ce684d99e7aa0b1 [17:34:56] oh great! I just didn't know what I was doing in the web ui. [17:35:45] * cwhite breathes sigh of relief [17:37:20] I feared something was broken given we just changed a lot about that env. I'm so glad it ultimately wasn't :) [17:37:33] with the cluster in deployment-prep shut down scap is failing the error check stage: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/26342/console [17:38:32] hmm, which host runs those checks? [17:38:47] deployment-deploy01? [17:38:59] as of now, yeah [17:40:16] (it's a stretch host that needs to be replaced with buster at some point) [17:41:02] Looks like it was trying to connect to deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud (which is responsive at the moment) [17:42:03] dancy: that's a separate error, I was looking at the error after "Executing check 'Logstash Error rate for deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud'" [17:42:31] scap tries to connect to logstash.svc.deployment-prep.eqiad1.wikimedia.cloud (which points to the deployment-logstash* hosts) to query recent logs [17:43:02] ah, I see it. Got lost in the noise. [17:43:49] cwhite: https://gerrit.wikimedia.org/g/operations/puppet/+/2ad64c08006b7fbce3aa27f5195b7d3463fc9775/modules/scap/templates/scap.cfg.erb#91 needs to be changed to something that scap can use to connect to the new cluster [17:46:12] I'll need to puppetize it to keep it up, but I've manually allowed access to logging-logstash-01.logging.eqiad1.wikimedia.cloud:9200 from deployment-deploy01 [17:46:19] Give that a try? [17:48:16] https://gerrit.wikimedia.org/r/c/operations/puppet/+/737108 [17:49:47] The logstash.svc.deployment-prep record should stay the same for everything else (udp2log, etc). [17:50:34] why? it currently doesn't point into any existing VM [17:51:44] can you merge the patch too? I don't have that level of access [17:53:44] logstash.svc.deployment-prep.eqiad1.wikimedia.cloud has deployment-logstash0[456] in its answer. It's possible it has no meaning anymore though, but I don't have the answer for that offhand. [17:55:13] I shut those VMs down earlier today as I thought the cluster in logging was meant to replace them [17:56:06] The new cluster is meant to replace them, but there are still parts of the pipeline that could not be excised from deployment-prep. [17:56:47] The rsyslog->kafka portion has to remain due to the SSL overhead. [17:57:37] I *hope* that we can remove logstash.svc., but I'm not confident it won't break something. [17:58:05] With the hosts shut down, I guess we'll find out :) [18:01:03] scap and udp2log are the 2 things I see referencing it, I already made a patch for scap [18:01:53] cwhite: Regarding `logstash.svc.deployment-prep.eqiad1.wikimedia.cloud`. If it remains, what hosts should it point to? [18:02:18] and in case it's helpful, I verified that scap uses the logstash_host setting to pass to `/usr/local/bin/logstash_checker.py` [18:04:35] dancy: good question. herron, do you know if that host is still needed for the udp2log pipeline? [18:08:24] I cherry picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/737108/, it seems to work, now just needs a proper merge to ops/puppet [18:10:03] Excellent. Thanks everyone for helping get this stuff cleaned up [18:21:10] cwhite: should be mwlog host but not sure off hand, I can have a closer look in a bit if needed [18:22:51] That begets another question, is mwlog available in deployment-prep? [18:23:24] there is a deployment-mwlog01 [18:23:50] but I don't think a hiera key named "logstash_host" should be pointed to that? [18:24:41] anyhow the scap settings fix still needs to be merged to ops/puppet, could one of you with access do that please? [18:53:48] majavah: you bet, just merged