[12:50:56] hey observability team! Does anyone know if logstash1003 is still the host to be sending logs to? [13:31:20] actually - maybe a better first question would be, are services able to send logs directly to logstash without being on k8s? [13:36:02] nikkinikk_: hi, the tl;dr to the second question is "yes", it is enough to log to the local syslog and then tell puppet to send logs to logstash for your service. Could you give us more context on what you're trying to achieve? [13:39:02] godog: for sure! Platform Engineering Team is in the process of defining best practices for api development, and wanted to define, document, and show a real example of best logging practices. I understand that you get logging "for free" if your service is in K8s, but for those that want to exist in cloudVPS or toolforge, that maybe want to log to the staging logstash instance, would there be a way to do that? [13:39:47] for example, an experimental API or something not yet productionized. [13:42:02] nikkinikk_: that's correct yes re: logging on k8s, for toolforge my recommendation is to check with WMCS folks though, for cloudvps I don't think there's a general solution, for beta/deployment-prep though the same production method I mentioned will work [13:42:11] looking for the production documentation [13:43:23] we don't have anything managed for cloudvps/toolforge yet, it's in the very long todo pile [13:43:30] nikkinikk_: for production (and beta/deployment-prep) this outlines what needs to happen https://wikitech.wikimedia.org/wiki/Logstash/Interface#Configuring_rsyslog_to_forward_your_logs [13:44:05] majavah: *nod* thank you, that's helpful to know for my own education too [13:45:52] godog: awesome thanks!! i may come back with some questions if i have any issues with it if thats ok. but one more question, so our nodejs scaffolding has a default configuration included with it, which defines `logstash1003` as the place to send logs to. [13:45:53] https://github.com/wikimedia/service-template-node/blob/master/config.prod.yaml#L11 [13:46:14] is `logstash1003` relevant in any way, or should this be modified? [13:47:03] sounds like this needs to send logs to kafka? [13:47:14] nikkinikk_: for sure! yeah logstash1003 is long gone and so is support for gelf, yeah pretty sure that needs changing [13:47:36] gotcha - ok [13:47:42] thank you!! [13:48:01] godog: hopefully some day, although you might be interested in my cloudvps wide prometheus project too: https://wikitech.wikimedia.org/wiki/Nova_Resource:Metricsinfra/Documentation [13:48:02] looking for the relevant tasks, we did the migration in production a while ago (cc cwhite) [13:50:00] nikkinikk_: this a sample task of what happened in production https://phabricator.wikimedia.org/T219919 [13:50:30] majavah: thank you that's interesting [13:51:08] godog: this is really helpful thank you : ) [13:52:08] for sure