[18:43:01] godog: i looked at the instructions you linked yesterday (https://wikitech.wikimedia.org/wiki/Logstash/Interface#Configuring_rsyslog_to_forward_your_logs) and have some potentially dumb questions . [18:43:26] Since my app is a node js application which by default logs to stdout, i would need to write those logs to a file and tail that file correct? And additionally, I would need to run my node app as a systemd service in that case yes? [19:01:12] nikkinikk: if your app is a systemd service and logs to stdout, the logs will be in the right position to forward to logstash. There is one additional change on our end to tell rsyslog to actually do the log forwarding, but that change is super easy. [19:52:38] cwhite: ok cool- just making sure it should be a systemd service. right now i am running it with `screen` 😬 haha [19:53:11] thanks! [19:55:29] Happy to help. Please don't hesitate to ask if anything else comes to mind. :) [21:13:23] cwhite: ok one more question...haha [21:13:33] was this the file that you said would need changing too? https://github.com/wikimedia/puppet/blob/production/modules/profile/files/rsyslog/lookup_table_output.json [21:13:48] yep, that's the one [21:25:58] how do these entries correspond to the file that the logs are written to? for example, what would the entry be for a systemd service for a node application that is running on a cloud vps instance? just want to understand the flow of it all [21:30:53] They don't coorespond to a file. They are the systemd unit name. [21:31:46] the "value" allows us to selectively enable certain features in the rsyslog processing pipeline. [21:33:54] its the unit name ok gotcha! thanks! [21:46:04] cwhite: and is there a way to forward logs to staging logstash? i ask because i'm not familiar with this setup and want to just see how the logs will show up without necessarily needing them in logstash. But if theres no harm in experimenting in production i can also do that : ) [21:48:55] nikkinikk: I'm not aware of a staging logstash. Do you mean deployment-prep? [21:50:06] probably...haha. i was looking at this: https://logstash-beta.wmcloud.org/app/dashboards#/list [21:59:02] nikkinikk: yep, that's deployment-prep. You can change that lookup_table_output.json on the host in deployment-prep your code is running on and it should forward the logs. Note that the cluster in deployment-prep is notoriously flaky. [22:00:23] oh, and rsyslog needs restarted after changing lookup_table_output.json [22:10:05] got it ok, i misunderstood how this is setup so my host is not in deployment-prep its in its own project, so that probably wont work then im assuming! [22:14:16] yeah, there's no centralized cloud logging I don't think [22:15:54] ah ok thanks for clearing that up !!