[14:37:47] hey cloud services! is there a way to configure logging from a toolforge tool to logstash? was asking in #wikimedia-observability and they said you would likely have the better answer. [14:50:07] nikkinikk_: I don't think there is. In particular, if you mean logstash.wikimedia.org, that's considered a largely different realm from things that run in the cloud. [15:06:59] logging to production logstash would almost certainly end up with a "can't get there from here". and there isn't an equivalent on the WMCS side [15:07:13] andrewbogott: ok! thanks for the info. yes i did mean logstash.wikimedia.org. Was wondering if there was a standard way to get logs from a toolforge tool to logstash (although now I've just learned we dont directly send them to logstash anymore but send them through rsyslog) [15:07:42] AntiComposite: cool cool thanks for confirming that theres no WMCS equivalent [15:22:58] nikkinikk: unfortunately the task (T97861) for log aggregation in Toolforge is 6 years old and still something that we do not have a real plan for providing. Multi-tenant log aggregation turns out to be a problem that does not have many FOSS solutions. [15:22:59] T97861: Provide centralized logging (logstash) for Toolforge - https://phabricator.wikimedia.org/T97861 [15:37:24] bd808: ah thanks for linking that ticket! this might be a stupid question but since toolforge runs in k8s, and we have logging from k8s to logstash "for free", is there no way to get per-pod logging for tool pods? [15:39:04] nikkinikk: that "for free" is in a different kubernetes cluster, sending data to a different security realm than Toolforge can access directly, and not multi-tenant (everyone with access can see all logs). [15:39:33] its is a similar, but different, problem that is solved in the production environment [15:40:40] I thought this would be an easy problem to solve around the same time that valhalla wrote the ticket. I was wrong. :) [15:45:22] There are multi-tenant bits that can be bolted on to an ELK stack, but I don't know if anyone has really looked deeply into them here. The official solution for this is non-free add-ons from Elastic. I think there is a FOSS attempt to replace that plugin, but it may be one of the packages that is also under copyright dispute by Elastic. It's all a bit messy. [15:48:28] thanks bd808 for the background [15:48:58] * nikkinikk goes to read about multi-tenancy in an elk stack for background [15:50:42] nikkinikk: the main thing we don't know how to handle for Toolforge is controlling access to the aggregated logs. We would not want a maintainer of "my-cool-tool-a" to be able to see the logs from "someones-other-tool" because the logs may contain passwords or other sensitive information. [15:54:32] hmm, indeed that sounds like the exact use case for elastic's non-free security features. Not sure what happened with search guard (oss security), only thing i'm finding is the 2019 announcement from elastic that they filed a lawsuit [15:56:58] bd808: ok ok that makes sense. Is that an issue given the nature of toolforge being used by community members not necessarily within the foundation? Since that doesnt seem to be an issue for wmf hosted services and tools right [15:59:23] nikkinikk: correct. Toolforge is a "low trust" environment where users (tenants) need to be isolated from each other. Our production environment at the Foundation is a higher trust environment, but we actually still do not send all system logs to the ELK cluster because of similar concerns (passwords and other secrets from the system level in logs) [15:59:44] makes sense! [16:01:20] T213902 is a task for the production ELK stack to unblock things like T1 [16:01:20] T213902: Implement sensitive logstash access control - https://phabricator.wikimedia.org/T213902 [16:01:21] T1: Get puppet runs into logstash - https://phabricator.wikimedia.org/T1 [16:12:56] ebernhardson: https://git.floragunn.com/search-guard/search-guard-suite is still out there. The similar thing from Amazon is archived on GitHub with no real explanation of why but I think it was copyright suite related. [16:16:57] bd808: ahh, i hadn't seen that amazon archived their copy. yea seems that's all still not clear then [16:21:40] * AntiComposite grumbles about pywikibot-oauth deciding to switch to verbose logging [16:22:32] to debug logging after I had set it to info, I mean [16:25:38] AntiComposite: T272088 or something else? [16:25:39] T272088: Logging (pywiki module) always verbose if enabled - https://phabricator.wikimedia.org/T272088 [16:26:19] heay pretty much that [16:26:27] *yeah [16:38:08] thankfully it hasn't been a problem for me on toolforge, but I have a local CLI tool that just started spewing debug logs everywhere [16:38:52] and...it looks like my smtp error handler broke. yay. [20:19:14] !log clouddb-services attempting to resync OSMDB back to Feb 21st 2019 T285668 [20:19:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [20:19:19] T285668: tiles.wmflabs.org OSM is outdated - https://phabricator.wikimedia.org/T285668 [20:50:12] !log mailman repoint lists.wmcloud.org at 185.15.56.43 (mailman03) [20:50:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mailman/SAL [20:55:40] oh, certbot needs fixing [20:56:27] !log mailman deleting mailman02 [20:56:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mailman/SAL [21:06:06] out of curiosity, I've seen references in Phab to a Maxmind DB - is that WMF-internal only, or is that something we can request access to for, say, toolforge? (not sure if there's a better place to ask this) [21:23:00] GenNotability: the paid license maxmind db is Foundation only as far as I know. I think there is some in-progress work by the Anti-Harrassment team to try and make some method for it to be used by CUs and others with need through the wikis. [21:23:19] bd808: all right, thanks :) [21:26:25] MaxMind really doesn't want anyone to use their DB as a service without giving them unspecified amounts of money -- https://www.maxmind.com/en/site-license-overview [21:31:43] yup, I saw that when looking into it myself - using GeoLite for my tool [21:35:34] GenNotability: to expand on what b.d808 said, the paid databases are proprietary so they're limited to use inside the production/analytics systems. AIUI https://phabricator.wikimedia.org/project/view/4923/ is the current project to expose it to wiki users, unclear what the terms on that will end up being and what the timeline is [21:36:19] got it [21:37:45] I sure hope they figure that out *before* ip masking... [21:38:00] * GenNotability subscribes to the project [21:43:50] AntiComposite: If you didn't see Gnom's talk about the IP masking stuff during the Wikimania unconference, you might feel better about the changes being proposed if you check out the slides -- https://wikimania.wikimedia.org/w/index.php?title=File:Wikimania_2021_IP_masking.pdf&page=6 [21:45:14] oh, I'm fine with the idea. I just recognize that it's the WMF implementing it [21:50:00] I've been in camp "build the tools first, then we'll talk about ip masking" since the first proposal [21:50:35] Too bad the folks who wrote the GDPR didn't ask us :)