[03:47:47] /help@wikilinksbot [08:01:27] bd808: sure, I will, but let me make my question a bit more precise: will "saper", the usual toolserver user with id 1288, be able to set up replication on the MySQL ToolsDB? [08:01:55] s/on/towards/ [10:45:34] !log tools bump istio traffic percentage 50% -> 75% T392356 [10:45:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:45:39] T392356: Replace ingress-nginx before upstream EOL date - https://phabricator.wikimedia.org/T392356 [14:47:38] do Python tools using the build service not have access logs anymore? [14:48:04] I thought the logs just moved from uwsgi.log (NFS, oh no!) to kubectl logs, but `kubectl logs lexeme-forms-78bd5f7478-fgntm | less` seems to contain only warning-level messages 🤔 [14:50:49] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Building_container_images/My_first_Buildpack_Python_tool doesn’t seem to have a section similar to https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool#Notes / https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Python#Logs … [14:51:56] (I’m aware of https://grafana.wmcloud.org/d/TJuKfnt4z/tool-dashboard?var-namespace=tool-lexeme-forms but I’m interested in accesses to a specific endpoint / path, not just general request numbers) [15:02:48] !log admin deploy per-service oslo.messaging shared memory file name - T423378 [15:02:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:02:57] T423378: Do not share shm files for openstack uwsgi processes - https://phabricator.wikimedia.org/T423378 [15:11:53] saper: No, the user accounts do not have admin rights. I'm not sure that connecting "external" databases to toolsdb would be an obviously good idea from a system stability point of view either. But I would guess we can think of some way to make the data more available if Magnus agrees that it could be shared and is willing to work with folks on making it happen. [15:15:31] This might be a use case that Trove (our DBaaS) could help with. Managing user accounts in Trove is a manual process though. It is an interesting general thing to think about: how to share a partial view of a tool specific database more widely. [15:15:55] bd808: yeah I am looking for ideas how to do it "the right way". I wouldn't be surprised if we have more cases like this (outreachdashboard is the next one on my list). Not an expert on MySQL replication, but maybe the "ToolsDB" database could operate independently from the "external" one. [15:18:55] The place to start is probably describing the utility you want (being able to lookup things if the tool is down?) and getting agreement from Magnus that it should be possible. Then we can maybe get some folks with deeper domain knowledge about the datastore to help think of ways it could work. [15:20:08] It might even be lower tech possible for the tool itself to write the data that can/should be public into a different system (like a schema on toolsdb) as a "simple" fix. [15:26:11] sure, will write Magnus anyway. I am afraid I don't know what a "deeper domain knowledge" might be. Petscan is 2 simple tables with no private data. [15:27:15] outreachdatabase is more complicated (might have some private data), but I am in contact with Sage Ross about it and it seems might be able to weed the private stuff out if needed. [15:28:14] thanks a lot, I actually wanted to know if something like this has been attempted before - and I guess the answer is "no", thank you very much! [15:37:49] saper: In think case I was thinking about folks who know a lot about how replication works with mariadb as the subject matter experts that would be needed at some point. Making toolsdb a less stable service generally would be bad. I really don't know enough about the possibilities to suggest what is practical. [15:43:32] understood. sure, we can mysqldump and publish SQL dumps over http as a starter. [19:34:24] answering my question from earlier: I probably need to run gunicorn with `--access-logfile=-` to log to stdout (⇒ kubectl logs) [19:35:38] let us know! (re @lucaswerkmeister: answering my question from earlier: I probably need to run gunicorn with --access-logfile=- to log to stdout (⇒ kubectl logs)) [19:36:40] or wait for, I think, T97861 (but I’m not 100% sure that’s the right task, there might be another one I’m half-remembering) [19:39:53] well, I already tried it in `docker run --rm -it --entrypoint=bash tools-harbor.wmcloud.org/tool-lexeme-forms/tool-lexeme-forms:latest` (run `/cnb/process/web --access-logfile=-`, background that, then `curl http://localhost:8000/x` and see the access log line on stdout interleaved with the curl output) (re @jeremy_b: let us know!) [19:40:20] so I’m fairly confident it would work, I just need to do it in the actual Procfile (and decide if I want this in just one tool, or all tools, or if it’s a bad idea after all, …) [19:40:31] (all *my tools, obviously :P) [19:41:20] (the ability to run the exact same image locally is one of the coolest parts of buildservice tools IMHO… as long as they don’t need NFS, of course ^^) [19:42:25] codesearch shows several tools already do this, at least: https://codesearch.wmcloud.org/search/?q=--access-logfile.- [19:42:34] so that increases the chance it’s not an inherently stupid idea ;) [19:43:02] (I don’t know what storage limits, if any, kubectl logs have. do they get rotated if the pod runs for long enough without being restarted? should I worry about this at all?) [19:43:46] (I guess I relatively recently restart at least some of my tools anyway – the translatable ones get l10n updates every few weeks or so. the non-translatable ones can run for much longer, so maybe I should give the translatable ones access logs first 🤔) [19:44:21] (either s/recently/regularly/ or s/recently/frequently/ – not sure what I *meant* to type there but “recently” is definitely not right :D) [19:56:05] hm, I noticed two more things: [19:56:06] 1. I still have an uwsgi.ini file despite no longer using uwsgi, oops https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/blob/fe623b17c0/uwsgi.ini [19:56:08] 2. if I enable access logs I definitely want an equivalent of `route = ^/healthz donotlog:` (don’t flood the access log with health check requests every X seconds), but I don’t immediately see an equivalent in https://gunicorn.org/reference/settings/ :/ [19:57:36] !log lokal-profil@tools-bastion-15 tools.heritage Deploy 3df98c8 [19:57:36] https://stackoverflow.com/a/59393519/1420237 sounds like it’s possible to do but a bit more tedious, meh [19:57:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [19:59:53] maybe the reverse problem: I think maybe you lose everything on every restart and maybe you want to keep some. (re @lucaswerkmeister: (I don’t know what storage limits, if any, kubectl logs have. do they get rotated if the pod runs for long enough without being ...) [20:00:24] no, I’m fine with having the restart as an automatic log rotation mechanism ;) [20:00:35] but wondering what happens if I don’t do a restart for months [20:01:09] the point was truncation vs rotation. but idk exactly how it works [20:01:41] and does it count for a quota [20:36:40] !log lokal-profil@tools-bastion-15 tools.heritage Deploy 244e67c [20:36:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL