[00:33:44] My service check-references.toolforge.org often returns 500 errors when my script hits it; is there any way to increase the worker count so it doesn't error out as often? [00:33:54] (Assuming that is the problem) [00:57:28] harej: webservice --replicas=... start [01:07:21] What is the default number and what would you recommend? [01:29:54] Default is one. Two would likely be plenty. But a 500 error is the app crashing, not a capacity issue generally. [02:27:40] Indeed [02:27:42] tools.check-references@tools-sgebastion-10:~$ webservice --backend=kubernetes shell [02:27:42] tools.check-references@shell-1683253564:~$ source www/python/venv/bin/activate [02:27:43] (venv) tools.check-references@shell-1683253564:~$ pip install --upgrade pip wheel [02:27:45] bash: pip: command not found [02:27:46] What's happening here [06:27:23] (venv) tools.check-references@shell-1683267944:~$ ./www/python/venv/bin/pip [06:27:23] bash: ./www/python/venv/bin/pip: /mnt/nfs/labstore-secondary-tools-project/check-references/www/python/venv/bin/python3: bad interpreter: No such file or directory [06:28:55] @harej: I'm actually not sure why it's unhappy, hmm. But recreating the virtualenv would be the first thing I'd suggest trying [06:33:17] most often when I see that error message, the python version in the container does not match what the venv was created with [06:43:40] I bumped T335009 btw, grid seems stuck again [06:43:41] T335009: Toolforge grid seems overloaded - https://phabricator.wikimedia.org/T335009 [06:49:38] !log tools.poty-stuff Updated from 6abe40c to b914286 [06:49:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.poty-stuff/SAL [07:25:06] !log tools.poty-stuff Updated from b914286 to 6730d38 [07:25:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.poty-stuff/SAL [07:51:42] !log tools.poty-stuff Updated from 6730d38 to 47762de [07:51:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.poty-stuff/SAL [08:11:37] legoktm: thanks! looking [09:13:19] !log tools rebooted tools-sgeexec-10-16 as it was stuck (T335009) [09:13:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:13:22] T335009: Toolforge grid seems overloaded - https://phabricator.wikimedia.org/T335009 [12:13:26] !log tools.lexeme-forms deployed 72a006c6ea (l10n updates: mrh, ta) [12:13:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [14:25:35] Where can I find usage stats for Wikimedia Cloud services (preferably by service)? [14:27:03] Guest30: this might be a good starting point https://wikitech.wikimedia.org/wiki/Main_Page [14:27:29] with a quick overview of what is each here: https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_introduction [14:28:56] dcaro I do not see anything about usage stats there. [14:29:16] Oh sorry, did not read the stats part [14:32:28] A sample question of the kind I am interested in is how many users does PAWS have on a monthly basis, and how has that changed over time? [14:35:23] I'm not sure where to find the exacts stats for paws, for cloudvps, you can find the list of projects here: https://openstack-browser.toolforge.org/project/, for toolforge the list of tools here: https://toolsadmin.wikimedia.org/ [14:35:32] let me see if I find a graph with historical data [14:35:32] I suspect this would be a more involved question than may be expected. Each project would require its own usage stats collection method, as they are connected to and used differently. I could provide how many unique users PAWS has had since the last rebuild or the like, but it is not collected regularly. Guest30 what are you looking to do with this information? [14:35:41] Similar questions could be asked about other services, or we could drill down into specific ones. To stick with PAWS, we could look, for instance, at how many users/ notebooks use R or Python. [14:36:47] I would argue that how many users use R or Python on paws is not a very easy question to answer. How is "use" defined? [14:38:18] Rook the notebook defines the kernel, so that information could be captured. "Use" could be by number of users, notebooks, CPU cycles, hours, PAWS-internal jobs or some such. [14:38:41] Rook: I'd look at the jupyterhub prometheus metrics, IIRC those contain some data about active users [14:42:07] If so the answers might appear in https://prometheus.wmflabs.org/graph?g0.expr=%7Bproject%3D%22paws%22%7D&g0.tab=1&g0.stacked=0&g0.range_input=1h&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D [14:44:07] just created https://grafana-rw.wmcloud.org/d/eV0M3UyVk/paws-usage-statistics-draft?orgId=1, though I did not find anything more specific (runtimes, unique users, etc) [14:44:42] I'm not sure yet how I would use the information, but I am looking at research-related cloud services fro multiple ends (mostly as a users, but also on the infrastructure end) and assumed that the Wikimedia instances would have some public data that could be useful for benchmarking. I am also interested in matters of environmental footprint. [14:45:14] Neat, it shows cluster redeploys [14:47:01] Rook: Feel free to add/move/remove anything you find useful :) [14:47:11] I might tinker, thank you! [14:49:54] dcaro that's a good starting point indeed. For instance, we can now ask questions like what the number of users from before and after the switch means in terms of community engagement, documentation and such. [14:49:58] Guest30: for operational stats, you can try searching dashboards here https://grafana.wikimedia.org/dashboards/f/6ztijHyik/wmcs , and here https://grafana.wmcloud.org/d/BZWk-npVk/home?orgId=1&search=open , there might be some reports from conferences and such in other places (others might know if there's some more usage stats) [14:52:14] maybe andrewbogott or bd808 might know better where to look for those [14:59:45] Gotta hop off for now. Thanks so far. Might come back after some digging. [15:00:26] 👍 [15:41:51] !log admin moved cloudvirt1055 and cloudvirt1056 from 'spare' to 'ceph' aggregate. Prep for removing two obsolete cloudvirts, 1023 and 1024. T336064 [15:41:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:41:55] T336064: decommission cloudvirt1023 and cloudvirt1024 - https://phabricator.wikimedia.org/T336064 [15:43:50] !log moved cloudvirt1028 from 'localdisk' aggregate to 'maintenance' aggregate. Nothing new should be scheduled here, local storage should now move to cloudvirtlocal100x [15:43:50] andrewbogott: Unknown project "moved" [15:44:02] !log admin moved cloudvirt1028 from 'localdisk' aggregate to 'maintenance' aggregate. Nothing new should be scheduled here, local storage should now move to cloudvirtlocal100x [15:44:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:44:53] !log admin moved cloudvirt1023 and cloudvirt1024 from 'ceph' aggregate to 'maintenance' aggregate, prep for decom T336064 [15:44:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:29:55] Visual editor causing "walls of text" in infoboxes? [20:29:55] [20:29:57] https://tr.wikipedia.org/w/index.php?title=Garip_Yolcu&diff=27874346&oldid=25525252 [20:40:51] @Yetkin: this channel is for discussion of the Toolforge and Cloud VPS projects and the volunteer managed tools and projects they host. If you are reporting a regression in VisualEditor behavior, Phabricator and the #wikimedia-tech IRC channel are likely better forums. [20:45:12] ok, thanks [21:04:06] Should I be able to use pip inside of a webservice shell? [21:04:13] Inside of my virtual env? [21:14:46] to clarify... the virtual env is activated inside of the webservice shell [21:15:09] which was launched on the toolforge login server [21:15:39] so many layers of abstraction 😣 [21:19:54] should work yeah [21:20:06] in fact that's the usual way you should install dependencies [21:43:05] (venv) tools.check-references@shell-1683320528:~/www/python/src$ pip install --upgrade wheel pip [21:43:06] bash: pip: command not found [21:57:08] do you have the venv activated? What's the output of `which pip` ? [21:57:18] er, `which python3` [22:00:43] yes, I have the venv activated, hence "(venv)" [22:00:44] @harej: that venv is build for python3.7 and you are using a PHP 7.4 shell. You need the runtimes to match. [22:00:54] /usr/bin/python3 [22:01:07] What backend do I want? [22:01:52] the same one you use to create the venv. In this case it looks like you created the venv from a bastion directly. [22:02:06] ohhhh [22:02:10] that is probably what I did wrong! [22:02:40] I would generally recommend using the latest Python. That's python3.9 at the moment [22:03:38] So that would be... `webservice --backend=kubernetes python3.9 shell` ? [22:05:26] correct [22:10:06] alright, thank you [22:20:36] !log tools Added [22:20:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:21:30] !log tools Added "RepoLookoutBot" to hiera key "dynamicproxy::blocked_user_agent_regex" to stop unnecessary scans by https://www.repo-lookout.org/ [22:21:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:22:16] so fun that they actively ignore robots.txt and instead tell you to blackhole requests from their user-agent :/ [22:34:00] “The security scanner is designed to be a good network citizen” lolwat [22:40:11] When doing `webservice --backend=kubernetes --replicas=2 python3.9 start`, do you have to do that within the context of a venv? [22:40:44] no, I think that has to be on a bastion again, not inside a webservice shell [22:40:58] (and the venv should only be activated in a webservice shell and not on a bastion, I think) [22:41:24] okay, on bastion without venv is what I did [22:52:33] I ran `webservice --backend=kubernetes python3.9 start` from the bastion, and as soon as I did so I get an error complaining that there was no module named Flask, even though it is definitely installed (I went into the webservice shell and then into the venv to verirfy) [23:12:42] @harej: your venv needs to be a $HOME/www/python/venv for `webservice` to find it [23:13:36] Yours is currently at $HOME/venv instead [23:13:47] Ah so it is directory sensitive [23:14:33] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Python#Creating_a_virtual_environment