[03:04:51] taavi: AntiComposite: you inspired me. I'm no longer running on my home-grown python. Eagerly awaiting the rollout of a bullseye bastion so I can move to 3.9 :-) [12:11:49] roy649: if you're running your app in kubernetes, you shouldn't even need a bullseye bastion since we have bullseye/python 3.9 containers available [14:07:01] !log tools.wikibugs Updated channels.yaml to: 4c7bf9f3c273443eba9989f32dd1f563d3378e94 quibble messages only go to dedicated channel [14:07:01] !log tools.wikibugs deploying https://gerrit.wikimedia.org/r/c/labs/tools/wikibugs2/+/757403 [14:07:03] ah [14:07:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [14:07:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [15:43:41] tavi [15:46:39] tavi: It makes things much simpler if the same python is on both the bastion and the pod. [15:47:02] roy649: that is exactly what `webservice shell` exists to solve [15:47:32] ssh to bastion, become tool, enter proper runtime, profit! [15:50:03] that's the theory, but it's just a lot more convenient to do the basic editing, unit test, etc without the extra hop. [15:52:55] roy649: I would say that editing beyond changing non-versioned config files and running unit tests are not things you should do on the bastion. But I understand that there are currently a lot of folks using the bastions as a remote laptop to some extent rather than just a deploy server. [15:58:01] To quote from https://wikitech.wikimedia.org/wiki/Portal:Toolforge/About_Toolforge#Bastion_hosts, "dev.toolforge.org [15:58:01] functionally identical, please use this for heavy processing such as compiles" [15:58:29] but we've had this conversation before :-) [16:00:16] In theory, I agree that nothing should happen on the bastions, for a bunch of reasons. When I've set up bastion hosts in the past, I've stripped everything off them except the bare minimum needed to ssh in and then ssh back out to a target system. Reduces the attack surface to a minimum. [16:01:56] but I went with what the WMF docs said to do and I just use the dev bastions for my basic development work. [16:07:34] I will continue to advocate for best practices. [16:09:56] and I appreciate that you do so. [16:19:40] anyway, to change to a new topic, I've got the basic workflow of my indexing app working. The next step is to think about scale-up. What I'd like to do is work up an outline of what I plan to do and how much resources I think it's going to take and then go through that with somebody from WMF to make sure I don't make your infrastructure keel over and die. Who's the right person to do that with? [16:20:55] roy649: I guess a phab ticket with the related Toolforge tags on it [16:21:37] but kubernetes does a pretty good job keeping web apps under constraints [16:22:03] so I don't think your tool would "make our infrastructure keel over and die" hehe [16:22:21] we can, however, extend quotas [16:22:25] unless roy649 crushes the little Elasticsearch cluster [16:22:33] yeah, elastic is another topic [16:22:49] Well, yeah, the volume of how much data I'm going to shove into ES is the thing that's worrying me the most. [16:23:24] if you are worried, it's probably too much data for the current cluster to hold... [16:23:39] * bd808 goes to look at the storage size on those instances [16:25:25] I've indexed one dump file out of about 700. I'm going to index a couple more and once I've done that I think we'll be in a better place to estimate what the whole thing will be and we can figure out where to go from there. [16:27:41] roy649: on tools-elastic-1 (which I assume is representative) there is ~100GiB usable space left for indexes. The current spi-tools-dev-es-index is ~40MiB [16:28:14] !log toolsbeta trying to join node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@nostromo [16:28:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:28:44] * bd808 wonders if anything is still using the similarity_* indexes [16:29:10] OK, so even if I go 1000-fold more data, that should fit, but it'll be on the order of half your available storage. [16:29:37] but I'll still creep up on that a bit at a time. [16:30:01] the more scarce resource is likely compute and ram for doing complex lookups [16:32:31] OK, I gotta bail. The snow is starting to fall here, so last chance to hit the supermarket before the big storm rolls in. [16:34:02] good luck! [17:13:30] Hi, its a very strange situation [17:13:31] when I'm running python script using jstart, getting: [17:13:33] "_pywrap_tensorflow_internal.so: failed to map segment from shared object" [17:13:34] but when I'm running same code on python3 interactive shell in tool account toolforge it running successfully [17:17:33] is your interactive shell directly on a bastion, and if yes, which one? [18:50:05] Aniket: what's the name of your tool account? [18:50:49] nevermind, I remember it is imagesimilarity [18:51:57] Lucas: Yes directly on bastion, I'm using python3 shell [18:52:45] and is it the normal bastion or maybe a Buster one? [18:55:17] (your prompt should say "imagesimilarity@...", what does it say after the @) [20:44:06] !log tools.integraality Deploy 13ed7a3 [20:44:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL