[00:12:39] !log tools Image builds completed. [00:12:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [00:16:03] Hi [00:19:12] greetings kakakakwkw, how may we be of assistance? [00:36:04] I got banned [00:36:42] I have an account called the on wikitech [05:17:59] hello all! further up with taavi, we were debugging a connection problem with https://tool.pageviews.org [05:18:02] Here's a more telling chart: https://tools-prometheus.wmflabs.org/tools/graph?g0.range_input=4w&g0.end_input=2022-02-16%2011%3A52&g0.expr=sum(nginx_ingress_controller_request_duration_seconds_sum%7Bnamespace%3D%22tool-pageviews%22%7D)%20by%20(status)%20%2F%20sum(nginx_ingress_controller_request_duration_seconds_count%7Bnamespace%3D%22tool-pageviews%22%7D)%20by%20(status)&g0.tab=0 [05:19:16] As far back as data goes we've had no problems, except for a few days of 499s starting Feb 3, then the big and ever-growing jump since Feb 10, and you see it go down after I increased the kubernetes pod replica count to 4. I'd like to increase that more, but it appears 4 is the maximum. Is there a way by special request to increase the pod replica count for pageviews? [05:19:32] I'm guessing you meant https://pageviews.toolforge.org [05:19:45] lol yes [05:19:46] thank you [05:22:40] found it! https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Quota_increases [05:22:50] I shall do that now :) [05:23:20] I don't know if there's a limit on replicas specifically, or if you're just hitting the CPU allocation limit quota [05:24:59] docs says the namespace has a quota of 4 pods, but anticompositetools has a quota of 10 pods [05:25:18] s/the namespace/a namespace by default/ [05:26:05] https://grafana-labs.wikimedia.org/d/toolforge-k8s-namespace-resources/kubernetes-namespace-resources?viewPanel=1&orgId=1&var-namespace=tool-pageviews&refresh=5m it doesn't look like you're actually using all that much CPU, so you could try decreasing the per-pod allocations to see if you can squeeze more in [05:43:35] oh, so you mean I can do that now [09:30:06] hello, antispam support needed in https://www.sub-bavaria.de/w/index.php?title=Spezial:Letzte_%C3%84nderungen&limit=1000&days=30 and probably many other small mediawiki's around the webs....... hundreds of spambots registering each and every day...... infodemic waste of attention,time and electricity :-( [12:10:33] !log cloudinfra enable gtid (master_use_gtid=current_pos) on cloudinfra-db04 [12:10:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [14:44:17] !Log cloudinfra delete old cloudinfra-db instances (cloudinfra-db01 cloudinfra-db02) [14:44:21] !log cloudinfra delete old cloudinfra-db instances (cloudinfra-db01 cloudinfra-db02) [14:44:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [15:42:21] oh my, Application Credentials, for horizon, i can has use api? :O [15:45:41] addshore: maybe? I haven't tried that yet [15:45:56] addshore: I don't think that's supported today [15:45:57] it looks like it is a step closer than last time I looked at least :) [15:48:10] addshore: inch by inch. Our local champions of getting to the point of being able to use Terraform have moved on to other jobs, but there is still some interest in removing the blockers to that. [15:48:43] I was just about to try using terraform, but curled the API endpoints and they appear to still be hidden away somewhere [15:48:57] (thats my dream too) but will continue clicking UI buttons for now [15:56:35] bd808: actually I think figuring out authentication is one of the only blockers for opening the base openstack apis :-) our custom apis (enc and novaproxy) still have some ways to go but are getting there [15:56:59] taavi: inch by inch. :) [15:57:13] and thanks for working on things like that! [15:58:09] https://gerrit.wikimedia.org/r/c/operations/puppet/+/762871 should be one of the last remaining things for novaproxy, and I can hopefully copy-paste most things to the ENC api [15:59:11] addshore: if you're interested in helping figure out the last remaining blockers, I'm sure we can arrange whatever access would be necessary (/me uses the opportunity to recruit more volunteer admins) [16:02:39] *looks* [16:08:47] Not sure how I can help out there, but certainly up for helping however I can :D [17:00:31] addshore: I guess the main thing is figuring out how to authenticate against the APIs, as we don't really want to encourage having ldap passwords laying around unencrypted on peoples laptops or VPS instances. That may be what OpenStack calls 'application credentials', but I don't think any of us has looked very deeply on what those actually are [17:02:33] in terms of terraform it looks like all of the application related auth stuff should work out of the box [17:04:55] i guess it would be nice to force everything to be done through SSH in some way? if possible [17:05:20] ie terraform on client -> ssh -> cloud land -> apis terraform wants [17:08:01] I'm also wondering how shared state might be able to end up working, so that multiple folks and interact with a single terraformed environment [17:08:45] running terraform from a shared tool account could solve that problem, as the state could just be kept "locally" [17:11:22] that brings the problem of giving/removing/logging access to that account and such (happily ignore my passing by comment xd) [17:12:06] when you say that are you thinking specifically relating to accessing the stored state? or something to do with credentials? [17:13:27] I was actually thinking about things like this for mwcli recently (having credentials on disk), I made a command that interacts with gerrit, and it needs folks gerrit HTTP password, but keeping that in plain text on disk is evil [17:14:16] I was thinking of mwcli having some encryption phrase, that needs to be provided by the user whenever they run the CLI, that then decrypts the actual secrets from disk. Not sure if that really relates, but certainly something that could be related [17:20:14] addshore: are you describing an ssh-agent that is not an ssh-agent? ;) [17:20:26] bd808: exactly ;) [17:22:16] maybe i add keepass integration, dun dun dun [17:22:43] *goes to eat* [17:39:48] !log tools.majavah-bot migrated webservice and srg helper to python 3.9 and kubernetes [17:39:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.majavah-bot/SAL [17:41:16] I have to do that too [17:45:40] !log tools.openstack-browser migrated update-cache cron job to kubernetes and jobs framework [17:45:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.openstack-browser/SAL [17:48:33] !log tools.sge-status restart webservice on buster [17:48:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sge-status/SAL [17:49:56] !log tools.sge-jobs migrated update-cache cron job to kubernetes and jobs framework [17:49:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sge-jobs/SAL [17:52:22] I wish grid-deprecation.toolforge.org removed jobs that have been seen running on k8s like it does for newer grids [18:00:36] taavi: is there a good way to "see" that? Seems like we could hack it into the tool if so. [18:01:07] bd808: you can probably list all Deployment and CronJob objects, and compare their titles [18:01:35] hi    i have a banned account [18:01:44] the names won't match for web services, but there's a good chance they will for user specific cron and continuous jobs [18:02:03] help [18:02:19] what's your username and where exactly have you been banned? [18:02:27] its the [18:02:32] the name is the [18:10:50] :( no idea why, but I can't ssh to my newly created `ssh wikibase-product-testing-2022.wikidata-dev.eqiad.wmflabs` [18:11:01] already tried deleting my first 2 instances and making them again, but still no joy [18:11:09] s/eqiad.wmflabs/eqiad1.wikimedia.cloud/ [18:11:15] gosh darnit [18:11:35] yup, that works [18:11:46] and I imagine my first 2 also worked :D [18:12:01] thanks! [18:12:07] we stopped creating new .wmflabs names back in 2020? [18:14:10] old habits die hard [18:14:15] indeed [18:14:44] i deleted an old instance and just updated the name in the docs, but I only updated the prefix, not the full name :D [18:14:48] I'm reading https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework and I think it'd be easier to just shut down my bot [18:32:09] bd808: do you know if https://www.mediawiki.org/wiki/Extension:DynamicSidebar is still wanted/needed on labswiki? [18:32:17] Ryan wrote it orignally and its only enabled there [18:33:32] It seems there are no relevant MediaWiki:Sidebar/* pages currently in existence so I suspect not [18:36:48] looks like thos were deleted last year [18:36:48] https://wikitech.wikimedia.org/wiki/Special:Log?type=delete&user=&page=&wpdate=2021-12-07&tagfilter=&subtype= [18:50:02] i need some help [18:51:08] oi [18:56:09]      guys [19:45:36] Krinkle: I think it's likely that we can do with out it, but we can test the change first on labstestwiki [19:46:48] labtestwiki == cloudweb2001-dev.wikimedia.org [20:03:19] Krinkle: yeah, I don't see any need for it these days. When we were still using OpenStackManager for all the things it was useful, but it's just old code now.