[12:50:07] !log paws ingress options for ingress-nginx 4b753ffd22bac5c8ed5085feb4d4e93de2e31e61 T325746 [12:50:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [12:50:10] T325746: ingress-nginx - https://phabricator.wikimedia.org/T325746 [14:12:05] !log clouddb-services deleted CNAME wikilabels.db.svc.eqiad.wmflabs (T307389) [14:12:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [14:12:08] T307389: Upgrade wikilabels databases to buster/bullseye - https://phabricator.wikimedia.org/T307389 [15:29:09] Has anyone ever tried to run a flutter app (https://flutter.dev/) on toolforge? It compiles to JS, so it should be possible with a generic webserver, like with the Lighttpd kubernetes container, but I haven't found any tool that uses it. Could be quite useful for tools with an additional mobile app.. [15:29:47] isn't flutter a mobile framework? how would you run that via a web server? [15:35:24] I think you can build a web app too, it seems it generates a js static file you can serve [16:41:58] !log quarry Fix various outdated URLs in Quarry website footer [16:42:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [17:11:46] exactly, it can generate a web interface too since a few releases :) (re @wmtelegram_bot: I think you can build a web app too, it seems it generates a js static file you can serve) [19:59:13] andrewbogott: fyi, we'll get on the debian upgrade for CVN shortly. Main thing we need is temporary quota bump, just floating IPs this time, rest is OK, I think. [19:59:57] Krinkle: sounds good! Did you make a quota request ticket already? [20:00:41] I haven't, I commented at https://phabricator.wikimedia.org/T306066#8499942 though [20:01:52] done ref T326269 [20:01:53] T326269: Temporary quota increase for 'cvn' - https://phabricator.wikimedia.org/T326269 [20:04:43] !log cvn bump floating ip quota from 2 to 4, T326269 [20:04:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cvn/SAL [20:04:50] thx taavi [20:11:10] looks like something might be up here. https://openstack-browser.toolforge.org/project/deployment-prep https://openstack-browser.toolforge.org/project/cvn [20:11:12] > "Unknown project 'deployment-prep'. Are you just guessing?" [20:11:51] I think that's just a bad error message for "something went wrong when gathering this data", let me have a look [20:12:13] Horizon is unresponsive as well it seems since just now. [20:12:48] Horizon is fine I think [20:13:26] Krinkle: try now? [20:13:55] Looks good now, thanks [20:14:25] the main reason I use openstack browser btw, is as quick way to get the correct FQDN when I'm not sure, which it seems Horizon doesn't display anywhere I think? cc andrewbogott [20:15:56] You're right that Horizon doesn't know much about fqdns [20:16:55] Krinkle: I restarted the dns servers (one at a time) but it may be that openstack-browser only knows about one of them. All done now, in any case. [20:17:26] andrewbogott: I see a new instance "cvn-nfs-1" do I need to do anything with this? I don't any volumes attached to the existing instances nor any server groups, so I'm guessing what it does I don't need to specify when creating new instances? [20:18:11] I see it listed as "Anti Affinity" under server group and "nfs" sec group, but I don't know if I need to use those or not. [20:18:17] we do use NFS for backups and sync between servers. [20:19:06] Krinkle: that's the NFS server for your project. You should be able to ignore it, VMs will attach to it or not as per puppet settings. [20:19:17] perfect [20:19:31] We're moving away from central hardware NFS servers to in-project virtual NFS servers -- that's what that is. [20:21:09] I notice that the flavour we used for our web (apache) server isn't there anymore (cores2.ram4.disk40) but we're fine with 20GB disk afaik, so no problem. The bot (app) servers do need that space but that flavour still exists, so all good :) [20:21:47] Krinkle: all flavors are 20GB now, any extra storage can be created as a separate volume and attached. [20:22:01] For new VMs that is. [20:22:32] This is part of making VMs more like cattle -- you keep the important data on detachable volumes which can be moved to new VMs as needed. [20:24:07] Sure. Is detachable different from NFS in terms of performance? We currently open 30-100 MB sqlite files many times per second. [20:25:26] # /dev/mapper/vd-second--local--disk 60G 1.1G 56G 2% /srv [20:25:34] hm.. okay, something changed here. Never mind I guess. [20:26:11] ah right, we're keeping our backups on NFS, and the 80GB was just what the default was for that instance size when you need more than 20GB. [20:26:13] All good. [20:26:48] Detachable storage is much faster but can only connect to one host at a time. [20:27:31] that'd be perfect yeah, but let's see how far we get without it [20:27:49] Thank you for moving off Stretch by the way, you're one of the last users. You're skipping Buster and going to Bullseye I hope? [20:28:01] Indeed [20:30:49] the idea of making /srv detachble and re-attacheable between upgrades seems quite appealing actually, even if we did fit our services within the 20GB / root [20:31:34] it would save the hassle of rsyncing it all to nfs and then to another instance [20:31:58] although we'd probably do that anyway given we can't do all bots on a given server in one go [20:32:09] so.. maybe another time, let's just do it the way we did before [20:32:33] alright, we'll spend a few days migrating, but the VMs look good [22:30:30] does it really make sense to use sqlite for that usage? I expect some contention there [22:43:47] Platonides: no, it'd absolutely terrible to use SQLite for thousands per minute synchronous processing, even worse, for every event we open and close the file as it's own connection. It's the single biggest bottleneck. But it's been this way for 10y. [22:45:14] Having said that, I don't think it causes contention specifically, as the is only one thread in one process processing events and querying a given Db. [22:45:29] Each bot has its own db [22:45:31] replication happens over IRC [22:45:43] I'm not familiar with the code, or if it's online, even, but it's probably using a library that can be pointed to another implementation, like mysql [22:46:07] hmm [22:46:20] if it's one db per bot, why does it need to open and close it? [22:46:29] Yeah, back when the bots where sores over there different basement computers on different continents, sql replication over IRC seemed like a good idea [22:46:43] the fsync()s are probably killing their performance, too [22:47:14] Yeah, back when the bots where spread over three different basement computers on different continents, sql replication over IRC seemed like a good idea [22:47:18] it uses Mono.Data.Sqlite [22:47:34] well, sql replication over IRC gains coolness points :D [22:47:42] Yep, it's pretty slow indeed. Just about keeps up with most wikis. [22:48:04] doesn't keep up well with the high-rate editing on commons and wikidata though [22:48:29] The replication thread is indeed an exception and the reason probably why it closes/opens just in case. [22:48:44] Could be managed into a shared thread for that purpose [22:48:46] if you disabled fsyncing, that surely would really speed up [22:48:48] Or indeed MySQL [22:48:54] you lose crash security, though [22:49:52] Yeah, most recent data tends to eg most relevant so that's kept me from messing with it. But more than anything it's time. Definitely welcome additional hands. It's ripe for improvement [22:49:55] ideally we'd rewrite it in a language the maintainers are more familiar with and using more modern tools [22:50:38] I'm also open to helping a port to Python https://github.com/countervandalism/CVNBot/issues/73 [22:51:00] https://meta.wikimedia.org/wiki/David_Gerard%27s_Law is unfortunately the limiting factor at the moment [22:51:02] Or more iteratively, MySQL first https://github.com/countervandalism/CVNBot/issues/17 [22:51:51] AntiComposite: indeed. Getting you on board was both one of the best things recently but also proof of the law [22:56:45] migrating to python, that could be done with a library that supports sqlite, but can then be switched to mysql [22:56:54] * bd808 sees talk about irc bots and python and wonders if he has enough spoons to help [22:58:30] you probably want sqlalchemy for mostly drop-in SQLite+MySQL support [22:59:39] yeah, pymysql and the built-in sqlite library do not use compatible syntax [22:59:54] they both implement dbapi, but differently [23:02:51] speaking of irc, we never really followed up on T278584 legoktm :/ [23:02:52] T278584: Promote use of SASL for Cloud VPS/Toolforge hosted Libera.chat / Freenode IRC bots - https://phabricator.wikimedia.org/T278584 [23:03:45] I wrote a Matrix bot and it was way too easy, I don't really have much motivation to fix IRC bots these days :< [23:06:18] I would like to spend some time implementing the 3-way Telegram<-->Matrix<-->IRC bridge for this and other channels though, it's working pretty well via Discord in #mediawiki