[11:20:42] !log tools.wd-image-positions deployed 43f0101e20 (Bootstrap 5.1) [11:20:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [12:03:50] !log tools.wd-image-positions deployed e878def771 (two CSS improvements) [12:03:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [13:06:59] Has anyone here ever used Deno? [13:56:11] what's the logstash host in beta? I see references to deployment-logstash03.deployment-prep.eqiad.wmflabs but it doesn't seem to exist [13:59:30] ori: looks like they are in the 'logging' project these days. everything should go via deployment-kafka-logging01, I believe gelf support was removed with the last logstash version upgrade [14:00:21] how do services using service-runner log, then? [14:00:45] they don't, if I had to take a guess [14:00:48] I also see logstash configured to 'localhost' in LabsServices.php in wmf-config, is there a local listener on each beta cluster app server? [14:00:58] brb [14:01:13] yep, that's handled by rsyslog [14:01:39] on production, logging is handled by the k8s infra directly [14:03:54] β€œhow does X happen” is always a dangerous question in beta (re @wmtelegram_bot: they don't, if I had to take a guess) [14:04:04] (see also, for example, grafana) [14:04:21] well, logs are making it to the logstash instance from mediawiki on the beta cluster [14:05:16] I'll need to figure out how to get logs from the beta cluster instances of the function-{orchestrator,evaluator} services to the logstash instance there. I'm not sure there's a rsyslog listener on the host that will forward to logstash, and if there is ,whether it would be accessible from the docker containers. [14:05:27] sorry, I realize this might be a better fit for #wikimedia-releng. [14:08:40] ori: I wonder if one could update role::beta::docker_services to send everything from /var/lib/docker/containers/*/*-json.log to rsyslog [14:08:56] or configure docker to send logs to rsyslog directly https://docs.docker.com/config/containers/logging/configure/ [14:10:59] I think that's a great idea [14:16:29] I think it should be reasonably straightforward to do, I'll file a phab task [14:17:52] I suspect you'll need to write a patch too if you want to get it implemented [14:18:54] nonsense, I'll create a slide deck advertising it as a blocker and present it at the SoS or an annual strategy meeting [14:19:36] how many story points do you figure [14:20:28] (I'm only kidding, obviously.) [14:20:47] are those story points fibonacci scale or t-shirt sizes? [14:22:06] > As of 18.03, Docker Engine Enterprise(EE) supports multiple log drivers, but it is not in the Community Edition(CE) [14:22:08] blergh [14:23:10] so maybe /var/lib/docker/containers/*/*-json.log -> rsyslog is the way [14:23:23] @lucaswerkmeister Are these messages coming from IRC? [14:23:35] su_Theta: yes [14:23:39] yes, the channel is mirrored with #wikimedia-cloud on freenode [14:23:45] libera :) [14:23:49] Does IRC support formatting? [14:23:50] 🀦 [14:23:56] sorry, yes, indeed libera [14:24:02] not the network whomst shall not be named [14:24:12] *Like is this bold on IRC?* [14:24:43] no [14:24:46] 𝔑𝔬, 𝔦𝔱 𝔦𝔰𝔫'𝔱. [14:24:53] Sad. [14:25:03] it's a feature, not a bug [14:25:10] and Unicode? (re @lucaswerkmeister: 🀦) [14:25:20] unicode works fine [14:25:59] On the web client I see that text between `` is formatted [14:26:30] what does the web client connect to? (re @su_Theta: On the web client I see that text between `` is formatted) [14:26:39] libera? telegram? [14:26:47] https://web.libera.chat/?channel=#mediawiki-core (re @jeremy_b: what does the web client connect to?) [14:26:51] I mean there [14:59:23] Hello, I'm setting up a new debian instance for migration, i've a couple of questions. First is that the project has reached it's quota so the new instance will have less cpu and ram than the one I'm replacing. I imagine it's easy, once the old one has been deleted, to update the new one with more resources? [15:00:09] Second is, I've done similar migration a couple of times. I set up a proxy via horizon, fine, but I see zones and recordsets now. Should I concern myself with setting up this for the server? [15:01:43] chippy: first, yes, you should be able to resize an existing server between any of the flavors whose name starts with `g3.` just fine [15:02:05] second, if you're just using a web proxy, you don't need to touch dns records manually [15:02:16] oh, whilst I'm here. In bastion, the new server name "maps-warper4" doesn't seem to be found, but I can ssh into it using the ip address [15:02:23] thanks taavi [15:02:50] seems to work fine for me: taavi@bastion-eqiad1-04:~$ host maps-warper4.maps.eqiad1.wikimedia.cloud [15:02:50] maps-warper4.maps.eqiad1.wikimedia.cloud has address 172.16.7.145 [15:04:21] taavi, although "host maps-warper4" gives no record, but the maps-warper3 one does [15:04:53] is that due to the zone perhaps? [15:05:53] it's all working anyhow, but just a small memory aid :) [15:06:12] oh, that was an intentional change a while ago, https://wikitech.wikimedia.org/wiki/News/Phasing_out_the_.wmflabs_domain [15:08:02] taavi, thanks! [15:18:02] taavi, further on from the proxy question. maps-warper3 doesn't have it's old proxy listed anymore. I will want to re-use the warper.wmflabs.org proxy name to point to the new server. Would I have to delete that recordset/zone which I imagine refers to the maps-warper3, then create a new proxy with the old proxy name? [15:19:29] which exact record set are you talking about? [15:19:59] in the DNS Zones for the maps project, warper.wmflabs.org. [15:20:30] ah, I see now what you are talking about [15:21:12] yeah, I want to keep on using that domain name but point it to the replacement instance when ready [15:21:24] the 'maps' project has traffic volumes so high that we split those proxies off the main self-managed proxy to a fully independent pair of proxies [15:22:00] unfortunately those can't be managed via horizon, so you'll need to ask someone with enough access to manually update it for you [15:22:32] okay, I imagine it would be a ticket on phabricator thing? [15:22:55] yeah, or ask on this channel [15:23:02] okay [15:23:15] thanks again :) [16:45:05] Is there an API for making a tool? [16:47:38] no [16:48:04] Can I share something I've been working on? [16:49:13] πŸ‘ [16:50:30] yes plz [16:51:13] Is that "yes plz" reply to my message? The reply threads are messed up here on Telegram (re @wmtelegram_bot: yes plz) [16:51:25] su_Theta: yes! [16:51:39] https://github.com/roj1512/mediawiki [16:52:48] In case if anyone uses Deno/TypeScript, it is going to be an overkill for making requests to the API. [16:53:01] addshore: I wondered when you get around to asking that one :) It could be possible by adding some stuff to Striker (toolsadmin.wikimedia.org), but today that app is where the magic lives for making new tool accounts. [16:53:25] su_Theta: that's neat! what led you to Deno (vs. Node?) I've been curious about it but haven't checked it out myself. [16:53:48] bd808: Yeah, I now remember looking at this 6 months ago or so, and determining there kind of is an API, but not one that would be super easy to use, I guess primarily because of auth [16:54:36] addshore: auth yes, but mostly because when I built Striker I was not thinking API first [16:55:28] I would love to rip that app apart and put it back together more like Toolhub works, but time is cruel and fleeting [16:55:41] I don't use Node too much. Also, I was writing a bot that used Deno. I made this for it. Although, the code is really clean and can be used in browsers already as an ES module. I can publish the NPM package, too, if there is a demand for it. It will not be much effort :) (re @wmtelegram_bot: su_Theta: that's neat! what led you to Deno (vs. Node?) I've been curious about it but haven't checked it out myself.) [16:56:31] I'm wondering if a small deploy command might be a fun next target. 1) copy files, 2) restart / start / stop webservice, 3) sync k8s deployments 4) allow "hook" for other commands to run between any of those steps [16:56:56] su_Theta: Cool, nice work. The API looks nice and clean. [16:57:04] It'd need some sort of tool manifest file, not sure one of those exists right now? There used to be this .description file I think? :D [16:57:12] @su_Theta: I don't know if [[User:Premeditated]] is here on irc/telegram, but they have been interested in the past about Deno -- https://phabricator.wikimedia.org/T253470 [16:59:05] addshore: you might want to chat with d.caro during Eu work hours. He's working on the custom container build stuff and I think that would be the most reasonable way for your wmcli to gain super powers for deploying tool code. [16:59:37] Thanks. The request types are all generated. I wish I could generate the response types, too. The MediaWiki API clients would have been much better and their developers would have less work to do if they provided a schema like Telegram's APIs. (re @wmtelegram_bot: su_Theta: Cool, nice work. The API looks nice and clean.) [17:00:16] The $HOME/.description thing still exists, but it is really just a weak ancestor of toolinfo.json records [17:00:26] bd808: ack, the other thing I was thinking of is a "emulated" environment for running tools locally, or also, running a web tool locally, but piping to the real tool db etc [17:01:18] su_Theta: the REST API has nice specs (https://www.mediawiki.org/wiki/Wikimedia_REST_API) but the "normal" API is from the time of the dinosaurs when nobody wrote API specs [17:01:55] addshore: that would be epic. T157733 is an old ticket that is related. [17:01:55] T157733: Create developer environment using Docker images from Tool Labs Kubernetes - https://phabricator.wikimedia.org/T157733 [17:02:15] It would be nice to have Deno pre-installed. I already installed it for myself. It doesn't need that much rights, it will do everything in the home folder. (re @wmtelegram_bot: @su_Theta: I don't know if [[User:Premeditated]] is here on irc/telegram, but they have been interested in the past abou...) [17:03:49] bd808: right, thats my evening thing to look at :D I already did a local dev thing for a tell using he images a year or more ago now, lets see if I can formalize it [17:04:30] @su_Theta: supporting it once it is available generally is the harder part than the first install. Without an apt repo managed by a trusted upstream it will be very hard to properly support yet another language runtime at a platform level. [17:05:59] It won't be possible either :) [17:06:00] [17:06:01] The official upstreams are getting updates almost like never, and Deno is releasing new versions too often (re @wmtelegram_bot: @su_Theta: supporting it once it is available generally is the harder part than the first install. Without an apt repo m...) [17:06:41] addshore: Jean-Fred did some related work at a past hackathon too -- https://github.com/JeanFred/wiki-replicas-proxy [17:06:53] OOOh, I remember that hackathon! and that projext! [17:07:05] yext [17:07:20] bd808: yes, that is exactly what I was thinking of for the non emulation of DBs mode [17:07:33] Is there a list of all other services provided by tools? redis etc? [17:08:48] addshore: the cheeky answer is https://wikitech.wikimedia.org/wiki/Special:PrefixIndex?prefix=Toolforge&namespace=12&hideredirects=1 [17:08:54] <3 [17:09:42] redis, elasticsearch, and smtp are the main ones I think beyond the replicas and toolsdb [18:20:41] !log commtech quota set --cores 10 --ram 18432 [18:20:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Commtech/SAL [18:21:04] !log commtech quota set --cores 10 --ram 18432 T309202 [18:21:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Commtech/SAL [18:21:06] T309202: Request increased quota for commtech Cloud VPS project - https://phabricator.wikimedia.org/T309202 [19:10:42] !log striker Deleted legacy stretch instances which had been shutdown for ~4 weeks (T306096) [19:10:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Striker/SAL [19:10:45] T306096: Cloud VPS "striker" project Stretch deprecation - https://phabricator.wikimedia.org/T306096 [20:34:54] !log tools.lexeme-forms deployed 24d9b273c5 (l10n updates) [20:34:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL