[00:19:29] !log tools.pbbot Add MissingUkrainianAudio servlet [00:19:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pbbot/SAL [04:09:36] !log wikisp Elemental maintenance to all vm [04:09:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikisp/SAL [12:39:17] !log admin-monitoring delete all VMs and restart the fullstack service on cloudcontrol1005 - T326626 [12:39:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin-monitoring/SAL [12:39:20] T326626: nova-fullstack failed on 2023-01-10 - https://phabricator.wikimedia.org/T326626 [17:37:36] !log paws setup dns for paws-dev 837395a6695d11ae9026df31dd57b25df02102b1 T326629 [17:37:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [17:37:39] T326629: Setup DNS for paws-dev.codfw1dev.wmcloud.org. - https://phabricator.wikimedia.org/T326629 [19:18:25] Would it be OK to enable Trove on deployment-prep? Was going to test the Terraform PG backend https://developer.hashicorp.com/terraform/language/settings/backends/pg [19:18:42] Happy to make a ticket, just wanted to get opinions before I move fwd [19:21:39] terraform is still not officially owned by anyone (T215217), so I'm not sure what kind of answer are you looking for? [19:21:40] T215217: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 [19:22:01] technically trove could probably be used for that, although iirc its postgres support is rather primitive [19:24:34] that's good context, thanks. I'm just looking at terraform backends that we could potentially use at the Foundation. S3 is probably better but I don't think there's a Thanos in DP [19:24:59] If Trove doesn't work well with PG that's OK, I could stand up my own PGDB or Minio for S3...just wanted to check [19:28:05] also, even more unofficially speaking, in case you set up any kind of automation in deployment-prep I 100% expect that someone will just ignore it and touch stuff manually to unbreak something slightly broken, even if there are clear and visible instructions not to do exactly that [19:30:29] I'm OK with someone breaking my stuff in DP, it won't be touching anything directly related to MW deploys or anything. I can always build on another project, it's just much easier on DP as it has some production-like stuff like Kafka, Elastic etc. [20:18:37] side note, is there a terraform users club (SIG or whatever) for WMF? I'd be interested in joining if so [21:00:26] For account-creation-assistance, I'm just using Terraform Cloud (both for state storage, and for remote execution from GitHub pull requests) [22:05:36] nice, I'll have to dig into your repo. I'm with WMF and we're not allowed to use PaaS but any lessons learned would be helpful [23:13:27] I'm finding that recently created instances for cvn aren't showing up in Grafana , e.g. at https://grafana-labs.wikimedia.org/d/000000059/cloud-vps-project-board. When manually setting it to the value (in case its cached for dropdown only) I get "No data" in each of the panels [23:13:36] cvn-apache10, cvn-app10, cvn-app11 [23:16:07] I do see /usr/bin/prometheus-node-exporter running on the newer instances (cc AntiComposite ) [23:16:35] inflatador: I'm in the middle of refactoring everything onto Terraform, but any repo starting terraform-* under https://github.com/enwikipedia-acc/ is the stuff I'm doing :)