[00:00:02] taavi: it should be a normal 'reboot after oom' but fstab was wrong and it took me surprisingly long to get it right. [00:00:16] taavi: do you happen to know how to 'check replication works fine'? [00:00:28] waiting for my jobs to try to write again [00:01:57] taavi: If you know what to check and what to do after, please do. I think andrewbogott and I are mostly mashing keys at this point [00:02:32] uhh, good question. I know how to turn it into r/w, but I'm not sure if there's a good way to check for issues beforehand [00:02:39] bd808: db is RO [00:02:49] taavi: great, then let's switch it to r/w. How do we do that? [00:03:02] * andrewbogott figures asking taavi is faster than googling [00:03:13] I see "read_only = ON" in /etc/my.cnf [00:03:16] andrewbogott: run `SET GLOBAL read_only = 0;` when logged in as root [00:03:29] ok! doing... [00:03:38] done [00:03:41] JJMC89: how about now? [00:04:21] tools-db-2 at least seems to be executing binary log transactions just fine. so I think that's ok [00:04:33] looks good - writing now [00:05:00] great, thanks for checking (both of you) [00:05:21] bd808: think we should send an email? Will users need to restart things? [00:06:11] "well behaved" tools should self-heal, but an announce that we had an outage is never a bad idea IMO [00:06:22] ok, I will write... [00:06:29] https://prometheus-alerts.wmcloud.org/?q=project%3Dtools is clear now [00:06:44] !status Ok [00:11:15] bd808: thank you for appearing! There is an impatient dog (and niece) waiting for me to go on evening walk so will linger for another few minutes and then depart. [00:12:04] andrewbogott: thank you too. I would have still be waving my arms in the air about the volume :) [00:13:32] I put some todos in the ticket but please add things you can think of (and links to the docs that were clearly not helpful) [08:38:55] it looks like it was a fun night :), glad it was resolved [08:53:31] if there is some data on a Cloud VPS machine that I really don't want to get lost, is a cinder volume a good solution for that, or should I store it outside the Cloud infrastructure entirely? [09:38:19] Currently we don't have backups of cinder volumes, so if something like an accidental 'rm -rf' happens, you might lose the data, so I'd recommend doing a backup on top of having a cinder volume. If you really don't want to lose it even in the case of a fire in the datacenter, then yes, better to do a backup outside of the cloud. [09:39:00] We are trying to setup backups for cinder volumes on a different datacenter, but it's not yet there ;) [15:03:05] The official word on Cloud VPS backups remains https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_Instances#Backups_of_Cloud_VPS_instances which says "No backups will be kept by Wikimedia Cloud Services." [15:04:18] the reality is that we do have some levels of redundancy and backups, but they really should not be considered stable storage by anyone [15:58:36] papaul: you see this? [16:03:20] topranks: he's not in here [16:03:24] He can /join [16:03:50] thanks! yep wrong channel sorry :) [20:44:53] Hi. I want to deploy a static node app to toolforge. Not sure how to use Gitlab's Ci/CD for that. [20:44:54] - I've create a repo on gitlab on wikimedia. [20:44:55] - I'm able to the app build with npm. [20:44:57] ...and I'm confused about shared and trusted runners... Do I really need a trusted runner to just copy static files to my tool? I need to create artifacts of the build and copy them (prefer not to keep them in the repo). [20:48:40] Here is my build job: [20:48:42] https://gitlab.wikimedia.org/toolforge-repos/zabytki-wlm-mapa/-/jobs/133391 [20:49:06] And current config: [20:49:07] https://gitlab.wikimedia.org/toolforge-repos/zabytki-wlm-mapa/-/blob/master/.gitlab-ci.yml [20:49:47] The files built supposed to go here: [20:49:48] https://zabytki.toolforge.org/ [21:15:26] I would be cool with an answer "not feasible, come back in 6 months" 😉 (I mean at least I would know it's not worth to purse this before WLM competition) [21:18:15] I think most do commit things to a git repo, and then use cron or something like https://wikitech.wikimedia.org/wiki/Help:Toolforge/Auto-update_a_tool_from_GitHub/GitLab [21:18:52] I would imagine that the deployment workflow for gitlab ci -> tool labs isn't well defined/documented/supported, at least as things stand [21:38:22] I was hoping there would be an advantage in connecting a GitLab repository to a tool definition (toolsadmin). It would be great if this connection could simplify the configuration process of a CI/CD. [21:39:25] But I think you answered my question, thanks 🙂 [22:27:20] @eccenux: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service is the thing that we hope will eventually fulfill a "push to deploy" workflow for tools. Today it is a partial solution in that triggering first a build and then a deploy is a manual process on the command line from inside Toolforge.