[01:09:47] !log tools.signatures remove old stale reports [01:09:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.signatures/SAL [14:50:15] !log tools backup /var/lib/puppet/server to /root/puppet-ca-backup-2022-06-27.tar.gz on tools-puppetmaster-02 before we do anything else to it T311412 [14:50:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:50:18] T311412: Toolforge Puppet CA expired - https://phabricator.wikimedia.org/T311412 [14:58:54] !log tools renew puppet ca cert and certificate for tools-puppetmaster-02 T311412 [14:58:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:58:57] T311412: Toolforge Puppet CA expired - https://phabricator.wikimedia.org/T311412 [16:12:49] hello. I really need to resize an instance back up to the old instance specs, now that I've deleted the older debian instance. Last time I tried resizing I encountered an error. Should I turn it off or suspend before doing this to ensure errors are lower? [16:17:19] andrewbogott: ^ any magic guidance for folks attempting instance resize via Horizon? [16:17:54] * bd808 still assumes that never works because it didn't work 4+ years ago ;) [16:22:43] I suppose I could make another instance and migrate everything across again if resizing doesn't/shouldn't work. its because the quota for the project had been used up, so there was no room to expand until the older instance was deleted [16:32:26] chippy: how long ago was 'last time'? [16:33:17] andrewbogott, around 1 month [16:34:19] ok. It will probably work better this time. you shouldn't need to shut down before the resize. BUT if at all possible, having your precious data on a cinder volume (rather than on the / drive of the VM) will make recovery much easier if disaster strikes. [16:34:41] also give me a few minutes to do some stability checks before you dive in [16:35:44] andrewbogott, thanks! it's maps-warper4 on maps project in case that helps. [16:36:04] yes most of the data is stored not on the actual vm, except the database of course [16:37:48] and the database /is/ on the VM? Or elsewhere? [16:38:01] disk performance should be the same on a cinder volume as in / (in case that was a concern) [16:38:20] Anyway -- you should be able to safely resize now, although it's never a total guarantee. [16:41:19] the db is on the instance. Not sure how to make a cinder volume. [16:41:41] sounds like a good idea to put the postgres data dir [16:41:51] lets try the resie [16:42:03] https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_instances#Cinder [16:42:31] !log paws Make renderer accessable in dev env #172 9f2f171ff2a519fa11a784e8236eaf337bb13d08 T308986 [16:42:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [16:42:34] T308986: renderer in dev env - https://phabricator.wikimedia.org/T308986 [16:44:29] thanks [16:45:48] Resizing didn't work and it's stuck in Error status. "Image exists" [16:47:11] Is this the same VM that failed to resize last time? (I should've asked that before) [16:47:20] yes it is [16:47:22] oh [16:47:29] it's probably cursed from the prior resize, sorry [16:47:34] ahh. [16:47:36] can you tell me what flavor you would like to resize it to? [16:47:45] And, are you resizing for the purposes of RAM/CPU or for storage? [16:48:22] g3.cores4.ram8.disk20 For RAM purposes [16:49:22] application was hitting too many oom and crashing. It's webserver but has some high ram operations particularly if more than one user does something more than viewing something. ram also used as cache to speed up views [16:51:08] ok, let me see what I can do [16:51:17] thanks andrewbogott I appreciate it [16:53:54] andrewbogott, chippy: https://bash.toolforge.org/quip/eRMVpoEB8Fs0LHO5Ql-t hope you don’t mind :) [16:54:14] :) [16:54:15] :) [17:14:50] chippy: took a few tries but I think you're set now [17:15:14] (lots of cruft was left sitting around from the failed resize which caused file collisions) [17:15:37] !log tools T311412 updating ca used by k8s-apiserver->etcd communication, breakage may happen [17:15:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:15:41] T311412: Toolforge Puppet CA expired - https://phabricator.wikimedia.org/T311412 [17:16:08] andrewbogott, thank you so much! I appreciate all you and friends do for us [17:16:20] you're welcome! [18:02:43] !log tools switchover active cron server to tools-sgecron-2 T284767 [18:02:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:02:46] T284767: Toolforge: migrate cron servers to Debian Buster - https://phabricator.wikimedia.org/T284767 [18:10:42] taavi: I think breakage happened. Since 17:33 I'm getting errors trying to connect to the enwiki API: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution [18:11:00] JJMC89: just saw that too for my own tools, looking into it [18:13:51] global-search fails with " The server said: Curl error: Could not resolve host: meta.wikimedia.org " [18:14:04] yes, I'm looking into it [18:14:05] .. and back up :) [18:14:40] !log tools restart calico, appears to have got stuck after the ca replacement operation [18:14:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:15:05] JJMC89: should work again now, can you check please? [18:16:04] yes, its working now [18:16:09] great, sorry about that