[00:12:02] !log quarry `kubectl -n quarry delete pod/redis-676b955f95-tkbb7` (T392107) [00:12:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [00:12:08] T392107: quarry.wmcloud.org: "This web service cannot be reached" - https://phabricator.wikimedia.org/T392107 [00:19:59] !log quarry `kubectl delete pod -n quarry --all` (T392107) [00:20:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [00:20:07] T392107: quarry.wmcloud.org: "This web service cannot be reached" - https://phabricator.wikimedia.org/T392107 [01:52:50] !log samtar@tools-bastion-13 tools.wmopbot toolforge-jobs restart bot [01:52:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wmopbot/SAL [10:55:33] !log quarry deploying T392138 T392141 T392143 patches [10:55:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [10:55:39] T392138: No alerting for quarry - https://phabricator.wikimedia.org/T392138 [10:55:39] T392141: Update quarry redis deployment - https://phabricator.wikimedia.org/T392141 [10:55:39] T392143: Quarry: Why so many web pods? - https://phabricator.wikimedia.org/T392143 [13:44:06] andrewbogott: I see now that there is a volume mounted on the instance that I have spun for recovering the data [13:48:08] CristianCantoro: more details in T392089 :) [13:48:09] T392089: [cinder] Volume failing to attach/detach - https://phabricator.wikimedia.org/T392089 [13:55:12] Yep [13:56:37] dhinus andrewbogott thanks! [14:25:35] andrewbogott is it safe to attach and mount 2 volume at the same machine? [14:25:53] CristianCantoro: usually but with your current luck you might want to avoid it if possible :) [14:26:16] andrewbogott roger that! I will back up one volume at a time, then :-D [14:33:23] yep I used two volumes on the same host many times, but as andrewbogott says maybe I would be extra safe this time :) [14:43:23] ok, I have backed up the first volume (frontrestore) now I can unmount, detach it from the horizon and then attach and mount the second volume (backdata) [14:47:21] hopefully! [14:50:00] andrewbogott: so, I have umont it (with `umount /srv`), but I see that there is a line in `/etc/fstab`. Should I comment that out before detaching the volume? [14:50:19] Maybe I take the occasion to add some documentation here: [14:50:20] https://wikitech.wikimedia.org/wiki/Help:Adding_disk_space_to_Cloud_VPS_instances [14:51:44] *unmounted [14:55:27] CristianCantoro: there is something in that page about fstab [14:55:49] but if you want to add more details that's appreciated :) [14:56:59] short answer: I think it will still work without commenting out, but you can also delete the old line [15:03:07] dhinus: right, I did not see the section about "reattaching to another instance" [15:07:55] dhinus: in that section it says "In the new instance use wmcs-prepare-cinder-volume to mount the volume. Using sudo mount /dev/sdX /path will also work, but using wmcs-prepare-cinder-volume is preferred because it will also add the volume ID to /etc/fstab and the volume will be remounted automatically if the instance is rebooted." but using [15:07:56] wmcs-prepare-cinder-volume doesn't also format the volume? So, are these instructions correct? [15:33:52] it should also format but ONLY IF it's not already formatted (I might be wrong) [15:34:46] if it's a brand new volume it will also format it [15:34:53] if it's an existing volume, it will only mount it [15:35:51] it can be confusing and maybe we should improve this (maybe we could have two scripts with different names) [16:47:34] /path (re @wmtelegram_bot: dhinus: in that section it says "In the new instance use wmcs-prepare-cinder-volume to mount the volume. Using...) [19:03:14] !log lucaswerkmeister@tools-bastion-13 tools.ranker deployed 44c3d7820d (l10n updates: zh-hans) [19:03:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ranker/SAL [19:04:11] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed 919afec938 (l10n updates: es) [19:04:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [22:42:31] are there any known issues with sending email from wikitech by any chance? [22:43:03] https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Changelog is on my watchlist, and I have the preference “Email me when a page or a file on my watchlist is changed” enabled (via global preferences), but I just noticed there was an edit the other day that I didn’t get an email for [22:43:20] (maybe GlobalPreferences got confused by the SULification of wikitech…) [22:44:30] (and I *think* I looked at the changelog page since my last email on 18 February, so I hope it’s not the “There will be no other notifications in case of further activity unless you visit this page while logged in” bit, but admittedly I can’t rule it out) [23:11:12] @lucaswerkmeister: Hmmm... I don't seem to have received any email notifs from Wikitech in the last 22 hours, but I'm not sure if that is missing anything or not. [23:13:06] * bd808 makes an edit on @lucaswerkmeister's talk page as an experiment [23:16:14] @lucaswerkmeister: Look for a notif for https://wikitech.wikimedia.org/w/index.php?title=User_talk:Lucas_Werkmeister&diff=prev&oldid=2294480 [23:16:38] email arrived [23:17:06] then maybe I hadn't visited the page (while logged in) after all :/ sorry for the noise [23:18:13] No worries. https://meta.wikimedia.org/wiki/User:BDavis_(WMF)/Scripts/GlobalWatchlistReset is my magic tool to make sure I get emails regularly. I run that whenever I have declared email notif bankruptcy (vacations and such) [23:58:11] yeah, I find you gotta hit the "mark all as read" button somewhat regularly for the email notifications to actually work