[08:00:47] Who does handle otrs? It seems it run out of disk space over the night [08:01:12] Is it colab? [08:03:40] yes [08:04:13] ^ jelto, eoghan, arnoldokoth [08:04:20] jynus: yes, you can always check /etc/wikimedia/contacts.yaml, also puppet is disabled since 4.5 ays wih message [08:04:25] Testing T355980 - root [08:04:25] T355980: [vrts] Move attachment storage out of mysql - https://phabricator.wikimedia.org/T355980 [08:04:41] 4.5 ays? [08:04:42] vrts1002 is a replica afaik [08:04:49] (6797 minutes ago) [08:04:59] on vrts1002 [08:05:34] I was looking at https://wikitech.wikimedia.org/wiki/SRE/Collaboration_Services/Documentation [08:05:39] which is the only page [08:05:45] but had no team members [08:07:33] their team interface doesn't seem to be complete :) but the contacts page on office wiki is up to date [08:09:46] I don't like that- it seems pages for SRE in general seem to be duplicated, but poorly [08:12:21] for example, service operation still says: "Gitlab, VRTS, Phabricator" [08:15:15] jynus: I think team interface pages are meant to be definitive [08:15:23] I have a proposal to fix them, but I don't want to touch them massively myself [08:15:35] Emperor: yeah, my proposal includes mass deletion :-D [08:17:47] when the currenty system::role refactoring is complete, we can also simply display the ownership in /etc/issue, so that e.g.it prints "vrts1002 is a VRTS Web Application Server (vrts), managed by Colloboration Services" [08:18:08] not /etc/issue per se, but via motd [08:18:17] nice [08:33:30] The team interface is complete and at https://office.wikimedia.org/wiki/Team_interfaces/SRE_-_Collaboration_Services [08:33:56] Which does not undermine the fact that the pages on Wikitech badly need to be updated [08:34:41] or deleted (?) [09:26:43] sobanski: ok if I silence for a few hours the vtrs1002 and comment on T355980 ? [09:26:44] T355980: [vrts] Move attachment storage out of mysql - https://phabricator.wikimedia.org/T355980 [09:26:53] *alert [09:38:27] jynus: Thanks for pointing out! vrts1002 is our test instance, definitely feel free to silence alerts. I'll take care of it. [09:38:57] I will silence it for, let's say, 24 hours [09:39:07] Perfect, thanks. [16:31:48] given https://phabricator.wikimedia.org/T309724 and the ssh failure when trying to connect to the console, what other way do we have of doing gnt-instance console FQDN given that it fails? [16:33:18] the context is that a cookreimage is stalling checking for reboot and I wanted to check why that is the case [16:33:24] er cookbook reimage [16:50:25] sukhe it would be ugly, but it's technically to mount the VM disk to the hypervisor and chroot into the VM [16:50:37] technically **possible** that is [16:50:43] :P [16:51:05] Probably want to mount read-only [16:51:12] coward ;p [16:51:59] Emperor: noted, will do it as your user :) [16:52:22] :) [16:53:08] I treat my home VMs a lot more roughly ;P . Had to quit using BTRFS b/c it didn't like my random rebooting hijinx [16:53:41] the train-presync timer failed to sync to a number of boxen overnight: https://phabricator.wikimedia.org/P56710 [16:54:09] that's actually a great sysadmin learning: whatever is theoretically better doesn't mean is logistically better [16:54:26] i'll re-run train-presync, but wanted to make sure those timeouts were known. [17:14:43] heads-up folks: we are merging https://gerrit.wikimedia.org/r/c/operations/dns/+/1002585 to lower TTLs for dyna.wm.org. no cause for concern, just awareness [17:14:58] topranks: all work in codfw completed, right? [17:17:19] https://phabricator.wikimedia.org/T355863#9538866 ok :) [17:18:48] denisse: brennen: ^ in case there are any pages (don't anticipate) [17:18:58] er sorry brennen, I meant brett [17:19:10] sukhe: ACK, thanks. [18:15:56] sukhe: yes all work completed sry was afk [18:17:09] np thanks, I found your comment and confirmed :)