[01:19:52] bd808: how early you want to go? [01:20:04] you could go back to the toolserver [02:05:38] bd808: a factoid you might want to mention is how once upon a time toolserver ran Solaris [02:06:11] (at least on some servers) [02:08:10] Platonides: here’s one that Lucas figured out recently, Toolserver has been gone for longer than it existed. [02:08:50] that's a good one [02:09:04] I find out I have plenty of emails about toolserver, though :P [02:11:31] "The project was shutdown on July 1, 2014" [02:11:39] nearly a decade ago... [02:37:32] Solaris, ZWS, JIRA, `cc` that wasn't gcc, not-vixie cron and Python 2.6 that was patched to fix a Unicode bug (which was very nice) [02:39:42] oh this isn't even the cloud channel :p [05:40:14] I have switched over zarcillo to db1215 - I am still checking everything [07:07:43] is there anything related to backups or databases which would speak against rebooting cumi2002 tomorrow? [07:07:46] cumin2002 even [08:04:54] not from my side, unless you do them super-early, backups finish around 8:30 UTC [08:06:35] I'm fully flexible, can do these at e.g. 9:00 UTC (and doublecheck withyou beforehand) [08:07:13] just waint for manuel or amir's feedback, in case they have some maintenance scheduled or something, but otherwise good to me [08:07:53] (for me you could even do them now) [08:08:55] moritzm: I have nothing on cumin2002 so no objections [08:09:16] maybe amir has some schema change or reboot process ongoing? [08:10:48] I see, however some open tmux sessions outside of data persistence [08:10:57] ^ moritzm [08:11:07] as well as screens [08:16:23] yeah, there's a few tmuxes by Search SREs, I'll check with them when they are around [08:26:29] If anyone can review this: https://gerrit.wikimedia.org/r/917818 I want to failover m5-master [08:29:33] thank you jaime! [08:40:21] I have stopped mariadb on old zarcillo master (db1115) [08:40:25] If something breaks let me know [08:49:13] what is the new primary (I cannot see it on orchestrator) [08:57:01] db1215 [08:57:11] I have picked 1:1 replacements for ease :) [08:57:17] so db1115 -> db1215 [08:57:20] or db1120 -> db1220 [08:57:23] stuff like that [08:57:42] nice [09:03:19] This is going to be an interesting one https://phabricator.wikimedia.org/T336228 [09:18:19] I found the issue with the shrinking backups- it was not a data issue nor a backup tooling issue, technically. The aria_log for the previous backup was 1GB, and the new one very small. [09:18:42] it took me a while because the name of the file for the log changed, so it wasn't matched in the metadata [09:19:43] given it takes 1GB, I may consider it removing it after a succesful backup (we shouldn't be using Aria for anything real data) [09:22:47] A better dashboard will make these explorations easy [09:24:09] moritzm: please let me know a bit earlier for cumin1001 [09:24:22] jynus: speaking of backups in mailing lists :D https://lists.wikimedia.org/hyperkitty/list/wikimedia-l@lists.wikimedia.org/message/NBTZ4S25LK7HVQU2G3U2UIYFYW5JNTXJ/ [09:24:37] (Response to https://lists.wikimedia.org/hyperkitty/list/wikimedia-l@lists.wikimedia.org/message/YI3QKMES4RKILNQR5MENAKIGIAJJ75T5/) [09:25:43] yeah, I try not to read wm-l, but that reached me (you may have sent it me before) [09:31:26] Amir1: sure thing, for cumin1001 the headsup will be more like a week or so [09:31:47] thanks! [14:41:06] jynus: https://phabricator.wikimedia.org/T335640#8837301 can I put this host down? or do you want to own it yourself? it is backup1-codfw host [14:41:15] yeah [14:41:27] to the second, I can do it myself [14:42:12] excellent, I will leave you to it [14:42:19] I downtimed it for 24h [14:42:24] ok