[12:14:24] I created https://calendar.google.com/calendar/u/1?cid=MnFhdWpodm51bzQzOXF1ZDR1YjRlZDkyODhAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ for downtime Windows [12:14:24] [url] Google Calendar - Sign in to Access & Edit Your Schedule | calendar.google.com [15:20:18] cool [15:21:10] dmehus: I added tonight too. Not sure exactly what's in Debian 10.10 but things might go bump for a minute or two [15:21:49] RhinosF1, oh, cool, is Bullseye out then? and if so, any idea when Miraheze will upgrade to Bullseye? [15:22:05] dmehus: just 10.10 not 11 [15:22:17] RhinosF1, ack. when's the 11 release date? [15:22:30] dmehus: July 31 [15:22:48] RhinosF1, ack, okay, so we've got a bit of time then to perform other tasks first [15:23:12] maybe capacity proposal task? [15:24:35] No [15:24:38] Capacity after [15:24:52] Or we have to spend twice the time installing stuff [15:34:23] RhinosF1, ah, yeah, that makes sense then :) [15:57:47] [02puppet] 07RhinosF1 pushed 031 commit to 03Icinga-disable [+0/-0/±1] 13https://git.io/Jn24O [15:57:48] [02puppet] 07RhinosF1 033032601 - Icinga: disable notifications on all hosts for Debian update [15:57:50] [02puppet] 07RhinosF1 created branch 03Icinga-disable - 13https://git.io/JJGvA [15:57:51] [02puppet] 07RhinosF1 opened pull request 03#171: Icinga: disable notifications on all hosts for Debian update - 13https://git.io/Jn243 [15:59:07] [02puppet] 07RhinosF1 pushed 031 commit to 03Phab-ro [+0/-0/±1] 13https://git.io/Jn24w [15:59:09] [02puppet] 07RhinosF1 034824aa2 - Phab: Read only for 10.10 [15:59:10] [02puppet] 07RhinosF1 created branch 03Phab-ro - 13https://git.io/JJGvA [15:59:12] [02puppet] 07RhinosF1 opened pull request 03#172: Phab: Read only for 10.10 - 13https://git.io/Jn24o [21:17:34] [02puppet] 07RhinosF1 closed pull request 03#171: Icinga: disable notifications on all hosts for Debian update - 13https://git.io/Jn243 [21:17:36] [02puppet] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jnacn [21:17:37] [02puppet] 07RhinosF1 032fe847a - Icinga: disable notifications on all hosts for Debian update (#171) [21:18:34] $log disable icinga for fleet wide main [21:18:38] maint* [21:21:29] [02puppet] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JnacP [21:21:31] [02puppet] 07RhinosF1 03ad25464 - Revert "Icinga: disable notifications on all hosts for Debian update (#171)" This reverts commit 2fe847aa30b1f345394ededda10ce7c9bda416da. [21:21:47] ^ manually downtiming instead [21:22:24] DOWNTIMESTART - Host bots1 is UP: PING OK - Packet loss = 0%, RTA = 134.13 ms rhinosf1 - [21:22:34] DOWNTIMESTART - Host db1 is UP: PING OK - Packet loss = 0%, RTA = 0.36 ms rhinosf1 - [21:22:38] DOWNTIMESTART - Host test2001 is UP: PING OK - Packet loss = 0%, RTA = 138.51 ms rhinosf1 - [21:22:44] DOWNTIMESTART - Host tools1 is UP: PING OK - Packet loss = 0%, RTA = 0.10 ms rhinosf1 - [21:24:10] $log upgrade test2001 to 10.10 [21:27:47] test2001 rebooting [21:28:02] PROBLEM - load on test2001 is UNKNOWN: Remote Icinga instance 'test2001.mirahezebots.org' is not connected to '112.node1.net.fosshost.org' [21:28:04] PROBLEM - ping6 on test2001 is UNKNOWN: Remote Icinga instance 'test2001.mirahezebots.org' is not connected to '112.node1.net.fosshost.org' [21:28:08] PROBLEM - Apache on test2001 is UNKNOWN: Remote Icinga instance 'test2001.mirahezebots.org' is not connected to '112.node1.net.fosshost.org' [21:28:24] PROBLEM - procs on test2001 is UNKNOWN: Remote Icinga instance 'test2001.mirahezebots.org' is not connected to '112.node1.net.fosshost.org' [21:28:26] PROBLEM - users on test2001 is UNKNOWN: Remote Icinga instance 'test2001.mirahezebots.org' is not connected to '112.node1.net.fosshost.org' [21:28:30] PROBLEM - Flask-site on test2001 is UNKNOWN: Remote Icinga instance 'test2001.mirahezebots.org' is not connected to '112.node1.net.fosshost.org' [21:28:32] PROBLEM - disk / on test2001 is UNKNOWN: Remote Icinga instance 'test2001.mirahezebots.org' is not connected to '112.node1.net.fosshost.org' [21:29:00] RECOVERY - load on test2001 is OK: OK - load average: 0.43, 0.12, 0.04 [21:29:08] RECOVERY - ping6 on test2001 is OK: PING OK - Packet loss = 0%, RTA = 0.06 ms [21:29:22] RECOVERY - procs on test2001 is OK: PROCS OK: 91 processes [21:29:23] test2001 is green [21:29:26] RECOVERY - users on test2001 is OK: USERS OK - 1 users currently logged in [21:29:28] RECOVERY - Flask-site on test2001 is OK: HTTP OK: HTTP/1.1 200 OK - 5431 bytes in 0.138 second response time [21:29:30] RECOVERY - disk / on test2001 is OK: DISK OK - free space: / 5338 MB (70% inode=86%); /boot/efi 99 MB (99% inode=-); [21:33:09] ^ doing bots1, will get noisy [21:34:15] PROBLEM - apt on bots1 is UNKNOWN: Remote Icinga instance '100.node4.net.fosshost.org' is not connected to '112.node1.net.fosshost.org' [21:35:13] RECOVERY - apt on bots1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:35:21] dmehus: all alerts are known [21:39:25] PROBLEM - Discord-relay on bots1 is WARNING: PROCS WARNING: 3 processes with command name 'node' [21:39:49] RhinosF1, ack [21:40:02] MacFan4000, Sario: ^ [21:40:13] MH-Discord is back, but MirahezeBot is not, in some channels [21:41:00] MirahezeBot takes a while to join all of it's channels [21:41:56] There's an intentional rate limit to prevent getting kicked for spamming [21:42:01] ^ [21:42:06] I'm checking icinga-mhbots [21:43:45] ACKNOWLEDGEMENT - Discord-relay on bots1 is WARNING: PROCS WARNING: 3 processes with command name 'node' rhinosf1 nothing broke [21:43:50] Sario: ^ [21:44:30] That's good to hear [21:45:34] yeah the Discord-relay error was what caught my eye, RhinosF1 [21:45:46] everything seems fine [21:45:49] okay [21:46:17] Sario, ack [21:46:28] [02puppet] 07RhinosF1 closed pull request 03#172: Phab: Read only for 10.10 - 13https://git.io/Jn24o [21:46:30] [02puppet] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jnaly [21:46:31] [02puppet] 07RhinosF1 03d0e005e - Phab: Read only for 10.10 (#172) [21:46:33] [02puppet] 07RhinosF1 deleted branch 03Phab-ro - 13https://git.io/JJGvA [21:46:34] [02puppet] 07RhinosF1 deleted branch 03Phab-ro [21:47:50] Sario: phab is RO [21:48:30] RhinosF1: ack [21:49:30] this shouldn't be as noisy [21:52:00] ^ that is why [21:52:09] tools1 rebooting [21:53:12] RECOVERY - Discord-relay on bots1 is OK: PROCS OK: 5 processes with command name 'node' [21:53:32] Sario: it got happy [21:53:46] db1 left [21:54:49] PROBLEM - Discord-relay on bots1 is WARNING: PROCS WARNING: 3 processes with command name 'node' [21:56:42] mysql RO [21:58:36] Sario: icinga crashed again [21:58:45] PROBLEM - apt on db1 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (3 critical updates). [21:58:48] looks like db1 has killed mysql [21:59:43] RECOVERY - apt on db1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:00:55] seems fine now [22:04:02] [02puppet] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jna4a [22:04:03] [02puppet] 07RhinosF1 0337a8cba - Revert "Phab: Read only for 10.10 (#172)" This reverts commit d0e005e3bc545c055bb013ab1dc43091b948001a. [22:04:18] Sario: ^ [22:04:26] MySQL is RW [22:04:38] phab should be too [22:06:22] DOWNTIMEEND - Discord-relay on bots1 is WARNING: PROCS WARNING: 3 processes with command name 'node' rhinosf1 nothing broke [22:06:24] PROBLEM - Discord-relay on bots1 is WARNING: PROCS WARNING: 3 processes with command name 'node' [22:06:26] DOWNTIMEEND - Host db1 is UP: PING OK - Packet loss = 0%, RTA = 0.32 ms rhinosf1 - [22:06:36] DOWNTIMEEND - Host tools1 is UP: PING OK - Packet loss = 0%, RTA = 0.08 ms rhinosf1 - [22:06:38] DOWNTIMEEND - Host bots1 is UP: PING OK - Packet loss = 0%, RTA = 134.28 ms rhinosf1 - [22:08:55] Sario: I call it done [22:10:49] dmehus: icinga-mhbots after now is a real alert [22:14:34] 43 minutes and 58 seconds from start to finish for all 4 VMs. [22:21:45] RhinosF1, ack, cool :)