[00:00:13] dmehus: what do you mean with your reply to that? [00:01:16] I might just close it and not move forward. It won't work on initial creation only when editing, so I may either just keep it when edited to be a non existent DB. [00:02:29] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.24, 8.66, 7.29 [00:03:12] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.80, 8.42, 6.71 [00:03:14] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown) [00:03:24] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.00, 6.61, 6.02 [00:03:31] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 33.25, 23.87, 16.98 [00:05:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.46, 6.44, 6.02 [00:11:33] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.40, 3.43, 2.59 [00:13:31] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.82, 3.29, 2.65 [00:16:58] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.46, 7.13, 7.58 [00:17:58] PROBLEM - wiki.cyberfurs.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cyberfurs.org could not be found [00:17:58] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for stardewmodding.wiki could not be found [00:18:00] PROBLEM - www.zenbuddhism.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.zenbuddhism.info could not be found [00:18:00] PROBLEM - wiki.redeemer.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.redeemer.live could not be found [00:18:05] PROBLEM - wiki.wysteria.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wysteria.ga could not be found [00:18:09] PROBLEM - guia.esporo.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for guia.esporo.net could not be found [00:18:09] PROBLEM - sarovia.graalmilitary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sarovia.graalmilitary.com could not be found [00:21:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.05, 5.55, 4.37 [00:24:41] RECOVERY - wiki.cyberfurs.org - reverse DNS on sslhost is OK: rDNS OK - wiki.cyberfurs.org reverse DNS resolves to cp13.miraheze.org [00:24:46] RECOVERY - stardewmodding.wiki - reverse DNS on sslhost is OK: rDNS OK - stardewmodding.wiki reverse DNS resolves to cp13.miraheze.org [00:24:51] RECOVERY - wiki.redeemer.live - reverse DNS on sslhost is OK: rDNS OK - wiki.redeemer.live reverse DNS resolves to cp13.miraheze.org [00:24:51] RECOVERY - www.zenbuddhism.info - reverse DNS on sslhost is OK: rDNS OK - www.zenbuddhism.info reverse DNS resolves to cp13.miraheze.org [00:24:57] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.97, 4.78, 4.39 [00:24:59] RECOVERY - wiki.wysteria.ga - reverse DNS on sslhost is OK: rDNS OK - wiki.wysteria.ga reverse DNS resolves to cp13.miraheze.org [00:25:05] RECOVERY - guia.esporo.net - reverse DNS on sslhost is OK: rDNS OK - guia.esporo.net reverse DNS resolves to cp13.miraheze.org [00:25:07] RECOVERY - sarovia.graalmilitary.com - reverse DNS on sslhost is OK: rDNS OK - sarovia.graalmilitary.com reverse DNS resolves to cp13.miraheze.org [00:26:25] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.39, 7.10, 7.85 [00:31:17] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.49, 3.50, 2.93 [00:32:41] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.70, 5.85, 4.94 [00:33:13] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:33:14] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.43, 3.13, 2.87 [00:34:25] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 13.55, 9.09, 8.14 [00:34:39] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.84, 5.29, 4.83 [00:34:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.93, 8.98, 7.79 [00:36:36] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.38, 5.74, 5.02 [00:38:25] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.32, 6.77, 7.43 [00:38:33] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.81, 5.64, 5.10 [00:38:50] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.56, 7.37, 7.43 [00:40:29] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.93, 4.83, 4.85 [00:44:44] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.58, 5.65, 6.68 [00:46:25] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.03, 5.29, 6.49 [00:47:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 9.52, 16.95, 23.74 [00:57:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 11.43, 14.72, 19.61 [01:03:31] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.10, 19.18, 19.79 [01:05:52] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.79, 7.18, 4.80 [01:06:42] PROBLEM - archive.a2b2.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for archive.a2b2.org could not be found [01:06:44] PROBLEM - en.famepedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.famepedia.org could not be found [01:06:45] PROBLEM - pastport.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pastport.org could not be found [01:06:46] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mineland.eu could not be found [01:06:47] PROBLEM - viileapedia.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for viileapedia.ga could not be found [01:06:49] PROBLEM - www.recyclerie-uzege.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.recyclerie-uzege.org could not be found [01:06:57] PROBLEM - bharatwiki.online - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bharatwiki.online could not be found [01:07:05] PROBLEM - wiki.beergeeks.co.il - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.beergeeks.co.il could not be found [01:07:10] PROBLEM - wiki.helioss.co - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.helioss.co could not be found [01:07:11] PROBLEM - wiki.make717.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.make717.org could not be found [01:07:12] PROBLEM - savage-wiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for savage-wiki.com could not be found [01:07:16] PROBLEM - adadevelopersacademy.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for adadevelopersacademy.wiki could not be found [01:07:18] PROBLEM - wiki.anglish.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.anglish.info could not be found [01:07:18] PROBLEM - wiki.wilderyogi.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wilderyogi.eu could not be found [01:07:19] PROBLEM - arru.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for arru.xyz could not be found [01:07:23] PROBLEM - matttest.mtwiki.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for matttest.mtwiki.cf could not be found [01:07:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.33, 22.84, 21.59 [01:07:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.48, 5.97, 4.66 [01:09:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.86, 19.09, 20.35 [01:09:46] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.57, 4.52, 4.30 [01:13:33] RECOVERY - archive.a2b2.org - reverse DNS on sslhost is OK: rDNS OK - archive.a2b2.org reverse DNS resolves to cp13.miraheze.org [01:13:36] RECOVERY - en.famepedia.org - reverse DNS on sslhost is OK: rDNS OK - en.famepedia.org reverse DNS resolves to cp13.miraheze.org [01:13:39] RECOVERY - pastport.org - reverse DNS on sslhost is OK: rDNS OK - pastport.org reverse DNS resolves to cp13.miraheze.org [01:13:40] RECOVERY - wiki.mineland.eu - reverse DNS on sslhost is OK: rDNS OK - wiki.mineland.eu reverse DNS resolves to cp13.miraheze.org [01:13:41] RECOVERY - viileapedia.ga - reverse DNS on sslhost is OK: rDNS OK - viileapedia.ga reverse DNS resolves to cp13.miraheze.org [01:13:45] RECOVERY - bharatwiki.online - reverse DNS on sslhost is OK: rDNS OK - bharatwiki.online reverse DNS resolves to cp13.miraheze.org [01:13:46] RECOVERY - www.recyclerie-uzege.org - reverse DNS on sslhost is OK: rDNS OK - www.recyclerie-uzege.org reverse DNS resolves to cp13.miraheze.org [01:13:50] RECOVERY - wiki.helioss.co - reverse DNS on sslhost is OK: rDNS OK - wiki.helioss.co reverse DNS resolves to cp13.miraheze.org [01:13:52] RECOVERY - wiki.make717.org - reverse DNS on sslhost is OK: rDNS OK - wiki.make717.org reverse DNS resolves to cp13.miraheze.org [01:13:53] RECOVERY - savage-wiki.com - reverse DNS on sslhost is OK: rDNS OK - savage-wiki.com reverse DNS resolves to cp13.miraheze.org [01:14:03] RECOVERY - wiki.beergeeks.co.il - reverse DNS on sslhost is OK: rDNS OK - wiki.beergeeks.co.il reverse DNS resolves to cp13.miraheze.org [01:14:03] RECOVERY - adadevelopersacademy.wiki - reverse DNS on sslhost is OK: rDNS OK - adadevelopersacademy.wiki reverse DNS resolves to cp13.miraheze.org [01:14:06] RECOVERY - wiki.anglish.info - reverse DNS on sslhost is OK: rDNS OK - wiki.anglish.info reverse DNS resolves to cp13.miraheze.org [01:14:07] RECOVERY - wiki.wilderyogi.eu - reverse DNS on sslhost is OK: rDNS OK - wiki.wilderyogi.eu reverse DNS resolves to cp13.miraheze.org [01:14:09] RECOVERY - arru.xyz - reverse DNS on sslhost is OK: rDNS OK - arru.xyz reverse DNS resolves to cp13.miraheze.org [01:14:16] RECOVERY - matttest.mtwiki.cf - reverse DNS on sslhost is OK: rDNS OK - matttest.mtwiki.cf reverse DNS resolves to cp13.miraheze.org [01:15:25] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jna7C [01:15:27] [02miraheze/ManageWiki] 07Universal-Omega 03e506de8 - Support help messages with ManageWikiExtensions [01:15:28] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns [01:15:30] [02ManageWiki] 07Universal-Omega opened pull request 03#278: Support help messages with ManageWikiExtensions - 13https://git.io/Jna7W [01:16:27] miraheze/ManageWiki - Universal-Omega the build passed. [01:18:25] [02mw-config] 07Universal-Omega edited pull request 03#3890: T7192: Move cache out of /srv/mediawiki/w directory - 13https://git.io/JsIit [01:19:19] [02miraheze/mw-config] 07Universal-Omega pushed 0339 commits to 03Universal-Omega-patch-2 [+0/-0/±64] 13https://git.io/Jna7h [01:19:21] [02miraheze/mw-config] 07Universal-Omega 036160650 - Merge branch 'master' into Universal-Omega-patch-2 [01:19:22] [02mw-config] 07Universal-Omega synchronize pull request 03#3890: T7192: Move cache out of /srv/mediawiki/w directory - 13https://git.io/JsIit [01:20:14] miraheze/mw-config - Universal-Omega the build passed. [01:20:55] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jna5C [01:20:56] [02miraheze/mw-config] 07Universal-Omega 0384d351c - Update LocalSettings.php [01:20:58] [02mw-config] 07Universal-Omega synchronize pull request 03#3890: T7192: Move cache out of /srv/mediawiki/w directory - 13https://git.io/JsIit [01:21:54] miraheze/mw-config - Universal-Omega the build passed. [01:24:30] [02mw-config] 07Universal-Omega edited pull request 03#3890: Move cache out of /srv/mediawiki/w directory - 13https://git.io/JsIit [01:26:32] [02mw-config] 07Universal-Omega edited pull request 03#3890: Move cache out of /srv/mediawiki/w directory - 13https://git.io/JsIit [01:27:30] [02puppet] 07Universal-Omega edited pull request 03#1772: Move cache out of /srv/mediawiki/w directory - 13https://git.io/JZt3Y [01:27:38] [02puppet] 07Universal-Omega edited pull request 03#1772: Move cache out of /srv/mediawiki/w directory - 13https://git.io/JZt3Y [01:36:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.31, 6.96, 6.03 [01:38:25] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.48, 6.17, 5.86 [01:45:04] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found [01:45:04] PROBLEM - stablestate.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for stablestate.org could not be found [01:45:04] PROBLEM - meta.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for meta.gyaanipedia.com could not be found [01:45:05] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fc.songcontests.eu could not be found [01:45:05] PROBLEM - wiki.shaazzz.ir - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.shaazzz.ir could not be found [01:45:07] PROBLEM - madzebrascience.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for madzebrascience.wiki could not be found [01:45:08] PROBLEM - pornwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pornwiki.org could not be found [01:45:12] PROBLEM - www.lab612.at - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.lab612.at could not be found [01:45:12] PROBLEM - kunwok.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kunwok.org could not be found [01:45:13] PROBLEM - thestarsareright.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for thestarsareright.org could not be found [01:45:18] PROBLEM - lg.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lg.gyaanipedia.com could not be found [01:45:20] PROBLEM - wiki.umbranet.mandelbrotgames.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.umbranet.mandelbrotgames.com could not be found [01:45:20] PROBLEM - wiki.bikinibottomdiscord.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.bikinibottomdiscord.org could not be found [01:45:26] PROBLEM - wiki.kali-team.cn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.kali-team.cn could not be found [01:45:27] PROBLEM - allthetropes.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for allthetropes.miraheze.org could not be found [01:51:49] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: rDNS OK - franchise.franchising.org.ua reverse DNS resolves to cp13.miraheze.org [01:51:49] RECOVERY - stablestate.org - reverse DNS on sslhost is OK: rDNS OK - stablestate.org reverse DNS resolves to cp13.miraheze.org [01:51:50] RECOVERY - meta.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - meta.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org [01:51:51] RECOVERY - fc.songcontests.eu - reverse DNS on sslhost is OK: rDNS OK - fc.songcontests.eu reverse DNS resolves to cp13.miraheze.org [01:51:52] RECOVERY - wiki.shaazzz.ir - reverse DNS on sslhost is OK: rDNS OK - wiki.shaazzz.ir reverse DNS resolves to cp13.miraheze.org [01:51:56] RECOVERY - madzebrascience.wiki - reverse DNS on sslhost is OK: rDNS OK - madzebrascience.wiki reverse DNS resolves to cp13.miraheze.org [01:51:57] RECOVERY - lg.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - lg.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org [01:51:58] RECOVERY - pornwiki.org - reverse DNS on sslhost is OK: rDNS OK - pornwiki.org reverse DNS resolves to cp13.miraheze.org [01:52:00] RECOVERY - wiki.umbranet.mandelbrotgames.com - reverse DNS on sslhost is OK: rDNS OK - wiki.umbranet.mandelbrotgames.com reverse DNS resolves to cp13.miraheze.org [01:52:01] RECOVERY - wiki.bikinibottomdiscord.org - reverse DNS on sslhost is OK: rDNS OK - wiki.bikinibottomdiscord.org reverse DNS resolves to cp13.miraheze.org [01:52:05] RECOVERY - www.lab612.at - reverse DNS on sslhost is OK: rDNS OK - www.lab612.at reverse DNS resolves to cp13.miraheze.org [01:52:06] RECOVERY - kunwok.org - reverse DNS on sslhost is OK: rDNS OK - kunwok.org reverse DNS resolves to cp13.miraheze.org [01:52:07] RECOVERY - thestarsareright.org - reverse DNS on sslhost is OK: rDNS OK - thestarsareright.org reverse DNS resolves to cp13.miraheze.org [01:52:14] RECOVERY - wiki.kali-team.cn - reverse DNS on sslhost is OK: rDNS OK - wiki.kali-team.cn reverse DNS resolves to cp13.miraheze.org [01:52:14] RECOVERY - allthetropes.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - allthetropes.miraheze.org reverse DNS resolves to cp13.miraheze.org [02:13:50] Universal_Omega, oh I just mean to have CreateWiki Extension return the decline reason depending on if the wiki is still active, closed, or in a deleted state. And please don't close that PR (#214). It will be a big workflow improvement I think. [02:51:22] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.81, 1.90, 1.23 [02:53:20] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.97, 1.64, 1.21 [03:55:46] PROBLEM - matttest.mtwiki.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'matttest.mtwiki.cf' expires in 15 day(s) (Tue 06 Jul 2021 03:50:41 GMT +0000). [03:56:39] PROBLEM - wiki.umbranet.mandelbrotgames.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.umbranet.mandelbrotgames.com' expires in 15 day(s) (Tue 06 Jul 2021 03:47:45 GMT +0000). [04:02:14] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JnV2B [04:02:16] [02miraheze/ssl] 07MirahezeSSLBot 0341a08df - Bot: Update SSL cert for matttest.mtwiki.cf [04:07:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JnVaX [04:07:27] [02miraheze/ssl] 07MirahezeSSLBot 036ccff0f - Bot: Update SSL cert for wiki.umbranet.mandelbrotgames.com [04:30:17] RECOVERY - matttest.mtwiki.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'matttest.mtwiki.cf' will expire on Sat 18 Sep 2021 03:02:07 GMT +0000. [04:30:26] RECOVERY - wiki.umbranet.mandelbrotgames.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.umbranet.mandelbrotgames.com' will expire on Sat 18 Sep 2021 03:07:19 GMT +0000. [05:45:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JnVdH [05:45:11] [02miraheze/services] 07MirahezeSSLBot 0308089a6 - BOT: Updating services config for wikis [06:04:25] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.32, 6.32, 5.14 [06:06:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.18, 6.26, 5.26 [06:20:25] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.10, 7.58, 6.10 [06:22:25] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.42, 6.93, 6.04 [06:24:25] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.44, 6.06, 5.83 [06:34:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.04, 8.07, 6.66 [06:36:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.54, 7.17, 6.26 [06:36:25] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.13, 7.50, 6.62 [06:40:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.51, 5.84, 5.94 [06:40:25] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.15, 6.15, 6.30 [06:48:07] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JnwvQ [06:48:09] [02miraheze/mediawiki] 07Reception123 03be39e07 - Update Citizen skin [07:00:37] [02puppet] 07Reception123 closed pull request 03#1786: mwscript: support foreachwikiindblist - 13https://git.io/JnINU [07:00:39] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JnwUV [07:00:40] [02miraheze/puppet] 07Universal-Omega 034acbce1 - mwscript: support foreachwikiindblist (#1786) [07:03:06] [02puppet] 07Universal-Omega synchronize pull request 03#1772: Move cache out of /srv/mediawiki/w directory - 13https://git.io/JZt3Y [07:03:57] [02puppet] 07Universal-Omega synchronize pull request 03#1772: Move cache out of /srv/mediawiki/w directory - 13https://git.io/JZt3Y [12:20:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JnrZH [12:20:11] [02miraheze/services] 07MirahezeSSLBot 033715450 - BOT: Updating services config for wikis [15:34:24] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.51, 6.52, 5.24 [15:36:14] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JnrxW [15:36:15] [02miraheze/ssl] 07Reception123 03b9823b9 - add diamowiki.ga cert [15:36:23] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.65, 5.81, 5.13 [15:39:06] !log deleted and dropped lakehubwiki (reset request) [15:39:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:41:14] !log renamed documentcontrolwiki to famedatawiki [15:41:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:45:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jnrhb [15:45:12] [02miraheze/services] 07MirahezeSSLBot 0348109de - BOT: Updating services config for wikis [15:54:20] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JnovO [15:54:22] [02miraheze/mw-config] 07Reception123 034fd5bd7 - documentcontrolwiki -> famedatawiki [15:54:39] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jnovc [15:54:41] [02miraheze/mw-config] 07Reception123 03e6bf0e5 - documentcontrolwiki -> famedatawiki [15:55:15] miraheze/mw-config - Reception123 the build passed. [15:55:37] miraheze/mw-config - Reception123 the build passed. [15:59:51] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.02, 6.31, 5.11 [16:01:50] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.60, 6.24, 5.23 [16:01:59] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.66, 7.53, 6.13 [16:04:08] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.05, 20.36, 16.19 [16:05:28] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.81, 2.88, 2.03 [16:05:48] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.83, 7.29, 5.90 [16:06:03] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.08, 8.66, 6.55 [16:06:05] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.18, 20.91, 16.93 [16:07:27] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.91, 2.84, 2.11 [16:07:35] !log manually replaced famedatawiki mw_settings entry [16:07:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:07:50] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.74, 7.35, 6.16 [16:07:59] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.45, 7.68, 6.43 [16:08:02] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 11.92, 17.79, 16.27 [16:08:19] !log $cw = new CreateWikiJson( 'famedatawiki' ); $cw->resetWiki(); [16:08:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:09:48] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.55, 7.47, 6.22 [16:11:44] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.83, 8.15, 6.69 [16:11:48] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.54, 7.36, 6.33 [16:11:53] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.11, 8.01, 6.84 [16:12:47] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.39, 5.71, 4.48 [16:13:40] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.87, 7.91, 6.77 [16:13:46] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.51, 7.68, 6.56 [16:13:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.32, 6.99, 6.60 [16:13:57] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 23.52, 24.30, 19.85 [16:14:42] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.19, 5.27, 4.47 [16:15:45] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.88, 6.97, 6.44 [16:15:45] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.88, 6.38, 6.43 [16:15:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.41, 21.47, 19.31 [16:16:36] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.16, 4.56, 4.31 [16:17:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.25, 7.42, 7.70 [16:17:44] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.12, 5.93, 6.12 [16:17:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.31, 19.83, 18.95 [16:19:28] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.06, 6.29, 6.45 [16:20:30] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.90, 5.24, 4.63 [16:21:45] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.03, 24.71, 21.21 [16:22:24] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.70, 4.85, 4.58 [16:23:42] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.72, 21.04, 20.26 [16:25:40] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 10.60, 17.56, 19.09 [16:35:19] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.59, 7.22, 6.45 [16:36:01] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki amicalwiki /home/reception/dumpamical.xml --username-prefix="amical" [16:36:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:39:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.74, 6.44, 6.30 [16:39:23] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.64, 6.21, 6.75 [16:59:30] !log [reception@jobrunner4] created local account for Mh maintenance on bngyaanipediawiki [16:59:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:01:23] !log [reception@jobrunner4] sudo -u www-data php deleteBatch.php -u Miraheze maintenance script -r Copyright violation: Per [[m:Stewards|Steward]] [[phab:T7503|request]] --wiki bngyaanipediawiki /home/reception/bngyaanipediawiki.txt [17:01:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:04:20] Hmmm, so the copyright violations were still standing on that wiki? [17:04:37] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.62, 7.60, 6.39 [17:04:54] [02puppet] 07Reception123 closed pull request 03#1772: Move cache out of /srv/mediawiki/w directory - 13https://git.io/JZt3Y [17:04:56] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/JnonP [17:04:57] [02miraheze/puppet] 07Universal-Omega 036735978 - Move cache out of /srv/mediawiki/w directory (#1772) [17:05:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.49, 7.22, 6.50 [17:06:02] For some reason, Vector automatically switches to the legacy version whenever I go to [[SN]]. [17:06:03]

https://meta.miraheze.org/wiki/SN [17:06:06] [url] Stewards' noticeboard - Miraheze Meta | meta.miraheze.org [17:06:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.28, 7.27, 6.40 [17:07:20] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.73, 6.91, 6.47 [17:08:05] PROBLEM - mw10 Disk Space on mw10 is CRITICAL: DISK CRITICAL - free space: / 799 MB (4% inode=75%); [17:09:10] [02mw-config] 07Universal-Omega closed pull request 03#3890: Move cache out of /srv/mediawiki/w directory - 13https://git.io/JsIit [17:09:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jnocw [17:09:13] [02miraheze/mw-config] 07Universal-Omega 03427689e - Move cache out of /srv/mediawiki/w directory (#3890) [17:09:14] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [17:09:16] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [17:10:08] miraheze/mw-config - Universal-Omega the build passed. [17:10:10] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.67, 6.43, 6.37 [17:10:16] PROBLEM - mw9 Disk Space on mw9 is CRITICAL: DISK CRITICAL - free space: / 87 MB (0% inode=76%); [17:11:28] PROBLEM - mw8 Disk Space on mw8 is CRITICAL: DISK CRITICAL - free space: / 297 MB (1% inode=75%); [17:12:03] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.81, 21.57, 18.37 [17:12:26] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 14.45, 23.23, 17.02 [17:12:26] PROBLEM - phab2 Current Load on phab2 is WARNING: WARNING - load average: 1.10, 1.80, 0.99 [17:12:51] PROBLEM - mw10 Disk Space on mw10 is WARNING: DISK WARNING - free space: / 1608 MB (8% inode=75%); [17:13:07] PROBLEM - test3 Disk Space on test3 is WARNING: DISK WARNING - free space: / 2084 MB (10% inode=73%); [17:14:00] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.16, 19.19, 17.87 [17:14:26] RECOVERY - phab2 Current Load on phab2 is OK: OK - load average: 0.32, 1.31, 0.91 [17:14:50] PROBLEM - mw10 Disk Space on mw10 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=75%); [17:15:12] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.37, 6.55, 6.54 [17:16:12] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:16:21] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 13.75, 18.63, 16.51 [17:18:53] Universal_Omega: disk space just ran out everywhere [17:19:02] Well 8,9,10 [17:19:16] PROBLEM - mw8 Disk Space on mw8 is WARNING: DISK WARNING - free space: / 1995 MB (10% inode=75%); [17:19:47] PROBLEM - mw9 Disk Space on mw9 is WARNING: DISK WARNING - free space: / 1776 MB (9% inode=76%); [17:22:24] uh oh [17:23:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.11, 6.75, 6.51 [17:25:11] PROBLEM - mw8 Disk Space on mw8 is CRITICAL: DISK CRITICAL - free space: / 946 MB (4% inode=75%); [17:25:34] dmehus: don't panic [17:25:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.14, 7.59, 6.85 [17:26:44] PROBLEM - mw10 Disk Space on mw10 is WARNING: DISK WARNING - free space: / 1608 MB (8% inode=75%); [17:27:30] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.45, 7.12, 6.76 [17:29:25] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.60, 6.71, 6.66 [17:30:02] okay, so everything is breaking now [17:31:25] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.23, 7.83, 6.31 [17:32:28] Yes [17:32:45] huh, strangely Bulbapedia is also down right now [17:32:51] coincidence? [17:32:54] Meta seems fine [17:33:03] But someone should check them disk alerts [17:33:11] Reception123, Universal_Omega: ^ [17:33:21] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 13.04, 9.06, 7.58 [17:33:23] I am looking [17:33:28] PROBLEM - mw9 Disk Space on mw9 is CRITICAL: DISK CRITICAL - free space: / 742 MB (3% inode=76%); [17:33:28] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.58, 6.83, 6.18 [17:33:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.12, 20.44, 18.94 [17:34:44] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.80, 7.41, 6.60 [17:35:43] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.92, 18.82, 16.27 [17:36:41] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.16, 6.53, 6.38 [17:36:59] Hi Voidwalker [17:37:03] hi [17:37:24] !log sudo -u www-data rm -R /srv/mediawiki/w/cache/ on mw* / jbr* / test3 [17:37:26] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.62, 7.08, 6.45 [17:37:29] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.47, 21.90, 19.83 [17:37:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:37:38] !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki && sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki=loginwiki on mw* / jbr* / test3 [17:37:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:38:17] PROBLEM - wiki.redeemer.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.redeemer.live could not be found [17:38:21] PROBLEM - viileapedia.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for viileapedia.ga could not be found [17:38:21] PROBLEM - cyberlaw.ccdcoe.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cyberlaw.ccdcoe.org could not be found [17:38:22] PROBLEM - ecole.science - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ecole.science could not be found [17:38:50] PROBLEM - madgenderscience.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for madgenderscience.wiki could not be found [17:38:51] PROBLEM - savagepedia.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for savagepedia.wiki could not be found [17:39:20] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.09, 7.62, 7.42 [17:39:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.60, 20.98, 19.73 [17:39:36] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.98, 19.73, 17.33 [17:40:36] RECOVERY - test3 Disk Space on test3 is OK: DISK OK - free space: / 3782 MB (19% inode=73%); [17:41:23] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.67, 7.65, 7.10 [17:41:24] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.02, 20.26, 19.67 [17:41:49] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:43:47] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [17:44:59] RECOVERY - cyberlaw.ccdcoe.org - reverse DNS on sslhost is OK: rDNS OK - cyberlaw.ccdcoe.org reverse DNS resolves to cp13.miraheze.org [17:45:00] RECOVERY - viileapedia.ga - reverse DNS on sslhost is OK: rDNS OK - viileapedia.ga reverse DNS resolves to cp13.miraheze.org [17:45:04] RECOVERY - ecole.science - reverse DNS on sslhost is OK: rDNS OK - ecole.science reverse DNS resolves to cp13.miraheze.org [17:45:05] RECOVERY - wiki.redeemer.live - reverse DNS on sslhost is OK: rDNS OK - wiki.redeemer.live reverse DNS resolves to cp13.miraheze.org [17:45:20] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.61, 6.39, 6.70 [17:45:26] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.12, 6.33, 6.41 [17:45:39] RECOVERY - madgenderscience.wiki - reverse DNS on sslhost is OK: rDNS OK - madgenderscience.wiki reverse DNS resolves to cp13.miraheze.org [17:45:48] RECOVERY - savagepedia.wiki - reverse DNS on sslhost is OK: rDNS OK - savagepedia.wiki reverse DNS resolves to cp13.miraheze.org [17:46:23] Universal_Omega: https://phabricator.miraheze.org/T7504#150479 [17:46:24] [url] ⚓ T7504 Localisation is not working | phabricator.miraheze.org [17:47:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.60, 5.77, 6.75 [17:48:51] RhinosF1: Thanks, and I will do incident report later also, there were issues with transition that were not anticipated. Should be fixed if page is purged, but I will try to fix it in mass. [17:49:30] ah, did we miss something with moving the cache? [17:49:36] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.17, 1.72, 1.19 [17:50:54] Universal_Omega: you can use a maint script to purge everywhere [17:51:04] purgeNamespace.php --all [17:51:07] RhinosF1: I know. [17:51:16] Voidwalker: disk space ran out. Cache got a tad word [17:51:18] Weird [17:51:35] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.99, 1.53, 1.18 [17:52:01] Hmm, if the cache was using a large portion of the disk space, it actually kind of makes sense that this would happen, it's just no one thought to check [17:52:26] RhinosF1: Because I tried to `cp` the old cache to new one before puppet ran but that did not work as expected because of disk space. So totally my fault. [17:52:31] Voidwalker: they were pretty close anyway to full [17:52:41] Universal_Omega: it's ok [17:53:02] exactly, you'd have to remove the old cache immediately to prevent the disk from maxing out storage [17:53:38] We should think about disk space when me image the bullseye ones [17:53:47] If it's that tight [17:54:02] That'll be august [17:59:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.80, 6.55, 6.47 [18:01:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.55, 6.13, 6.32 [18:01:27] Voidwalker: on that note, what would you do for cloud*? [18:01:35] db we've done migrations before [18:01:41] But cloud* scares me [18:02:02] We've never done an evacuation of a cloud server without destroying the VMs on it and simply recreating new [18:02:12] Whereas this would be in place [18:02:28] Things like mw*/cp* can just be reimage [18:02:45] definitely something I'd try and research first [18:03:06] Voidwalker: I doubt a single cloud server would have capacity [18:03:16] So short of a new cloud server [18:03:19] Forgetting it [18:03:34] Or having very good backups of VM states and attempting in place [18:03:42] That's my options [18:03:51] at the moment though, I'm just trying to figure out if I can create automatic filters for my mail account, as sorting through 400+ monitoring emails for everything else is getting pretty annoying [18:04:12] Yeah sorry for bots spam last night [18:04:18] I fixed bots phab [18:04:31] That was dns issues resolving the db hostname [18:04:39] So I just went ahead and made that use the ip [18:05:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.84, 7.34, 6.78 [18:07:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.97, 7.55, 6.92 [18:09:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.44, 6.79, 6.72 [18:12:33] !log [universalomega@jobrunner3] sudo -u www-data php /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/resetWikiCaches.php --wiki=tuscriaturaswiki [18:12:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:19:00] RhinosF1, I don't think we personally need all the MirahezeBots notification e-mails. Should be able to read all the notifications in Icinga, I think? [18:19:35] dmehus: last night was I didn't tick the button to downtime services too [18:19:48] Today with phabricator was a genuine outage [18:19:53] While it was short [18:20:02] It was a valid reason to send alerts [18:20:33] I do want to make monitoring better though [18:20:48] So it'll probably be improved once we're on the new infra [18:21:21] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.29, 7.88, 6.96 [18:21:56] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.78, 6.88, 6.29 [18:22:15] RhinosF1, ah, true, but I mean, it is a bit spammy. Couldn't we just use push notifications for Icinga or something, and then have Hund.io e-mail us if the server's actually down? Load issues probably don't need to be e-mail notifications [18:22:52] dmehus: I mean the reason last night it alerted was because the servers were down [18:23:17] yeah [18:23:27] I just mean all the e-mails I get to my Gmail [18:23:46] Icinga should alert before hund [18:23:50] that one was probably fine, but we could look into reducing when it e-mails [18:23:54] But I'd like alerting to be more sane [18:23:59] yeah [18:24:05] maybe a digest once a day e-mail [18:24:08] We tried zenduty but that was a disaster [18:24:19] No we just need it to be appropiate [18:24:31] Load warning alerts don't need to email everyone [18:24:34] MirahezeBots is a small project, wouldn't be terrible if it were down for 1-2 hours, I think [18:24:38] true [18:24:40] yeah [18:24:52] It'd probably be best for us to all to be able to adjust our own email notification settings [18:24:59] maybe have it e-mail a general IMAP mailbox that everyone could log into [18:25:06] My suggestion for now is filter your email [18:25:11] > It'd probably be best for us to all to be able to adjust our own email notification settings [18:25:11] yeah... that'd be good [18:25:12] dmehus: definately not [18:25:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 8.00, 7.88, 7.19 [18:25:20] why not? [18:25:27]