[00:04:23] <icinga-miraheze>	 PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:06:04] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.67, 7.07, 5.94
[00:06:31] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.29, 22.62, 17.98
[00:08:09] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.00, 6.51, 5.87
[00:09:49] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[00:10:42] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.36, 22.43, 19.02
[00:12:46] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.40, 20.21, 18.63
[00:23:52] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[00:32:14] <icinga-miraheze>	 RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:35:57] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.55, 6.66, 5.61
[00:36:12] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.67, 7.10, 6.00
[00:42:37] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.22, 7.59, 6.56
[00:44:46] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.10, 6.56, 6.30
[00:45:13] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.40, 7.55, 6.84
[00:47:21] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.05, 6.63, 6.59
[00:54:12] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[01:29:09] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[01:38:22] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[02:13:16] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[02:57:29] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[03:32:33] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[03:41:08] <icinga-miraheze>	 PROBLEM - mabi.myomi.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mabi.myomi.net could not be found
[03:41:08] <icinga-miraheze>	 PROBLEM - wiki.conworlds.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.conworlds.org could not be found
[03:41:12] <icinga-miraheze>	 PROBLEM - wiki.valkyrienskies.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.valkyrienskies.org could not be found
[03:41:13] <icinga-miraheze>	 PROBLEM - n64brew.dev - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for n64brew.dev could not be found
[03:41:56] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[03:47:51] <icinga-miraheze>	 RECOVERY - mabi.myomi.net - reverse DNS on sslhost is OK: rDNS OK - mabi.myomi.net reverse DNS resolves to cp13.miraheze.org
[03:47:56] <icinga-miraheze>	 RECOVERY - wiki.conworlds.org - reverse DNS on sslhost is OK: rDNS OK - wiki.conworlds.org reverse DNS resolves to cp13.miraheze.org
[03:48:05] <icinga-miraheze>	 RECOVERY - n64brew.dev - reverse DNS on sslhost is OK: rDNS OK - n64brew.dev reverse DNS resolves to cp13.miraheze.org
[03:48:12] <icinga-miraheze>	 RECOVERY - wiki.valkyrienskies.org - reverse DNS on sslhost is OK: rDNS OK - wiki.valkyrienskies.org reverse DNS resolves to cp13.miraheze.org
[04:16:58] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[04:22:43] <icinga-miraheze>	 PROBLEM - www.mh142.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mh142.com could not be found
[04:22:46] <icinga-miraheze>	 PROBLEM - madgenderscience.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for madgenderscience.wiki could not be found
[04:22:46] <icinga-miraheze>	 PROBLEM - miraheze.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.tk could not be found
[04:22:47] <icinga-miraheze>	 PROBLEM - wiki.minecraftathome.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.minecraftathome.com could not be found
[04:22:48] <icinga-miraheze>	 PROBLEM - munwiki.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for munwiki.info could not be found
[04:22:50] <icinga-miraheze>	 PROBLEM - wiki.grottocenter.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.grottocenter.org could not be found
[04:22:50] <icinga-miraheze>	 PROBLEM - kunwok.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kunwok.org could not be found
[04:22:51] <icinga-miraheze>	 PROBLEM - miraheze.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.com could not be found
[04:22:54] <icinga-miraheze>	 PROBLEM - wiki.slimeapedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.slimeapedia.org could not be found
[04:22:56] <icinga-miraheze>	 PROBLEM - wiki.wysteria.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wysteria.ga could not be found
[04:29:37] <icinga-miraheze>	 RECOVERY - wiki.minecraftathome.com - reverse DNS on sslhost is OK: rDNS OK - wiki.minecraftathome.com reverse DNS resolves to cp13.miraheze.org
[04:29:39] <icinga-miraheze>	 RECOVERY - munwiki.info - reverse DNS on sslhost is OK: rDNS OK - munwiki.info reverse DNS resolves to cp13.miraheze.org
[04:29:43] <icinga-miraheze>	 RECOVERY - miraheze.com - reverse DNS on sslhost is OK: rDNS OK - miraheze.com reverse DNS resolves to cp13.miraheze.org
[04:29:43] <icinga-miraheze>	 RECOVERY - wiki.grottocenter.org - reverse DNS on sslhost is OK: rDNS OK - wiki.grottocenter.org reverse DNS resolves to cp13.miraheze.org
[04:29:49] <icinga-miraheze>	 RECOVERY - www.mh142.com - reverse DNS on sslhost is OK: rDNS OK - www.mh142.com reverse DNS resolves to cp13.miraheze.org
[04:29:50] <icinga-miraheze>	 RECOVERY - wiki.slimeapedia.org - reverse DNS on sslhost is OK: rDNS OK - wiki.slimeapedia.org reverse DNS resolves to cp13.miraheze.org
[04:29:53] <icinga-miraheze>	 RECOVERY - madgenderscience.wiki - reverse DNS on sslhost is OK: rDNS OK - madgenderscience.wiki reverse DNS resolves to cp13.miraheze.org
[04:29:54] <icinga-miraheze>	 RECOVERY - miraheze.tk - reverse DNS on sslhost is OK: rDNS OK - miraheze.tk reverse DNS resolves to cp13.miraheze.org
[04:30:01] <icinga-miraheze>	 RECOVERY - kunwok.org - reverse DNS on sslhost is OK: rDNS OK - kunwok.org reverse DNS resolves to cp13.miraheze.org
[04:30:03] <icinga-miraheze>	 RECOVERY - wiki.wysteria.ga - reverse DNS on sslhost is OK: rDNS OK - wiki.wysteria.ga reverse DNS resolves to cp13.miraheze.org
[04:33:07] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[04:47:19] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[04:56:23] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[05:31:03] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[05:47:14] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[05:48:53] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.00, 1.96
[05:51:03] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.69, 2.95, 2.08
[06:22:10] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[06:31:18] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[06:45:16] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[07:21:59] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[07:56:49] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[08:06:05] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[08:40:50] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[08:57:10] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[09:11:03] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[09:22:04] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JG24h
[09:22:05] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega 03253637a - T7383: Undeploy ModernSkylight (#3933)
[09:22:07] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4
[09:23:08] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[09:23:43] <Universal_Omega>	 !log sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/modernskylight.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php modernskylight --disable
[09:23:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:26:51] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±4] 13https://git.io/JG2Rf
[09:26:52] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega 03f1c9189 - Merge branch 'master' into REL1_36 (#3934)
[09:30:09] <Universal_Omega>	 !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki && sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* / jbr* / test3
[09:30:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:33:44] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.61, 20.15, 14.20
[09:34:15] <Not-58c7>	 [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3
[09:34:33] <Not-58c7>	 [02mw-config] 07Universal-Omega closed pull request 03#3933: T7383: Undeploy ModernSkylight - 13https://git.io/JGzv1
[09:34:33] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[09:35:39] <Not-58c7>	 [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-1/±1] 13https://git.io/JG2ut
[09:35:41] <Not-58c7>	 [02miraheze/mediawiki] 07Universal-Omega 030babe1e - Remove ModernSkylight
[09:35:52] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.55, 20.31, 15.08
[09:36:39] <Not-58c7>	 [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-1/±1] 13https://git.io/JG2un
[09:36:40] <Not-58c7>	 [02miraheze/mediawiki] 07Universal-Omega 03247af0b - Remove ModernSkylight
[09:36:49] <Not-58c7>	 [02mw-config] 07Universal-Omega opened pull request 03#3934: Merge branch 'master' into REL1_36 - 13https://git.io/JG2uW
[09:37:02] <Not-58c7>	 [02mw-config] 07Universal-Omega closed pull request 03#3934: Merge branch 'master' into REL1_36 - 13https://git.io/JG2uW
[09:37:27] <Not-58c7>	 [02mw-config] 07Universal-Omega synchronize pull request 03#3824: Update config for 1.36 - 13https://git.io/JYjbk
[09:38:10] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[09:38:42] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[09:41:49] <Not-58c7>	 [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JG2zY
[09:41:50] <Not-58c7>	 [02miraheze/mediawiki] 07Universal-Omega 033f03d55 - Update Cosmos
[09:43:08] <Not-58c7>	 [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JG2zV
[09:43:09] <Not-58c7>	 [02miraheze/mediawiki] 07Universal-Omega 03b4c38d0 - Update Cosmos
[09:43:14] <icinga-miraheze>	 PROBLEM - en.phgalaxy.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:44:11] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JG2zH
[09:44:13] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega 0337e476f - Rename two Cosmos configs
[09:44:59] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JG2zA
[09:45:00] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega 03ce48711 - Update LocalSettings.php
[09:45:32] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JG2zj
[09:45:34] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega 03fa6a006 - Rename two Cosmos configs (#3935)
[09:45:35] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4
[09:46:55] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[09:46:55] <Universal_Omega>	 !log UPDATE mw_settings SET s_settings = REPLACE(s_settings,'wgCosmosProfileTagGroups', 'wgCosmosSocialProfileTagGroups');
[09:46:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:47:58] <Universal_Omega>	 !log UPDATE mw_settings SET s_settings = REPLACE(s_settings,'wgCosmosNumberofGroupTags', 'wgCosmosSocialProfileNumberofGroupTags');
[09:48:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:48:21] <Not-58c7>	 [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3
[09:48:27] <Not-58c7>	 [02mw-config] 07Universal-Omega opened pull request 03#3935: Rename two Cosmos configs - 13https://git.io/JG2gu
[09:49:04] <Not-58c7>	 [02mw-config] 07Universal-Omega synchronize pull request 03#3935: Rename two Cosmos configs - 13https://git.io/JG2gu
[09:49:15] <paladox>	 https://fosshost.org/news/freenode
[09:49:16] <Not-58c7>	 [02mw-config] 07Universal-Omega closed pull request 03#3935: Rename two Cosmos configs - 13https://git.io/JG2gu
[09:49:19] <paladox>	 RhinosF1
[09:49:22] <Not-58c7>	 [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3
[09:49:35] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[09:49:38] <RhinosF1>	 paladox: I'm very aware
[09:50:12] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[09:58:23] <icinga-miraheze>	 PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[09:58:38] <RhinosF1>	 Universal_Omega: ^
[09:59:53] <Universal_Omega>	 RhinosF1: I know, updated MediaWiki.
[10:00:32] <icinga-miraheze>	 RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[10:01:01] <RhinosF1>	 Ack
[10:05:29] <icinga-miraheze>	 PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[10:06:18] <icinga-miraheze>	 PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[10:06:18] <icinga-miraheze>	 PROBLEM - en.phgalaxy.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.phgalaxy.xyz reverse DNS resolves to a51062ecadbb5a26e.awsglobalaccelerator.com
[10:06:55] <icinga-miraheze>	 PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[10:07:15] <RhinosF1>	 Another expired domain
[10:07:26] <icinga-miraheze>	 PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[10:09:22] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[10:09:47] <icinga-miraheze>	 RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[10:10:26] <icinga-miraheze>	 RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[10:11:08] <Universal_Omega>	 !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* / jbr* / test3
[10:11:10] <icinga-miraheze>	 RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures
[10:11:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:11:40] <icinga-miraheze>	 RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[10:13:44] <Not-58c7>	 [02mw-config] 07Universal-Omega opened pull request 03#3936: Merge branch 'master' into REL1_36 - 13https://git.io/JG2rc
[10:14:15] <Not-58c7>	 [02mw-config] 07Universal-Omega closed pull request 03#3936: Merge branch 'master' into REL1_36 - 13https://git.io/JG2rc
[10:14:16] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±2] 13https://git.io/JG2ru
[10:14:18] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega 039db4da2 - Merge branch 'master' into REL1_36 (#3936)
[10:14:19] <Not-58c7>	 [02mw-config] 07Universal-Omega synchronize pull request 03#3824: Update config for 1.36 - 13https://git.io/JYjbk
[10:14:44] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[10:15:15] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[10:18:37] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[10:32:33] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[10:55:37] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[12:52:46] <icinga-miraheze>	 PROBLEM - jobrunner4 JobRunner Service on jobrunner4 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService'
[12:52:50] <icinga-miraheze>	 PROBLEM - mw10 JobRunner Service on mw10 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService'
[12:53:08] <RhinosF1>	 Fun
[12:53:20] <RhinosF1>	 Please recover
[12:53:35] <icinga-miraheze>	 PROBLEM - mw9 JobRunner Service on mw9 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService'
[12:53:47] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[12:54:12] <RhinosF1>	 No they a bit dead
[12:54:16] <RhinosF1>	 !sre got access?
[12:54:30] <icinga-miraheze>	 PROBLEM - mw11 JobRunner Service on mw11 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService'
[12:54:32] <icinga-miraheze>	 PROBLEM - mw8 JobRunner Service on mw8 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService'
[12:55:06] <RhinosF1>	 paladox: I'm in a meeting while 4. You free to restart the service via cumin?
[12:55:24] <paladox>	 ok
[12:55:47] <RhinosF1>	 mw* and jbr4 dead
[12:56:11] <paladox>	 that was me when i touched jobrunner3 redis config
[12:56:19] <paladox>	 fixed anyways, restarting servide
[12:56:21] <paladox>	 *service
[12:56:23] <RhinosF1>	 Okay
[12:56:28] <RhinosF1>	 At least we know a cause
[12:56:35] <icinga-miraheze>	 RECOVERY - mw11 JobRunner Service on mw11 is OK: PROCS OK: 1 process with args 'redisJobRunnerService'
[12:56:35] <icinga-miraheze>	 RECOVERY - mw8 JobRunner Service on mw8 is OK: PROCS OK: 1 process with args 'redisJobRunnerService'
[12:56:47] <icinga-miraheze>	 RECOVERY - mw10 JobRunner Service on mw10 is OK: PROCS OK: 1 process with args 'redisJobRunnerService'
[12:56:51] <icinga-miraheze>	 RECOVERY - jobrunner4 JobRunner Service on jobrunner4 is OK: PROCS OK: 1 process with args 'redisJobRunnerService'
[12:57:49] <icinga-miraheze>	 RECOVERY - mw9 JobRunner Service on mw9 is OK: PROCS OK: 1 process with args 'redisJobRunnerService'
[12:57:57] <RhinosF1>	 All green
[12:58:32] <RhinosF1>	 Dash looks good on grafana
[13:03:08] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[13:17:10] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[13:33:35] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[13:35:45] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.52, 6.44, 5.26
[13:37:51] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.77, 5.84, 5.17
[13:47:33] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[13:55:06] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 90%, RTA = 239.61 ms
[13:57:48] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw11 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[13:57:51] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb
[13:58:00] <icinga-miraheze>	 PROBLEM - cp3 Current Load on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[13:58:06] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for test3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[13:58:25] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw9 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[13:58:36] <icinga-miraheze>	 PROBLEM - cp3 PowerDNS Recursor on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[13:58:39] <icinga-miraheze>	 PROBLEM - cp3 HTTPS on cp3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:58:44] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb
[13:58:57] <RhinosF1>	 paladox: ^
[13:58:58] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw8 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[13:59:06] <paladox>	 hmm
[13:59:13] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[13:59:19] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw10 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[13:59:31] <icinga-miraheze>	 PROBLEM - cp3 APT on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[13:59:41] <RhinosF1>	 going to cp3.miraheze.org loss fine for me
[13:59:43] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[13:59:45] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mon2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[14:00:06] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[14:00:09] <icinga-miraheze>	 PROBLEM - cp3 NTP time on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[14:01:08] <icinga-miraheze>	 PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[14:04:12] <icinga-miraheze>	 RECOVERY - cp3 PowerDNS Recursor on cp3 is OK: DNS OK: 0.485 seconds response time. miraheze.org returns 2001:41d0:801:2000::58af,51.38.69.175
[14:04:24] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw8 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15284 bytes in 0.944 second response time
[14:04:34] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 20 minutes ago with 0 failures
[14:04:46] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw10 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15285 bytes in 1.083 second response time
[14:04:58] <icinga-miraheze>	 RECOVERY - cp3 APT on cp3 is OK: APT OK: 27 packages available for upgrade (0 critical updates).
[14:04:58] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 4879 MB (20% inode=94%);
[14:04:59] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mon2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 35473 bytes in 1.018 second response time
[14:06:06] <icinga-miraheze>	 RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 0.06, 0.04, 0.13
[14:06:07] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw11 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15285 bytes in 0.927 second response time
[14:06:24] <icinga-miraheze>	 RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 9%
[14:06:26] <icinga-miraheze>	 RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2260 bytes in 4.722 second response time
[14:06:29] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for test3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15286 bytes in 1.157 second response time
[14:06:32] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw9 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15284 bytes in 4.612 second response time
[14:07:08] <paladox>	 seems it recovered though i was sshd in
[14:07:17] <paladox>	 so must have been the link between cp3 and ovh
[14:07:26] <paladox>	 as it didn't go down for the other cp*s
[14:07:53] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy
[14:07:56] <icinga-miraheze>	 RECOVERY - cp3 NTP time on cp3 is OK: NTP OK: Offset -0.002756327391 secs
[14:08:02] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[14:08:42] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[14:08:45] <icinga-miraheze>	 RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 248.97 ms
[14:11:10] <RhinosF1>	 Yeah
[14:12:00] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[14:25:57] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[14:35:13] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[14:49:11] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[14:58:27] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[15:09:41] <icinga-miraheze>	 PROBLEM - wiki.graalmilitary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.graalmilitary.com could not be found
[15:10:45] <icinga-miraheze>	 PROBLEM - pt.graalmilitary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pt.graalmilitary.com could not be found
[15:12:29] <icinga-miraheze>	 PROBLEM - sarovia.graalmilitary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sarovia.graalmilitary.com could not be found
[15:12:29] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[15:23:38] <icinga-miraheze>	 PROBLEM - wiki.graalmilitary.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.graalmilitary.com reverse DNS resolves to premium102-5.web-hosting.com
[15:43:31] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.85, 6.47, 5.89
[15:44:09] <icinga-miraheze>	 PROBLEM - wiki.redeemer.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.redeemer.live could not be found
[15:44:14] <icinga-miraheze>	 PROBLEM - cubashpedia.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cubashpedia.cf could not be found
[15:44:14] <icinga-miraheze>	 PROBLEM - www.allthetropes.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.allthetropes.org could not be found
[15:44:19] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.98, 7.25, 6.24
[15:44:46] <icinga-miraheze>	 RECOVERY - wiki.graalmilitary.com - reverse DNS on sslhost is OK: rDNS OK - wiki.graalmilitary.com reverse DNS resolves to cp13.miraheze.org
[15:45:42] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.26, 6.33, 5.91
[15:46:26] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.33, 6.52, 6.09
[15:50:15] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.57, 7.94, 6.86
[15:50:30] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.32, 7.37, 6.51
[15:50:38] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.13, 8.63, 6.68
[15:50:58] <paladox>	 !log ran GDPR script for T&S (earlier)
[15:51:17] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.78, 7.81, 6.85
[15:51:17] <icinga-miraheze>	 RECOVERY - cubashpedia.cf - reverse DNS on sslhost is OK: rDNS OK - cubashpedia.cf reverse DNS resolves to cp13.miraheze.org
[15:51:19] <icinga-miraheze>	 RECOVERY - www.allthetropes.org - reverse DNS on sslhost is OK: rDNS OK - www.allthetropes.org reverse DNS resolves to cp13.miraheze.org
[15:51:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:51:27] <icinga-miraheze>	 RECOVERY - wiki.redeemer.live - reverse DNS on sslhost is OK: rDNS OK - wiki.redeemer.live reverse DNS resolves to cp13.miraheze.org
[15:52:51] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.42, 8.44, 7.08
[15:53:26] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.41, 6.69, 6.55
[15:55:01] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.15, 7.78, 7.01
[15:57:04] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.18, 7.08, 6.80
[15:57:46] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.58, 7.46, 6.84
[15:59:11] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.29, 8.72, 7.68
[15:59:16] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.93, 9.23, 7.66
[15:59:21] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.13, 9.09, 7.69
[15:59:55] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.02, 7.59, 7.00
[16:01:23] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.18, 7.41, 7.34
[16:01:24] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 3.92, 7.07, 7.06
[16:01:31] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.80, 7.41, 7.25
[16:01:57] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.05, 6.18, 6.56
[16:03:29] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.68, 6.35, 6.78
[16:03:36] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.59, 8.58, 7.77
[16:04:18] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[16:08:00] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.98, 6.90, 7.32
[16:10:03] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.00, 5.90, 6.64
[16:14:23] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.07, 5.43, 6.52
[16:15:34] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Databases db11 on bacula2 is CRITICAL: CRITICAL: Full, 442654 files, 36.11GB, 2021-05-02 16:15:00 (4.4 weeks ago)
[16:18:16] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[16:32:09] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.04, 6.83, 6.69
[16:34:19] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.10, 6.67, 6.66
[16:34:42] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.66, 6.91, 6.62
[16:35:14] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.12, 7.00, 6.31
[16:36:53] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.47, 6.21, 6.39
[16:37:14] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.24, 6.45, 6.20
[16:41:35] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[16:55:31] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[17:04:47] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[17:19:02] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[17:21:32] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.92, 6.75, 5.17
[17:22:17] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.00, 6.66, 5.47
[17:23:07] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.80, 8.54, 6.30
[17:23:39] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.48, 7.25, 5.54
[17:25:17] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.33, 6.95, 6.01
[17:25:43] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.79, 6.89, 5.62
[17:26:45] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.40, 6.20, 5.67
[17:27:23] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.35, 6.10, 5.82
[17:27:45] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.02, 6.18, 5.51
[17:28:19] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[17:35:06] <icinga-miraheze>	 PROBLEM - jobrunner4 APT on jobrunner4 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (2 critical updates).
[17:35:15] <icinga-miraheze>	 PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (2 critical updates).
[17:36:18] <icinga-miraheze>	 PROBLEM - jobrunner3 APT on jobrunner3 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (2 critical updates).
[17:36:26] <icinga-miraheze>	 PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (2 critical updates).
[17:36:38] <icinga-miraheze>	 PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (2 critical updates).
[17:36:42] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.49, 7.30, 6.36
[17:36:55] <icinga-miraheze>	 PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (2 critical updates).
[17:37:16] <icinga-miraheze>	 PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (2 critical updates).
[17:37:53] <icinga-miraheze>	 PROBLEM - www.pornwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.pornwiki.org' expires in 15 day(s) (Fri 18 Jun 2021 17:30:52 GMT +0000).
[17:38:54] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.76, 7.42, 6.53
[17:40:59] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.62, 6.78, 6.41
[17:42:20] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[17:45:28] <Not-58c7>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JGahY
[17:45:29] <Not-58c7>	 [02miraheze/ssl] 07MirahezeSSLBot 0317bb579 - Bot: Update SSL cert for www.pornwiki.org
[17:51:36] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[17:53:01] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Databases db13 on bacula2 is CRITICAL: CRITICAL: Full, 279527 files, 66.56GB, 2021-05-02 17:53:00 (4.4 weeks ago)
[17:58:40] <icinga-miraheze>	 RECOVERY - www.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.pornwiki.org' will expire on Tue 31 Aug 2021 16:45:22 GMT +0000.
[18:05:39] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[18:06:06] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.82, 7.22, 6.48
[18:08:17] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.25, 6.69, 6.38
[18:36:01] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[18:50:00] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[18:59:13] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[19:34:01] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[19:43:13] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[19:57:14] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[20:04:34] <icinga-miraheze>	 PROBLEM - indiancannabis.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.22491845965632:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1544:SSL alert number 40
[20:06:38] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[20:08:10] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JGV4O
[20:08:12] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega 03f5a148e - Update LocalSettings.php
[20:08:13] <Not-58c7>	 [02mw-config] 07Universal-Omega synchronize pull request 03#3926: Set WebChat to default to Libera - 13https://git.io/JGY3k
[20:09:13] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[20:09:24] <Not-58c7>	 [02puppet] 07Universal-Omega synchronize pull request 03#1764: add kiwiirc.com to CSP Whitelist for WebChat extension - 13https://git.io/JGsTZ
[20:09:33] <Not-58c7>	 [02puppet] 07Universal-Omega edited pull request 03#1764: add web.libera.chat to CSP Whitelist for WebChat extension - 13https://git.io/JGsTZ
[20:14:50] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JGV4x
[20:14:52] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega 0358a0123 - Update LocalSettings.php
[20:14:53] <Not-58c7>	 [02mw-config] 07Universal-Omega synchronize pull request 03#3926: Set WebChat to default to Libera - 13https://git.io/JGY3k
[20:15:50] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[20:15:56] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JGVBk
[20:15:57] <Not-58c7>	 [02miraheze/mw-config] 07Universal-Omega 03f3c904b - Update ManageWikiSettings.php
[20:15:59] <Not-58c7>	 [02mw-config] 07Universal-Omega synchronize pull request 03#3926: Set WebChat to default to Libera - 13https://git.io/JGY3k
[20:17:01] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[20:40:20] <icinga-miraheze>	 PROBLEM - indiancannabis.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - indiancannabis.wiki reverse DNS resolves to ec2-44-227-76-166.us-west-2.compute.amazonaws.com
[21:02:25] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[21:05:07] <icinga-miraheze>	 PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.06, 1.90, 1.57
[21:07:15] <icinga-miraheze>	 RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.17, 1.62, 1.50
[21:18:39] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[21:30:06] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.93, 7.14, 6.53
[21:34:24] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.99, 6.77, 6.54
[21:39:44] <Not-58c7>	 [02mw-config] 07BukkitIsEnchanted opened pull request 03#3937: Creation SECURITY.md - 13https://git.io/JGVww
[21:40:46] <miraheze-github>	 miraheze/mw-config - BukkitIsEnchanted the build passed.
[21:41:51] <Not-58c7>	 [02mw-config] 07BukkitIsEnchanted edited pull request 03#3937: Creation SECURITY.md - 13https://git.io/JGVww
[21:43:24] <Not-58c7>	 [02mw-config] 07BukkitIsEnchanted edited pull request 03#3937: Creation of SECURITY.md - 13https://git.io/JGVww
[21:53:35] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[21:55:53] <Not-58c7>	 [02mw-config] 07Universal-Omega commented on pull request 03#3937: Creation of SECURITY.md - 13https://git.io/JGVrb
[21:57:40] <MH-Discord>	 <Lake> I know this won't happen, but if Miraheze made its own task manager, it could be a Task extension for MW, lol
[22:09:50] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[22:10:04] <Not-58c7>	 [02mw-config] 07BukkitIsEnchanted commented on pull request 03#3937: Creation of SECURITY.md - 13https://git.io/JGVKm
[22:10:06] <Not-58c7>	 [02mw-config] 07BukkitIsEnchanted closed pull request 03#3937: Creation of SECURITY.md - 13https://git.io/JGVww
[22:23:52] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[22:40:02] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[23:05:36] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.97, 6.39, 5.34
[23:07:38] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.43, 5.53, 5.16
[23:14:44] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[23:17:35] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.93, 6.63, 5.75
[23:19:43] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.29, 6.59, 5.86
[23:31:06] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[23:33:08] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.62, 7.98, 6.89
[23:33:35] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.07, 7.08, 6.44
[23:36:36] <icinga-miraheze>	 PROBLEM - cyberlaw.ccdcoe.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cyberlaw.ccdcoe.org could not be found
[23:36:36] <icinga-miraheze>	 PROBLEM - www.lab612.at - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.lab612.at could not be found
[23:39:42] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.36, 7.54, 7.14
[23:40:01] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.97, 7.73, 6.90
[23:43:29] <icinga-miraheze>	 RECOVERY - cyberlaw.ccdcoe.org - reverse DNS on sslhost is OK: rDNS OK - cyberlaw.ccdcoe.org reverse DNS resolves to cp13.miraheze.org
[23:43:29] <icinga-miraheze>	 RECOVERY - www.lab612.at - reverse DNS on sslhost is OK: rDNS OK - www.lab612.at reverse DNS resolves to cp13.miraheze.org
[23:44:14] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.46, 6.35, 6.57
[23:45:04] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[23:46:09] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.84, 6.36, 6.73
[23:56:04] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.85, 6.83, 6.52
[23:58:08] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.03, 6.31, 6.36