[00:02:41] <icinga-miraheze>	 RECOVERY - cp12 Disk Space on cp12 is OK: DISK OK - free space: / 8715 MB (22% inode=96%);
[00:02:47] <icinga-miraheze>	 PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:05:02] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.85, 6.54, 5.15
[00:05:56] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.33, 6.83, 5.46
[00:07:02] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.42, 6.01, 5.13
[00:07:56] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.20, 6.39, 5.46
[00:16:07] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[00:29:28] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[00:30:45] <icinga-miraheze>	 RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures
[00:40:43] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.75, 6.90, 6.10
[00:41:04] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.53, 6.98, 5.96
[00:42:43] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.13, 7.33, 6.35
[00:47:57] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.86, 6.89, 5.99
[00:50:43] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.97, 7.99, 7.14
[00:51:04] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.45, 6.77, 6.41
[00:51:30] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[00:51:57] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.27, 6.61, 6.09
[00:52:43] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.55, 8.38, 7.37
[00:58:43] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.46, 7.54, 7.42
[01:02:43] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.60, 7.73, 7.47
[01:08:43] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.74, 7.23, 7.45
[01:14:43] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.67, 5.62, 6.66
[02:03:42] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.99, 6.16, 5.19
[02:05:40] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.17, 5.93, 5.23
[02:34:07] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[02:36:43] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.13, 6.90, 5.95
[02:40:43] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.07, 6.32, 5.95
[02:43:07] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[03:16:26] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[03:25:07] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[03:38:27] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[04:33:53] <icinga-miraheze>	 PROBLEM - mw9 Disk Space on mw9 is WARNING: DISK WARNING - free space: / 1941 MB (10% inode=70%);
[05:17:33] <icinga-miraheze>	 PROBLEM - wiki.tigcity.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.tigcity.tk' expires in 15 day(s) (Fri 25 Jun 2021 05:14:26 GMT +0000).
[05:22:33] <Not-4fc0>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZs5L
[05:22:35] <Not-4fc0>	 [02miraheze/ssl] 07MirahezeSSLBot 03f821424 - Bot: Update SSL cert for wiki.tigcity.tk
[05:31:13] <icinga-miraheze>	 PROBLEM - famedata.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'famedata.org' expires in 15 day(s) (Fri 25 Jun 2021 05:27:49 GMT +0000).
[05:33:34] <Not-4fc0>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZsb0
[05:33:35] <Not-4fc0>	 [02miraheze/ssl] 07MirahezeSSLBot 0386f0e8c - Bot: Update SSL cert for famedata.org
[05:39:40] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[05:47:15] <icinga-miraheze>	 PROBLEM - www.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.petrawiki.org' expires in 15 day(s) (Fri 25 Jun 2021 05:43:40 GMT +0000).
[05:48:03] <icinga-miraheze>	 PROBLEM - en.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.petrawiki.org' expires in 15 day(s) (Fri 25 Jun 2021 05:43:40 GMT +0000).
[05:51:03] <Not-4fc0>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZsxO
[05:51:05] <Not-4fc0>	 [02miraheze/ssl] 07MirahezeSSLBot 03e4d76f5 - Bot: Update SSL cert for www.petrawiki.org
[05:52:59] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[05:54:16] <icinga-miraheze>	 PROBLEM - files.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'files.petrawiki.org' expires in 15 day(s) (Fri 25 Jun 2021 05:46:50 GMT +0000).
[05:55:45] <Not-4fc0>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZsxD
[05:55:46] <Not-4fc0>	 [02miraheze/ssl] 07MirahezeSSLBot 0346ef4ac - Bot: Update SSL cert for en.petrawiki.org
[05:58:16] <icinga-miraheze>	 RECOVERY - famedata.org - LetsEncrypt on sslhost is OK: OK - Certificate 'famedata.org' will expire on Tue 07 Sep 2021 04:33:27 GMT +0000.
[05:59:30] <icinga-miraheze>	 RECOVERY - wiki.tigcity.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tigcity.tk' will expire on Tue 07 Sep 2021 04:22:27 GMT +0000.
[06:00:23] <Not-4fc0>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZspC
[06:00:25] <Not-4fc0>	 [02miraheze/ssl] 07MirahezeSSLBot 03b574337 - Bot: Update SSL cert for files.petrawiki.org
[06:01:46] <icinga-miraheze>	 PROBLEM - petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.petrawiki.org' expires in 15 day(s) (Fri 25 Jun 2021 05:54:36 GMT +0000).
[06:04:47] <icinga-miraheze>	 PROBLEM - translate.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'translate.petrawiki.org' expires in 15 day(s) (Fri 25 Jun 2021 05:57:27 GMT +0000).
[06:06:54] <Not-4fc0>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZsh0
[06:06:55] <Not-4fc0>	 [02miraheze/ssl] 07MirahezeSSLBot 03e39e761 - Bot: Update SSL cert for translate.petrawiki.org
[06:14:57] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[06:28:47] <icinga-miraheze>	 RECOVERY - en.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.petrawiki.org' will expire on Tue 07 Sep 2021 04:55:39 GMT +0000.
[06:29:05] <icinga-miraheze>	 RECOVERY - petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.petrawiki.org' will expire on Tue 07 Sep 2021 04:50:56 GMT +0000.
[06:31:41] <icinga-miraheze>	 RECOVERY - translate.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'translate.petrawiki.org' will expire on Tue 07 Sep 2021 05:06:47 GMT +0000.
[06:34:15] <icinga-miraheze>	 RECOVERY - www.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.petrawiki.org' will expire on Tue 07 Sep 2021 04:55:39 GMT +0000.
[06:34:30] <icinga-miraheze>	 RECOVERY - files.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'files.petrawiki.org' will expire on Tue 07 Sep 2021 05:00:17 GMT +0000.
[06:54:55] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[07:30:10] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[08:03:29] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[08:37:26] <icinga-miraheze>	 PROBLEM - hr.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hr.petrawiki.org' expires in 15 day(s) (Fri 25 Jun 2021 08:34:38 GMT +0000).
[08:38:07] <Not-4fc0>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZGVP
[08:38:09] <Not-4fc0>	 [02miraheze/ssl] 07MirahezeSSLBot 03b33deda - Bot: Update SSL cert for hr.petrawiki.org
[08:47:07] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[09:00:26] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[09:04:33] <icinga-miraheze>	 RECOVERY - hr.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'hr.petrawiki.org' will expire on Tue 07 Sep 2021 07:38:00 GMT +0000.
[09:15:42] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[09:29:05] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[10:39:37] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[10:52:54] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[11:08:11] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[11:21:30] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[13:20:31] <icinga-miraheze>	 PROBLEM - adadevelopersacademy.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for adadevelopersacademy.wiki could not be found
[13:20:32] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found
[13:20:32] <icinga-miraheze>	 PROBLEM - orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for orain.org could not be found
[13:20:32] <icinga-miraheze>	 PROBLEM - wiki.wysteria.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wysteria.ga could not be found
[13:20:35] <icinga-miraheze>	 PROBLEM - miraheze.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.ga could not be found
[13:20:35] <icinga-miraheze>	 PROBLEM - erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for erikapedia.com could not be found
[13:22:44] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[13:27:12] <icinga-miraheze>	 RECOVERY - adadevelopersacademy.wiki - reverse DNS on sslhost is OK: rDNS OK - adadevelopersacademy.wiki reverse DNS resolves to cp13.miraheze.org
[13:27:15] <icinga-miraheze>	 RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: rDNS OK - franchise.franchising.org.ua reverse DNS resolves to cp13.miraheze.org
[13:27:15] <icinga-miraheze>	 RECOVERY - orain.org - reverse DNS on sslhost is OK: rDNS OK - orain.org reverse DNS resolves to cp13.miraheze.org
[13:27:21] <icinga-miraheze>	 RECOVERY - miraheze.ga - reverse DNS on sslhost is OK: rDNS OK - miraheze.ga reverse DNS resolves to cp13.miraheze.org
[13:27:22] <icinga-miraheze>	 RECOVERY - erikapedia.com - reverse DNS on sslhost is OK: rDNS OK - erikapedia.com reverse DNS resolves to cp13.miraheze.org
[13:27:32] <icinga-miraheze>	 RECOVERY - wiki.wysteria.ga - reverse DNS on sslhost is OK: rDNS OK - wiki.wysteria.ga reverse DNS resolves to cp13.miraheze.org
[13:47:22] <icinga-miraheze>	 PROBLEM - dreamsit.com.br - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dreamsit.com.br could not be found
[13:49:17] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[14:04:33] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[14:17:53] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[14:33:09] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[14:43:23] <icinga-miraheze>	 RECOVERY - dreamsit.com.br - reverse DNS on sslhost is OK: rDNS OK - dreamsit.com.br reverse DNS resolves to cp13.miraheze.org
[14:46:29] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[15:06:31] <MH-Discord>	 <R4356th> Did something regarding caching change recently? Load time is terrible.
[15:10:01] <RhinosF1>	 Not that I'm aware of
[15:10:13] <RhinosF1>	 When did it start being an issue?
[15:10:59] <MH-Discord>	 <R4356th> About 5 hours ago, I believe.
[15:14:00] <RhinosF1>	 Don't see anything
[15:48:24] <icinga-miraheze>	 PROBLEM - dreamsit.com.br - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dreamsit.com.br could not be found
[15:57:08] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[16:04:43] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.42, 7.81, 6.53
[16:06:27] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.29, 6.62, 5.90
[16:06:43] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.24, 7.16, 6.44
[16:08:24] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.31, 6.41, 5.91
[16:08:43] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.15, 7.70, 6.69
[16:10:43] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.59, 6.97, 6.56
[16:12:43] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.92, 6.14, 6.30
[16:30:24] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[16:38:59] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[16:52:19] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[17:03:10] <icinga-miraheze>	 PROBLEM - db1.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for db1.miraheze.wiki could not be found
[17:03:17] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 17.70, 11.47, 7.12
[17:03:40] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb
[17:04:04] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 17.42, 11.62, 7.32
[17:04:44] <icinga-miraheze>	 PROBLEM - wiki.slimeapedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.slimeapedia.org could not be found
[17:04:49] <icinga-miraheze>	 PROBLEM - wiki.kourouklides.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.kourouklides.com could not be found
[17:05:34] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.28, 9.07, 6.76
[17:05:39] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[17:05:59] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.90, 7.85, 6.38
[17:07:58] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.94, 8.67, 6.87
[17:09:52] <icinga-miraheze>	 RECOVERY - db1.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - db1.miraheze.wiki reverse DNS resolves to cp13.miraheze.org
[17:09:56] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.31, 7.02, 6.49
[17:11:06] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 3.44, 7.42, 7.30
[17:11:23] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.99, 6.79, 6.61
[17:11:29] <icinga-miraheze>	 RECOVERY - wiki.kourouklides.com - reverse DNS on sslhost is OK: rDNS OK - wiki.kourouklides.com reverse DNS resolves to cp13.miraheze.org
[17:11:41] <icinga-miraheze>	 RECOVERY - wiki.slimeapedia.org - reverse DNS on sslhost is OK: rDNS OK - wiki.slimeapedia.org reverse DNS resolves to cp13.miraheze.org
[17:11:55] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.97, 5.90, 6.14
[17:11:57] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.22, 6.94, 6.98
[17:14:17] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[17:15:58] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 2.99, 5.60, 6.50
[17:17:02] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.70, 5.67, 6.63
[17:20:16] <RhinosF1>	 paladox: I just had a quick glance at grafana and db13 just had quite a load spike
[17:20:54] <paladox>	 i see nothing in https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=db13.miraheze.org&var-port=9100 ?
[17:21:23] <RhinosF1>	 paladox: not it jumping to 2 towards to the end?
[17:21:45] <paladox>	 that doesn't look like a load spike seeing as it went to ~1.5 quite often.
[17:34:15] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[17:54:43] <Not-4fc0>	 [02miraheze/WikiDiscover] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JZnKY
[17:54:45] <Not-4fc0>	 [02miraheze/WikiDiscover] 07paladox 03c40533c - Fix "PHP Notice: Undefined index: "
[17:54:46] <Not-4fc0>	 [02WikiDiscover] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vhUAp
[17:54:48] <Not-4fc0>	 [02WikiDiscover] 07paladox opened pull request 03#48: Fix "PHP Notice: Undefined index: " - 13https://git.io/JZnK3
[17:55:48] <miraheze-github>	 miraheze/WikiDiscover - paladox the build passed.
[18:00:15] <Not-4fc0>	 [02WikiDiscover] 07paladox closed pull request 03#48: Fix "PHP Notice: Undefined index: " - 13https://git.io/JZnK3
[18:00:17] <Not-4fc0>	 [02miraheze/WikiDiscover] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZn6z
[18:00:18] <Not-4fc0>	 [02miraheze/WikiDiscover] 07paladox 039d8490d - Fix "PHP Notice: Undefined index: " (#48)
[18:00:20] <Not-4fc0>	 [02WikiDiscover] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vhUAp
[18:00:21] <Not-4fc0>	 [02miraheze/WikiDiscover] 07paladox deleted branch 03paladox-patch-1
[18:01:12] <miraheze-github>	 miraheze/WikiDiscover - paladox the build passed.
[18:31:23] <Not-4fc0>	 [02WikiDiscover] 07JohnFLewis reviewed pull request 03#48 commit - 13https://git.io/JZnyX
[18:31:27] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[18:34:03] <JohnLewis>	 paladox: ^
[18:34:25] <paladox>	 Changing, thanks!
[18:35:28] <Not-4fc0>	 [02miraheze/WikiDiscover] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JZnS2
[18:35:30] <Not-4fc0>	 [02miraheze/WikiDiscover] 07paladox 03b5d4128 - Say Uncategorised if the category doesn't exist
[18:35:31] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.50, 6.88, 5.27
[18:35:31] <Not-4fc0>	 [02WikiDiscover] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vhUAp
[18:35:33] <Not-4fc0>	 [02WikiDiscover] 07paladox opened pull request 03#49: Say Uncategorised if the category doesn't exist - 13https://git.io/JZnSa
[18:36:05] <Not-4fc0>	 [02WikiDiscover] 07paladox closed pull request 03#49: Say Uncategorised if the category doesn't exist - 13https://git.io/JZnSa
[18:36:06] <Not-4fc0>	 [02miraheze/WikiDiscover] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZnSM
[18:36:07] <paladox>	 JohnLewis: ^
[18:36:08] <Not-4fc0>	 [02miraheze/WikiDiscover] 07paladox 035d214ef - Say Uncategorised if the category doesn't exist (#49)
[18:36:31] <miraheze-github>	 miraheze/WikiDiscover - paladox the build passed.
[18:37:02] <miraheze-github>	 miraheze/WikiDiscover - paladox the build passed.
[18:37:27] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.44, 6.17, 5.20
[18:38:07] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[19:20:07] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[19:53:26] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[20:02:02] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[20:06:28] <RhinosF1>	 paladox: do we keep db error logs?
[20:07:44] <paladox>	 we keep a mysql-error log on the db server, if you mean mediawiki then  maybe the mediawiki error logs
[20:08:52] <paladox>	 anyways i'm going to go
[20:18:48] <Southparkfan>	 paladox: can I talk to you tomorrow? there are multiple high priority tasks on the infra board
[20:19:02] <Southparkfan>	 (and I'm going to look at one right away)
[20:19:10] <paladox>	 sure
[20:22:59] <Not-4fc0>	 [02puppet] 07Universal-Omega opened pull request 03#1776: Add cron for updateSpecialPages.php - 13https://git.io/JZcUr
[20:32:45] <Not-4fc0>	 [02puppet] 07Universal-Omega synchronize pull request 03#1776: Add cron for updateSpecialPages.php - 13https://git.io/JZcUr
[20:33:40] <Not-4fc0>	 [02puppet] 07Universal-Omega synchronize pull request 03#1776: Add cron for updateSpecialPages.php - 13https://git.io/JZcUr
[20:35:23] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[20:37:29] <Not-4fc0>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JZcI6
[20:37:31] <Not-4fc0>	 [02miraheze/mw-config] 07Universal-Omega 03791fa1f - Set wgMiserMode to true
[20:37:32] <Not-4fc0>	 [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3
[20:37:34] <Not-4fc0>	 [02mw-config] 07Universal-Omega opened pull request 03#3945: Set wgMiserMode to true - 13https://git.io/JZcIP
[20:38:34] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[20:40:19] <Southparkfan>	 Universal_Omega: re wgMiserMode, any metrics on the performance improvements? _where_ do you see the load time difference?
[20:42:36] * dmehus thinks he remembers someone, possibly JohnLewis, talking about `$wgMiserMode` one time
[20:42:37] <Southparkfan>	 miser mode may help with reducing the database load, but running a script to update the special pages creates database load and mediawiki server load too, hence I'm curious what the trade-offs of each option are
[20:43:25] <Universal_Omega>	 Southparkfan: For example proxing from test3 to allthetropeswiki, I noticed a significant decrease of load time (some overall) but especially on special pages like Special:WantedPages and other expensive query pages. (Which is most of what would really be effected by this change anyways) but that's true about running the script.
[20:44:03] <Universal_Omega>	 Special:WantedPages took 30-45 seconds to load without and <5 seconds with it set.
[20:44:10] <Southparkfan>	 is there a decrease in response time for articles (or any page that's not a special page)?
[20:44:23] <dmehus>	 I guess if we implemented this, we'd want to run a test pattern for a week or two and see if there are measurable differences and what the tradeoffs are?
[20:45:07] <Universal_Omega>	 Southparkfan: Some yes. (Depending on loaded extensions and used skin also)
[20:45:32] <Southparkfan>	 do you have URLs?
[20:46:23] <Southparkfan>	 would be nice to load articles even quicker
[20:47:42] <Universal_Omega>	 Southparkfan: No, I'm mobile now. But alot of the performance testing was on allthetropeswiki also some on my own dcmultiversewiki.
[20:48:24] <Southparkfan>	 as far as I can recall miser mode caches the results of special page queries
[20:49:12] <Southparkfan>	 if there are extensions performing such queries while someone is loading an article...
[20:58:40] <Universal_Omega>	 Southparkfan: yeah, it may have just been local cache causing the noticeable load differences on normal articles now that I think about it. But I do still think setting it is useful for when visiting the query special pages, which is what it's meant for.
[21:10:38] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[21:12:15] <Not-4fc0>	 [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JZcYX
[21:12:17] <Not-4fc0>	 [02miraheze/mw-config] 07RhinosF1 039f699ec - Add my Staff account
[21:12:18] <Not-4fc0>	 [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3
[21:12:20] <Not-4fc0>	 [02mw-config] 07RhinosF1 opened pull request 03#3946: Add my Staff account - 13https://git.io/JZcYD
[21:12:33] <RhinosF1>	 Universal_Omega: mind a +1 ^
[21:13:15] <miraheze-github>	 miraheze/mw-config - RhinosF1 the build passed.
[21:23:00] <RhinosF1>	 Southparkfan: if you have access to graylog, was there any db errors at 17:04:03
[21:23:19] <RhinosF1>	 hopefully on cplostwikiswiki
[21:23:19] <Southparkfan>	 UTC?
[21:23:30] <RhinosF1>	 server time so ye
[21:23:55] <RhinosF1>	 I got mediawiki errors in https://graylog.miraheze.org/search?q=mediawiki_wiki%3Acplostwikiswiki&rangetype=absolute&from=2021-06-07T17%3A04%3A00.274Z&to=2021-06-09T17%3A06%3A59.274Z but they don't say why it failed
[21:23:57] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[21:34:18] <icinga-miraheze>	 PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (2 critical updates).
[21:35:05] <icinga-miraheze>	 PROBLEM - jobrunner3 APT on jobrunner3 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (2 critical updates).
[21:35:19] <icinga-miraheze>	 PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (2 critical updates).
[21:35:26] <Southparkfan>	 I have added more info to https://phabricator.miraheze.org/T7434#148571
[21:35:36] <icinga-miraheze>	 PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (2 critical updates).
[21:35:52] <icinga-miraheze>	 PROBLEM - jobrunner4 APT on jobrunner4 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (2 critical updates).
[21:36:22] <icinga-miraheze>	 PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (2 critical updates).
[21:36:30] <icinga-miraheze>	 PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (2 critical updates).
[21:36:44] <RhinosF1>	 Southparkfan: it could be. I'm not sure why it would cause DB error though
[21:40:00] <Southparkfan>	 seems to me one of the queries (transactions) failed, hence we couldn't commit
[21:40:13] <Southparkfan>	 but you need to know which query failed
[21:42:22] <Southparkfan>	 Uncancelable atomic section canceled (got MediaWiki\Revision\RevisionStore::insertRevisionOn)
[21:44:41] <RhinosF1>	 that's poor handling then
[21:51:35] <Not-4fc0>	 [02mw-config] 07RhinosF1 closed pull request 03#3946: Add my Staff account - 13https://git.io/JZcYD
[21:51:37] <Not-4fc0>	 [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JZcnj
[21:51:38] <Not-4fc0>	 [02miraheze/mw-config] 07RhinosF1 03f2c6334 - Add my Staff account (#3946)
[21:51:40] <Not-4fc0>	 [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-1
[21:51:41] <Not-4fc0>	 [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3
[21:52:33] <miraheze-github>	 miraheze/mw-config - RhinosF1 the build passed.
[21:56:18] <Universal_Omega>	 Southparkfan: do you think we can try wgMiserMode for say a week and if issues arise we can revert or what do you recommend in that?
[22:00:21] <Southparkfan>	 sorry, cannot comment now, urgent IRL issue
[22:40:08] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[22:53:28] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[23:02:07] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[23:15:27] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
[23:24:01] <icinga-miraheze>	 PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
[23:27:58] <Southparkfan>	 Universal_Omega: if the other SREs agree, sure
[23:28:39] <Universal_Omega>	 Southparkfan: Sounds good. Thanks!
[23:30:46] <Southparkfan>	 although the change will have negative impact on the users (pages outdated by X hours, removal of query intensive functionality) hence I recommend deploying to a limited number of wikis first (wikis with communities that don't object to such a change?)
[23:53:50] <dmehus>	 Southparkfan, I wouldn't object to trying it on `metawiki`. That seems like a reasonable test pattern approach to take
[23:54:47] <dmehus>	 I'd maybe just ask John for this thoughts first, though