[00:01:53] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.41, 19.66, 19.85
[00:02:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.19, 22.73, 21.86
[00:05:44] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.76, 21.57, 20.70
[00:05:57] <icinga-miraheze>	 PROBLEM - evil.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'evil.wiki' expires in 15 day(s) (Sun 28 Jul 2024 11:55:32 PM GMT +0000).
[00:06:11] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bb0f139c1691...f783b3315fca
[00:06:13] <botifico>	 [02ssl] 07WikiTideSSLBot 03f783b33 - Bot: Update SSL cert for evil.wiki
[00:06:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.14, 22.18, 21.92
[00:08:44] <icinga-miraheze>	 PROBLEM - wiki.evscope.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.evscope.org' expires in 15 day(s) (Sun 28 Jul 2024 11:39:40 PM GMT +0000).
[00:08:56] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f783b3315fca...21173b3fa85c
[00:08:57] <botifico>	 [02ssl] 07WikiTideSSLBot 0321173b3 - Bot: Update SSL cert for wiki.evscope.org
[00:12:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.95, 21.88, 21.61
[00:14:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.69, 21.89, 21.69
[00:15:31] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.62, 22.70, 21.60
[00:16:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.50, 23.48, 22.26
[00:17:18] <icinga-miraheze>	 RECOVERY - legacygt.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'legacygt.wiki' will expire on Thu 10 Oct 2024 10:18:15 PM GMT +0000.
[00:19:31] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.02, 23.39, 22.24
[00:20:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 17.01, 21.80, 22.04
[00:24:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.76, 23.29, 22.49
[00:26:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.26, 22.76, 22.45
[00:31:43] <icinga-miraheze>	 PROBLEM - wiki.kachow.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.kachow.org' expires in 15 day(s) (Mon 29 Jul 2024 12:18:45 AM GMT +0000).
[00:31:56] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/21173b3fa85c...8107b567f2ec
[00:31:57] <botifico>	 [02ssl] 07WikiTideSSLBot 038107b56 - Bot: Update SSL cert for wiki.kachow.org
[00:33:43] <icinga-miraheze>	 PROBLEM - wiki.recaptime.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.recaptime.eu.org' expires in 15 day(s) (Mon 29 Jul 2024 12:10:01 AM GMT +0000).
[00:33:54] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8107b567f2ec...2b254cbe123d
[00:33:56] <botifico>	 [02ssl] 07WikiTideSSLBot 032b254cb - Bot: Update SSL cert for wiki.recaptime.eu.org
[00:35:24] <icinga-miraheze>	 RECOVERY - evil.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'evil.wiki' will expire on Thu 10 Oct 2024 11:06:03 PM GMT +0000.
[00:37:57] <icinga-miraheze>	 RECOVERY - wiki.evscope.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.evscope.org' will expire on Thu 10 Oct 2024 11:08:50 PM GMT +0000.
[00:38:19] <icinga-miraheze>	 PROBLEM - wiki.veloren.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.veloren.net' expires in 15 day(s) (Mon 29 Jul 2024 12:33:08 AM GMT +0000).
[00:38:20] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 10.97, 16.71, 19.95
[00:38:31] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2b254cbe123d...d367b6b68894
[00:38:32] <botifico>	 [02ssl] 07WikiTideSSLBot 03d367b6b - Bot: Update SSL cert for wiki.veloren.net
[00:39:31] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 10.68, 16.76, 19.52
[00:39:35] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[00:41:36] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [58 system event log (SEL) entries present]
[00:43:31] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.01, 19.23, 19.86
[00:45:31] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.12, 19.05, 19.74
[00:46:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.97, 21.83, 20.45
[00:48:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.85, 21.06, 20.35
[00:50:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.75, 23.04, 21.15
[00:53:31] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.89, 21.18, 20.58
[00:55:31] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 19.01, 20.00, 20.19
[00:58:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.36, 23.86, 22.66
[01:01:27] <icinga-miraheze>	 RECOVERY - wiki.kachow.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kachow.org' will expire on Thu 10 Oct 2024 11:31:49 PM GMT +0000.
[01:02:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.32, 24.36, 22.98
[01:02:26] <icinga-miraheze>	 RECOVERY - wiki.recaptime.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.recaptime.eu.org' will expire on Thu 10 Oct 2024 11:33:48 PM GMT +0000.
[01:04:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.85, 23.84, 22.96
[01:06:40] <icinga-miraheze>	 PROBLEM - mariopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mariopedia.org' expires in 15 day(s) (Mon 29 Jul 2024 12:45:18 AM GMT +0000).
[01:06:52] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d367b6b68894...7746e23b1bfb
[01:06:53] <botifico>	 [02ssl] 07WikiTideSSLBot 037746e23 - Bot: Update SSL cert for mariopedia.org
[01:07:09] <icinga-miraheze>	 RECOVERY - wiki.veloren.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.veloren.net' will expire on Thu 10 Oct 2024 11:38:25 PM GMT +0000.
[01:07:16] <botifico>	 [02puppet] 07The-Voidwalker closed pull request 03#3879: Use xkey to purge all resources with the same host + url - 13https://github.com/miraheze/puppet/pull/3879
[01:07:19] <botifico>	 [02puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/706206b92cb4...05cd1ddf44e2
[01:07:20] <botifico>	 [02puppet] 07The-Voidwalker 0305cd1dd - Use xkey to purge all resources with the same host + url (#3879)
[01:07:21] <botifico>	 [02puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1
[01:07:24] <botifico>	 [02puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 - 13https://github.com/miraheze/puppet
[01:10:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.22, 23.72, 23.12
[01:12:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.17, 22.17, 22.65
[01:21:46] <icinga-miraheze>	 PROBLEM - persist.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'persist.wiki' expires in 15 day(s) (Mon 29 Jul 2024 12:53:32 AM GMT +0000).
[01:21:58] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7746e23b1bfb...174b54d9dc47
[01:21:59] <botifico>	 [02ssl] 07WikiTideSSLBot 03174b54d - Bot: Update SSL cert for persist.wiki
[01:22:20] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.89, 17.05, 19.88
[01:26:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.48, 20.53, 20.63
[01:26:29] <icinga-miraheze>	 PROBLEM - wiki.aoemods.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.aoemods.com' expires in 15 day(s) (Mon 29 Jul 2024 01:15:03 AM GMT +0000).
[01:26:41] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/174b54d9dc47...38c2d3ac7c4f
[01:26:42] <botifico>	 [02ssl] 07WikiTideSSLBot 0338c2d3a - Bot: Update SSL cert for wiki.aoemods.com
[01:27:31] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.37, 22.70, 17.95
[01:28:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.64, 21.88, 21.08
[01:29:31] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.57, 21.98, 19.34
[01:29:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.28, 20.93, 16.05
[01:30:34] <icinga-miraheze>	 PROBLEM - swests.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'swests.org' expires in 15 day(s) (Mon 29 Jul 2024 01:08:25 AM GMT +0000).
[01:30:46] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/38c2d3ac7c4f...627652722016
[01:30:49] <botifico>	 [02ssl] 07WikiTideSSLBot 036276527 - Bot: Update SSL cert for swests.org
[01:31:24] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.80, 23.68, 19.52
[01:31:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.13, 21.03, 16.64
[01:33:21] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.45, 23.93, 20.09
[01:35:19] <icinga-miraheze>	 RECOVERY - mariopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'mariopedia.org' will expire on Fri 11 Oct 2024 12:06:46 AM GMT +0000.
[01:35:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.51, 22.61, 18.19
[01:37:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.03, 23.11, 18.95
[01:38:47] <icinga-miraheze>	 PROBLEM - knifepointhorror.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'knifepointhorror.wiki' expires in 15 day(s) (Mon 29 Jul 2024 01:16:17 AM GMT +0000).
[01:39:01] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/627652722016...9f4c7f097ac9
[01:39:03] <botifico>	 [02ssl] 07WikiTideSSLBot 039f4c7f0 - Bot: Update SSL cert for knifepointhorror.wiki
[01:39:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.00, 23.70, 19.64
[01:41:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.55, 23.38, 20.02
[01:45:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.06, 23.30, 20.70
[01:47:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.54, 22.60, 20.77
[01:50:32] <icinga-miraheze>	 RECOVERY - persist.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'persist.wiki' will expire on Fri 11 Oct 2024 12:21:52 AM GMT +0000.
[01:51:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.19, 23.10, 21.35
[01:55:29] <icinga-miraheze>	 RECOVERY - wiki.aoemods.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.aoemods.com' will expire on Fri 11 Oct 2024 12:26:35 AM GMT +0000.
[01:55:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:57:56] <icinga-miraheze>	 PROBLEM - roll-of-arms.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'roll-of-arms.com' expires in 15 day(s) (Mon 29 Jul 2024 01:32:21 AM GMT +0000).
[01:58:09] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9f4c7f097ac9...3a06ce3c5bc3
[01:58:11] <botifico>	 [02ssl] 07WikiTideSSLBot 033a06ce3 - Bot: Update SSL cert for roll-of-arms.com
[01:59:08] <icinga-miraheze>	 RECOVERY - swests.org - LetsEncrypt on sslhost is OK: OK - Certificate 'swests.org' will expire on Fri 11 Oct 2024 12:30:40 AM GMT +0000.
[02:01:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.96, 23.59, 22.91
[02:03:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.27, 24.14, 23.17
[02:05:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.49, 23.58, 23.09
[02:07:20] <icinga-miraheze>	 PROBLEM - buildabearwiki.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'buildabearwiki.info' expires in 15 day(s) (Mon 29 Jul 2024 01:40:09 AM GMT +0000).
[02:07:31] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3a06ce3c5bc3...5c6dbae46b10
[02:07:34] <botifico>	 [02ssl] 07WikiTideSSLBot 035c6dbae - Bot: Update SSL cert for buildabearwiki.info
[02:08:04] <icinga-miraheze>	 RECOVERY - knifepointhorror.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'knifepointhorror.wiki' will expire on Fri 11 Oct 2024 12:38:56 AM GMT +0000.
[02:11:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.95, 24.06, 23.37
[02:13:31] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 14.78, 21.39, 23.75
[02:14:02] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[02:16:03] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [59 system event log (SEL) entries present]
[02:16:37] <icinga-miraheze>	 PROBLEM - wiki.infomedia.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.infomedia.co.uk' expires in 15 day(s) (Mon 29 Jul 2024 02:13:46 AM GMT +0000).
[02:16:48] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5c6dbae46b10...b9c6a661b9c9
[02:16:50] <botifico>	 [02ssl] 07WikiTideSSLBot 03b9c6a66 - Bot: Update SSL cert for wiki.infomedia.co.uk
[02:19:31] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.47, 23.70, 23.80
[02:19:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.55, 23.08, 23.74
[02:21:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.23, 23.01, 23.58
[02:23:31] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.52, 22.98, 23.53
[02:25:06] <icinga-miraheze>	 PROBLEM - iol.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'iol.wiki' expires in 15 day(s) (Mon 29 Jul 2024 01:56:41 AM GMT +0000).
[02:25:18] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b9c6a661b9c9...1678ef48f8d1
[02:25:19] <botifico>	 [02ssl] 07WikiTideSSLBot 031678ef4 - Bot: Update SSL cert for iol.wiki
[02:25:30] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:25:36] <icinga-miraheze>	 PROBLEM - en.omniversalis.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.omniversalis.org' expires in 15 day(s) (Mon 29 Jul 2024 01:54:15 AM GMT +0000).
[02:25:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.12, 21.77, 22.97
[02:25:47] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1678ef48f8d1...77eb0e7d28bc
[02:25:48] <botifico>	 [02ssl] 07WikiTideSSLBot 0377eb0e7 - Bot: Update SSL cert for en.omniversalis.org
[02:26:52] <icinga-miraheze>	 RECOVERY - roll-of-arms.com - LetsEncrypt on sslhost is OK: OK - Certificate 'roll-of-arms.com' will expire on Fri 11 Oct 2024 12:58:02 AM GMT +0000.
[02:27:40] <icinga-miraheze>	 PROBLEM - wiki.virtualjet.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.virtualjet.net' expires in 15 day(s) (Mon 29 Jul 2024 02:25:28 AM GMT +0000).
[02:27:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.82, 23.18, 23.34
[02:27:51] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/77eb0e7d28bc...e411ff8749a6
[02:27:53] <botifico>	 [02ssl] 07WikiTideSSLBot 03e411ff8 - Bot: Update SSL cert for wiki.virtualjet.net
[02:29:31] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.29, 22.63, 23.14
[02:29:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.99, 20.88, 22.48
[02:31:31] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.61, 21.90, 22.79
[02:32:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 15.37, 20.15, 23.31
[02:33:26] <icinga-miraheze>	 PROBLEM - royal-wiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'royal-wiki.org' expires in 15 day(s) (Mon 29 Jul 2024 02:04:26 AM GMT +0000).
[02:33:37] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e411ff8749a6...ef0773dbab5a
[02:33:40] <botifico>	 [02ssl] 07WikiTideSSLBot 03ef0773d - Bot: Update SSL cert for royal-wiki.org
[02:35:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.03, 20.35, 23.46
[02:35:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.87, 23.65, 23.01
[02:36:40] <icinga-miraheze>	 RECOVERY - buildabearwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'buildabearwiki.info' will expire on Fri 11 Oct 2024 01:07:26 AM GMT +0000.
[02:39:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.60, 22.46, 22.73
[02:40:20] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 13.22, 15.60, 19.91
[02:45:31] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.63, 17.84, 20.23
[02:46:13] <icinga-miraheze>	 RECOVERY - wiki.infomedia.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.infomedia.co.uk' will expire on Fri 11 Oct 2024 01:16:42 AM GMT +0000.
[02:47:09] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.97, 16.78, 20.09
[02:50:50] <icinga-miraheze>	 PROBLEM - wiki.mostly.vet - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mostly.vet' expires in 15 day(s) (Mon 29 Jul 2024 02:42:02 AM GMT +0000).
[02:51:02] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ef0773dbab5a...229a04ebb990
[02:51:04] <botifico>	 [02ssl] 07WikiTideSSLBot 03229a04e - Bot: Update SSL cert for wiki.mostly.vet
[02:52:00] <icinga-miraheze>	 PROBLEM - squareenixwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'squareenixwiki.com' expires in 15 day(s) (Mon 29 Jul 2024 02:20:39 AM GMT +0000).
[02:52:12] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/229a04ebb990...cbafdbbbfe4f
[02:52:15] <botifico>	 [02ssl] 07WikiTideSSLBot 03cbafdbb - Bot: Update SSL cert for squareenixwiki.com
[02:54:12] <icinga-miraheze>	 RECOVERY - iol.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'iol.wiki' will expire on Fri 11 Oct 2024 01:25:12 AM GMT +0000.
[02:55:13] <icinga-miraheze>	 RECOVERY - en.omniversalis.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.omniversalis.org' will expire on Fri 11 Oct 2024 01:25:42 AM GMT +0000.
[02:56:19] <icinga-miraheze>	 RECOVERY - wiki.virtualjet.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.virtualjet.net' will expire on Fri 11 Oct 2024 01:27:45 AM GMT +0000.
[02:57:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.85, 21.77, 20.75
[02:57:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.55, 20.75, 21.08
[02:59:43] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.79, 18.18, 20.10
[03:00:23] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.24, 21.19, 20.58
[03:01:09] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.10, 20.07, 20.35
[03:03:21] <icinga-miraheze>	 RECOVERY - royal-wiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'royal-wiki.org' will expire on Fri 11 Oct 2024 01:33:31 AM GMT +0000.
[03:04:13] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.68, 24.12, 21.80
[03:06:07] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.19, 20.43, 19.08
[03:08:03] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 19.28, 20.11, 19.14
[03:09:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.32, 22.01, 20.77
[03:11:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.27, 19.39, 19.17
[03:11:56] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.56, 20.70, 19.61
[03:13:43] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.08, 18.51, 18.89
[03:15:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.38, 22.53, 20.51
[03:17:42] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.12, 20.71, 20.09
[03:19:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.15, 23.36, 22.51
[03:20:10] <icinga-miraheze>	 RECOVERY - wiki.mostly.vet - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mostly.vet' will expire on Fri 11 Oct 2024 01:50:56 AM GMT +0000.
[03:21:01] <icinga-miraheze>	 RECOVERY - squareenixwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'squareenixwiki.com' will expire on Fri 11 Oct 2024 01:52:06 AM GMT +0000.
[03:23:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.08, 23.57, 22.71
[03:23:29] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.32, 22.00, 20.83
[03:25:25] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.94, 22.70, 21.27
[03:29:16] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.36, 22.70, 21.47
[03:30:07] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.00, 19.84, 18.71
[03:32:02] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.09, 18.94, 18.49
[03:33:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:36:59] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.04, 23.31, 22.40
[03:38:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:43:31] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 17.78, 21.76, 23.99
[03:48:33] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.88, 21.74, 21.50
[03:55:31] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.74, 21.81, 22.52
[03:57:31] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.18, 21.05, 22.15
[04:00:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.86, 22.54, 22.74
[04:03:31] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 13.62, 17.37, 20.24
[04:05:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 15.48, 20.22, 23.86
[04:10:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.69, 22.87, 22.25
[04:11:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.93, 22.70, 23.56
[04:12:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 13.09, 19.36, 21.07
[04:13:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.33, 21.74, 23.12
[04:16:20] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 10.42, 15.79, 19.32
[04:16:36] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[04:17:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.39, 22.98, 23.24
[04:18:37] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [60 system event log (SEL) entries present]
[04:19:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.90, 22.07, 22.91
[04:29:09] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.85, 17.13, 20.06
[05:47:58] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:08:33] <icinga-miraheze>	 PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2]
[06:13:49] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures
[06:25:20] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[06:27:20] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [61 system event log (SEL) entries present]
[06:34:33] <icinga-miraheze>	 RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[07:07:10] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[07:11:28] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:13:27] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 4.711747169e-05 secs
[08:17:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[08:25:47] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[08:27:48] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [62 system event log (SEL) entries present]
[08:37:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.15, 21.45, 15.29
[08:37:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.20, 18.13, 13.28
[08:45:43] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.13, 19.56, 16.25
[08:51:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.40, 22.55, 20.86
[09:17:38] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[09:29:09] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.35, 19.41, 20.40
[09:43:45] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures
[09:56:01] <icinga-miraheze>	 PROBLEM - wiki.albrusmusic.ru - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.albrusmusic.ru' expires in 15 day(s) (Mon 29 Jul 2024 09:33:41 AM GMT +0000).
[09:56:19] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cbafdbbbfe4f...12e132fc6d40
[09:56:20] <botifico>	 [02ssl] 07WikiTideSSLBot 0312e132f - Bot: Update SSL cert for wiki.albrusmusic.ru
[09:59:28] <icinga-miraheze>	 PROBLEM - wiki.msnld.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.msnld.com' expires in 15 day(s) (Mon 29 Jul 2024 09:45:57 AM GMT +0000).
[10:17:05] <icinga-miraheze>	 PROBLEM - wiki.themarkside.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.themarkside.com All nameservers failed to answer the query.
[10:20:14] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[10:22:15] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [64 system event log (SEL) entries present]
[10:24:22] <icinga-miraheze>	 PROBLEM - encyclopedia.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'encyclopedia.eu.org' expires in 15 day(s) (Mon 29 Jul 2024 10:16:49 AM GMT +0000).
[10:24:33] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/12e132fc6d40...10242d092330
[10:24:34] <botifico>	 [02ssl] 07WikiTideSSLBot 0310242d0 - Bot: Update SSL cert for encyclopedia.eu.org
[10:24:34] <icinga-miraheze>	 RECOVERY - wiki.albrusmusic.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.albrusmusic.ru' will expire on Fri 11 Oct 2024 08:56:11 AM GMT +0000.
[10:33:55] <icinga-miraheze>	 PROBLEM - looneypyramids.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'looneypyramids.wiki' expires in 15 day(s) (Mon 29 Jul 2024 10:02:53 AM GMT +0000).
[10:34:06] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/10242d092330...85da42a04157
[10:34:09] <botifico>	 [02ssl] 07WikiTideSSLBot 0385da42a - Bot: Update SSL cert for looneypyramids.wiki
[10:43:37] <MirahezeRelay>	 <reception123> !log bash ./wikibackups.sh /home/reception/publicwikis.txt /srv/mediawiki/w/maintenance/dumpBackup.php (public wikibackups)
[10:43:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:45:01] <icinga-miraheze>	 PROBLEM - wiki.meeusen.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.meeusen.net' expires in 15 day(s) (Mon 29 Jul 2024 10:28:15 AM GMT +0000).
[10:45:13] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/85da42a04157...34627656f99d
[10:45:14] <botifico>	 [02ssl] 07WikiTideSSLBot 033462765 - Bot: Update SSL cert for wiki.meeusen.net
[10:47:01] <icinga-miraheze>	 PROBLEM - wiki.themarkside.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.themarkside.com could not be found
[10:52:17] <MirahezeLSBot>	 !log [salt-user@mwtask181] Renamed witheredhopewiki to reignsoffirewiki using renamewiki.py
[10:52:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:52:56] <MirahezeLSBot>	 !log [salt-user@mwtask181] Renamed javewiki to ibbwiki using renamewiki.py
[10:53:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:53:51] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=tsebulevewiki --no-updates --username-prefix=wikia:tsebuleve /home/reception/tsebulevewiki-20240711123818.xml (START)
[10:53:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:54:07] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=tsebulevewiki --no-updates --username-prefix=wikia:tsebuleve /home/reception/tsebulevewiki-20240711123818.xml (START)
[10:54:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:54:13] <icinga-miraheze>	 RECOVERY - encyclopedia.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopedia.eu.org' will expire on Fri 11 Oct 2024 09:24:27 AM GMT +0000.
[10:55:32] <MirahezeLSBot>	 !log [salt-user@mwtask181] Renamed warcasteracademywiki to warmachineacademywiki using renamewiki.py
[10:55:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:00:06] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=cyberpediawiki /home/reception/undertale_pages_full.xml.7z --username-prefix=wikia:undertale (START)
[11:00:07] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=cyberpediawiki /home/reception/undertale_pages_full.xml.7z --username-prefix=wikia:undertale (END - exit=256)
[11:00:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:00:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:00:18] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=cyberpediawiki /home/reception/undertale_pages_full.xml --username-prefix=wikia:undertale (START)
[11:00:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:00:25] <MirahezeLSBot>	 !log [salt-user@mwtask181] Renamed dotzipwiki to dollavitaworldwiki using renamewiki.py
[11:00:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:01:22] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=bignatecommentswiki /home/reception/bignatecomments_pages_full.xml --username-prefix=wikia:big-nate-comments (START)
[11:01:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:02:15] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=tsebulevewiki --no-updates --username-prefix=wikia:tsebuleve /home/reception/tsebulevewiki-20240711123818.xml (END - exit=0)
[11:02:16] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=tsebulevewiki (START)
[11:02:17] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=tsebulevewiki (END - exit=256)
[11:02:19] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=tsebulevewiki --active --update (END - exit=0)
[11:02:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:02:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:02:25] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=tsebulevewiki --no-updates --username-prefix=wikia:tsebuleve /home/reception/tsebulevewiki-20240711123818.xml (END - exit=0)
[11:02:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:02:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:02:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:02:50] <icinga-miraheze>	 RECOVERY - looneypyramids.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'looneypyramids.wiki' will expire on Fri 11 Oct 2024 09:34:01 AM GMT +0000.
[11:04:57] <MirahezeLSBot>	 !log [salt-user@mwtask181] Renamed wordhoardwiki to nihontouwiki using renamewiki.py
[11:05:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:10:30] <MirahezeLSBot>	 !log [salt-user@mwtask181] Renamed dormitabistrilogywiki to dormitabisremasteredwiki using renamewiki.py
[11:10:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:14:32] <icinga-miraheze>	 RECOVERY - wiki.meeusen.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meeusen.net' will expire on Fri 11 Oct 2024 09:45:07 AM GMT +0000.
[11:14:43] <MirahezeLSBot>	 !log [salt-user@mwtask181] Renamed impostorrebornwiki to regeneratedwiki using renamewiki.py
[11:14:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:16:22] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=cyberpediawiki /home/reception/undertale_pages_full.xml --username-prefix=wikia:undertale (END - exit=2)
[11:16:23] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=bignatecommentswiki /home/reception/bignatecomments_pages_full.xml --username-prefix=wikia:big-nate-comments (END - exit=2)
[11:16:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:16:33] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=cyberpediawiki /home/reception/undertale_pages_full.xml --username-prefix=wikia:undertale (START)
[11:16:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:16:37] <icinga-miraheze>	 PROBLEM - landofliberos.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'landofliberos.com' expires in 15 day(s) (Mon 29 Jul 2024 10:55:28 AM GMT +0000).
[11:16:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:16:44] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=bignatecommentswiki /home/reception/bignatecomments_pages_full.xml --username-prefix=wikia:big-nate-comments (START)
[11:16:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:16:50] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/34627656f99d...4a2b286d42ab
[11:16:51] <botifico>	 [02ssl] 07WikiTideSSLBot 034a2b286 - Bot: Update SSL cert for landofliberos.com
[11:20:09] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 48.48, 19.80, 7.60
[11:21:54] <icinga-miraheze>	 PROBLEM - db181 PowerDNS Recursor on db181 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[11:22:00] <icinga-miraheze>	 PROBLEM - db181 SSH on db181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:23:53] <icinga-miraheze>	 RECOVERY - db181 PowerDNS Recursor on db181 is OK: DNS OK: 0.195 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[11:23:54] <icinga-miraheze>	 RECOVERY - db181 SSH on db181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[11:25:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.96, 18.19, 14.91
[11:27:09] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 12.98, 16.18, 14.58
[11:30:09] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 0.43, 9.10, 10.45
[11:32:09] <icinga-miraheze>	 RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.22, 6.15, 9.21
[11:36:41] <icinga-miraheze>	 PROBLEM - wiki.harlonserver.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.harlonserver.net' expires in 15 day(s) (Mon 29 Jul 2024 11:09:53 AM GMT +0000).
[11:36:52] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4a2b286d42ab...973e106d7f32
[11:36:54] <botifico>	 [02ssl] 07WikiTideSSLBot 03973e106 - Bot: Update SSL cert for wiki.harlonserver.net
[11:37:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:46:14] <icinga-miraheze>	 RECOVERY - landofliberos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'landofliberos.com' will expire on Fri 11 Oct 2024 10:16:43 AM GMT +0000.
[11:52:43] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[11:54:42] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0004959106445 secs
[11:56:41] <icinga-miraheze>	 PROBLEM - wiki.site97.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.site97.fr' expires in 15 day(s) (Mon 29 Jul 2024 11:44:40 AM GMT +0000).
[11:56:52] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/973e106d7f32...2f087228c99b
[11:56:54] <botifico>	 [02ssl] 07WikiTideSSLBot 032f08722 - Bot: Update SSL cert for wiki.site97.fr
[12:02:23] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[12:04:23] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0001165568829 secs
[12:05:23] <icinga-miraheze>	 RECOVERY - wiki.harlonserver.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.harlonserver.net' will expire on Fri 11 Oct 2024 10:36:47 AM GMT +0000.
[12:20:08] <icinga-miraheze>	 PROBLEM - wiki.mastodon.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mastodon.kr' expires in 15 day(s) (Mon 29 Jul 2024 12:07:27 PM GMT +0000).
[12:20:19] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2f087228c99b...311af3bfc54c
[12:20:22] <botifico>	 [02ssl] 07WikiTideSSLBot 03311af3b - Bot: Update SSL cert for wiki.mastodon.kr
[12:25:52] <icinga-miraheze>	 RECOVERY - wiki.site97.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.site97.fr' will expire on Fri 11 Oct 2024 10:56:47 AM GMT +0000.
[12:28:05] <icinga-miraheze>	 PROBLEM - abeyancewiki.bumblebee-studios.se - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'abeyancewiki.bumblebee-studios.se' expires in 15 day(s) (Mon 29 Jul 2024 12:02:31 PM GMT +0000).
[12:28:16] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/311af3bfc54c...5b1692c48608
[12:28:19] <botifico>	 [02ssl] 07WikiTideSSLBot 035b1692c - Bot: Update SSL cert for abeyancewiki.bumblebee-studios.se
[12:32:43] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[12:34:43] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [65 system event log (SEL) entries present]
[12:35:02] <icinga-miraheze>	 PROBLEM - www.cgradegames.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.cgradegames.net' expires in 15 day(s) (Mon 29 Jul 2024 12:11:13 PM GMT +0000).
[12:35:13] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5b1692c48608...b3f373070e1d
[12:35:14] <botifico>	 [02ssl] 07WikiTideSSLBot 03b3f3730 - Bot: Update SSL cert for www.cgradegames.net
[12:35:15] <icinga-miraheze>	 PROBLEM - wiki.10key.plus - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.10key.plus' expires in 15 day(s) (Mon 29 Jul 2024 12:12:39 PM GMT +0000).
[12:35:26] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b3f373070e1d...58fafb2ade36
[12:35:28] <botifico>	 [02ssl] 07WikiTideSSLBot 0358fafb2 - Bot: Update SSL cert for wiki.10key.plus
[12:44:58] <icinga-miraheze>	 PROBLEM - eng.archiopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'eng.archiopedia.org' expires in 15 day(s) (Mon 29 Jul 2024 12:41:20 PM GMT +0000).
[12:45:10] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/58fafb2ade36...8332f3a8d23b
[12:45:12] <botifico>	 [02ssl] 07WikiTideSSLBot 038332f3a - Bot: Update SSL cert for eng.archiopedia.org
[12:47:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:57:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.15, 18.77, 15.83
[12:57:10] <icinga-miraheze>	 RECOVERY - abeyancewiki.bumblebee-studios.se - LetsEncrypt on sslhost is OK: OK - Certificate 'abeyancewiki.bumblebee-studios.se' will expire on Fri 11 Oct 2024 11:28:11 AM GMT +0000.
[12:59:09] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.36, 18.95, 16.26
[13:04:00] <icinga-miraheze>	 RECOVERY - wiki.10key.plus - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.10key.plus' will expire on Fri 11 Oct 2024 11:35:19 AM GMT +0000.
[13:04:51] <icinga-miraheze>	 RECOVERY - www.cgradegames.net - LetsEncrypt on sslhost is OK: OK - Certificate 'www.cgradegames.net' will expire on Fri 11 Oct 2024 11:35:08 AM GMT +0000.
[13:06:08] <icinga-miraheze>	 PROBLEM - webkinzguide.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'webkinzguide.com' expires in 15 day(s) (Mon 29 Jul 2024 12:53:16 PM GMT +0000).
[13:06:19] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8332f3a8d23b...ab591c5285d4
[13:06:22] <botifico>	 [02ssl] 07WikiTideSSLBot 03ab591c5 - Bot: Update SSL cert for webkinzguide.com
[13:10:18] <icinga-miraheze>	 PROBLEM - itora.bunnies1230.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'itora.bunnies1230.com' expires in 15 day(s) (Mon 29 Jul 2024 12:38:37 PM GMT +0000).
[13:10:30] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ab591c5285d4...0ea33133b27c
[13:10:32] <botifico>	 [02ssl] 07WikiTideSSLBot 030ea3313 - Bot: Update SSL cert for itora.bunnies1230.com
[13:14:21] <icinga-miraheze>	 PROBLEM - wiki.otir.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.otir.nl' expires in 15 day(s) (Mon 29 Jul 2024 12:53:43 PM GMT +0000).
[13:14:33] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0ea33133b27c...13ba04b57fed
[13:14:34] <botifico>	 [02ssl] 07WikiTideSSLBot 0313ba04b - Bot: Update SSL cert for wiki.otir.nl
[13:18:08] <icinga-miraheze>	 RECOVERY - wiki.mastodon.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mastodon.kr' will expire on Fri 11 Oct 2024 11:20:14 AM GMT +0000.
[13:35:46] <icinga-miraheze>	 RECOVERY - webkinzguide.com - LetsEncrypt on sslhost is OK: OK - Certificate 'webkinzguide.com' will expire on Fri 11 Oct 2024 12:06:14 PM GMT +0000.
[13:43:54] <icinga-miraheze>	 RECOVERY - eng.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'eng.archiopedia.org' will expire on Fri 11 Oct 2024 11:45:04 AM GMT +0000.
[13:53:33] <icinga-miraheze>	 PROBLEM - 321nails.crpteam.club - LetsEncrypt on sslhost is WARNING: WARNING - Certificate '321nails.crpteam.club' expires in 15 day(s) (Mon 29 Jul 2024 01:34:50 PM GMT +0000).
[13:53:45] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/13ba04b57fed...dc4b5bfa68da
[13:53:47] <botifico>	 [02ssl] 07WikiTideSSLBot 03dc4b5bf - Bot: Update SSL cert for 321nails.crpteam.club
[13:55:37] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.78, 22.07, 19.35
[13:56:25] <icinga-miraheze>	 PROBLEM - wiki.milkywayidle.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.milkywayidle.com' expires in 15 day(s) (Mon 29 Jul 2024 01:26:38 PM GMT +0000).
[13:56:37] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/dc4b5bfa68da...9788a65e75cd
[13:56:38] <botifico>	 [02ssl] 07WikiTideSSLBot 039788a65 - Bot: Update SSL cert for wiki.milkywayidle.com
[13:57:34] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 24.00, 22.61, 19.87
[13:58:41] <icinga-miraheze>	 PROBLEM - solaswiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'solaswiki.org' expires in 15 day(s) (Mon 29 Jul 2024 01:40:40 PM GMT +0000).
[13:58:58] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9788a65e75cd...c48463db4f49
[13:59:00] <botifico>	 [02ssl] 07WikiTideSSLBot 03c48463d - Bot: Update SSL cert for solaswiki.org
[14:03:24] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.11, 23.12, 20.87
[14:04:20] <icinga-miraheze>	 PROBLEM - dkpedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dkpedia.com' expires in 15 day(s) (Mon 29 Jul 2024 01:51:09 PM GMT +0000).
[14:04:31] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c48463db4f49...faad5c7d350a
[14:04:34] <botifico>	 [02ssl] 07WikiTideSSLBot 03faad5c7 - Bot: Update SSL cert for dkpedia.com
[14:05:21] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.04, 21.81, 20.67
[14:07:18] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.87, 22.42, 20.98
[14:08:35] <icinga-miraheze>	 PROBLEM - largedu.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'largedu.eu.org' expires in 15 day(s) (Mon 29 Jul 2024 01:38:10 PM GMT +0000).
[14:08:45] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/faad5c7d350a...dd6fd24add2d
[14:08:47] <botifico>	 [02ssl] 07WikiTideSSLBot 03dd6fd24 - Bot: Update SSL cert for largedu.eu.org
[14:08:55] <icinga-miraheze>	 RECOVERY - itora.bunnies1230.com - LetsEncrypt on sslhost is OK: OK - Certificate 'itora.bunnies1230.com' will expire on Fri 11 Oct 2024 12:10:24 PM GMT +0000.
[14:09:15] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.35, 21.21, 20.70
[14:11:12] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.70, 22.13, 21.06
[14:12:21] <icinga-miraheze>	 RECOVERY - wiki.otir.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.otir.nl' will expire on Fri 11 Oct 2024 12:14:27 PM GMT +0000.
[14:13:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.07, 21.99, 21.17
[14:15:09] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.58, 19.38, 20.29
[14:20:37] <icinga-miraheze>	 PROBLEM - grk.archiopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'grk.archiopedia.org' expires in 15 day(s) (Mon 29 Jul 2024 02:08:39 PM GMT +0000).
[14:20:47] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/dd6fd24add2d...4363c91862c3
[14:20:50] <botifico>	 [02ssl] 07WikiTideSSLBot 034363c91 - Bot: Update SSL cert for grk.archiopedia.org
[14:22:37] <icinga-miraheze>	 RECOVERY - 321nails.crpteam.club - LetsEncrypt on sslhost is OK: OK - Certificate '321nails.crpteam.club' will expire on Fri 11 Oct 2024 12:53:39 PM GMT +0000.
[14:23:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.22, 19.65, 19.80
[14:25:21] <icinga-miraheze>	 RECOVERY - wiki.milkywayidle.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.milkywayidle.com' will expire on Fri 11 Oct 2024 12:56:31 PM GMT +0000.
[14:28:23] <icinga-miraheze>	 RECOVERY - solaswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'solaswiki.org' will expire on Fri 11 Oct 2024 12:58:53 PM GMT +0000.
[14:29:09] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.74, 19.80, 20.02
[14:33:39] <icinga-miraheze>	 RECOVERY - dkpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dkpedia.com' will expire on Fri 11 Oct 2024 01:04:25 PM GMT +0000.
[14:37:39] <icinga-miraheze>	 RECOVERY - largedu.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'largedu.eu.org' will expire on Fri 11 Oct 2024 01:08:40 PM GMT +0000.
[14:41:12] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[14:43:13] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [66 system event log (SEL) entries present]
[14:49:43] <icinga-miraheze>	 RECOVERY - grk.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'grk.archiopedia.org' will expire on Fri 11 Oct 2024 01:20:42 PM GMT +0000.
[15:08:41] <icinga-miraheze>	 PROBLEM - grk.repository.archiopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'grk.repository.archiopedia.org' expires in 15 day(s) (Mon 29 Jul 2024 02:55:31 PM GMT +0000).
[15:08:53] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4363c91862c3...f747f8dbd8fa
[15:08:55] <botifico>	 [02ssl] 07WikiTideSSLBot 03f747f8d - Bot: Update SSL cert for grk.repository.archiopedia.org
[15:37:51] <icinga-miraheze>	 RECOVERY - grk.repository.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'grk.repository.archiopedia.org' will expire on Fri 11 Oct 2024 02:08:47 PM GMT +0000.
[15:53:36] <icinga-miraheze>	 PROBLEM - isocasg.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'isocasg.org' expires in 7 day(s) (Sun 21 Jul 2024 03:43:22 PM GMT +0000).
[16:10:10] <icinga-miraheze>	 PROBLEM - mem161 memcached on mem161 is CRITICAL: connect to address 10.0.16.131 and port 11211: Connection refused
[16:12:23] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.62, 22.32, 17.46
[16:14:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.03, 22.53, 18.13
[16:15:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.23, 18.55, 15.46
[16:16:15] <icinga-miraheze>	 PROBLEM - mem151 memcached on mem151 is CRITICAL: connect to address 10.0.15.113 and port 11211: Connection refused
[16:17:09] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 20.01, 19.17, 16.08
[16:21:05] <botifico>	 [02ImportDump] 07coderabbitai[bot] edited a comment on pull request 03#99: Update for MediaWiki 1.42 and use virtual domain - 13https://github.com/miraheze/ImportDump/pull/99#issuecomment-2116880811
[16:21:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.92, 21.16, 17.53
[16:23:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.34, 21.59, 18.14
[16:25:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.64, 23.45, 19.24
[16:28:10] <icinga-miraheze>	 RECOVERY - mem161 memcached on mem161 is OK: TCP OK - 0.007 second response time on 10.0.16.131 port 11211
[16:31:15] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.83, 22.25, 20.37
[16:33:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.75, 24.10, 21.27
[16:34:15] <icinga-miraheze>	 RECOVERY - mem151 memcached on mem151 is OK: TCP OK - 0.000 second response time on 10.0.15.113 port 11211
[16:37:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.13, 22.90, 21.55
[16:37:25] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:39:31] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.07, 22.48, 23.30
[16:40:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.59, 22.67, 23.69
[16:42:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.38, 23.78, 24.00
[16:42:25] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:43:09] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.42, 17.93, 19.75
[16:46:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.11, 22.41, 23.44
[16:48:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.31, 22.81, 23.41
[16:50:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.59, 21.26, 22.74
[16:51:31] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 11.94, 16.49, 19.83
[16:56:20] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.52, 15.92, 19.69
[17:00:55] <icinga-miraheze>	 PROBLEM - segawiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'segawiki.com' expires in 15 day(s) (Mon 29 Jul 2024 04:31:10 PM GMT +0000).
[17:01:07] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f747f8dbd8fa...c012f1ed5dbe
[17:01:09] <botifico>	 [02ssl] 07WikiTideSSLBot 03c012f1e - Bot: Update SSL cert for segawiki.com
[17:26:50] <icinga-miraheze>	 PROBLEM - petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.petrawiki.org' expires in 15 day(s) (Mon 29 Jul 2024 05:03:08 PM GMT +0000).
[17:28:07] <icinga-miraheze>	 PROBLEM - en.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.petrawiki.org' expires in 15 day(s) (Mon 29 Jul 2024 05:03:08 PM GMT +0000).
[17:28:20] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c012f1ed5dbe...9a2d856f01d4
[17:28:23] <botifico>	 [02ssl] 07WikiTideSSLBot 039a2d856 - Bot: Update SSL cert for en.petrawiki.org
[17:30:47] <icinga-miraheze>	 RECOVERY - segawiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'segawiki.com' will expire on Fri 11 Oct 2024 04:01:01 PM GMT +0000.
[17:33:35] <icinga-miraheze>	 PROBLEM - resources.africanvision.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'resources.africanvision.org' expires in 15 day(s) (Mon 29 Jul 2024 05:14:12 PM GMT +0000).
[17:33:46] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9a2d856f01d4...733b0033aa15
[17:33:47] <botifico>	 [02ssl] 07WikiTideSSLBot 03733b003 - Bot: Update SSL cert for resources.africanvision.org
[17:34:41] <icinga-miraheze>	 PROBLEM - wiki.zahira.me - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.zahira.me' expires in 15 day(s) (Mon 29 Jul 2024 05:20:11 PM GMT +0000).
[17:34:53] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/733b0033aa15...ecd3a0134333
[17:34:55] <botifico>	 [02ssl] 07WikiTideSSLBot 03ecd3a01 - Bot: Update SSL cert for wiki.zahira.me
[17:38:54] <icinga-miraheze>	 PROBLEM - wiki.artmechanicum.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.artmechanicum.com' expires in 15 day(s) (Mon 29 Jul 2024 05:08:57 PM GMT +0000).
[17:39:05] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ecd3a0134333...f51118f8b4f6
[17:39:07] <botifico>	 [02ssl] 07WikiTideSSLBot 03f51118f - Bot: Update SSL cert for wiki.artmechanicum.com
[17:46:15] <icinga-miraheze>	 PROBLEM - wiki.bluevertigo.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.bluevertigo.org' expires in 15 day(s) (Mon 29 Jul 2024 05:23:32 PM GMT +0000).
[17:46:26] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f51118f8b4f6...2136e190f214
[17:46:27] <botifico>	 [02ssl] 07WikiTideSSLBot 032136e19 - Bot: Update SSL cert for wiki.bluevertigo.org
[17:46:38] <icinga-miraheze>	 PROBLEM - metroidpedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'metroidpedia.com' expires in 15 day(s) (Mon 29 Jul 2024 05:34:51 PM GMT +0000).
[17:46:50] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2136e190f214...6deef1cc9f63
[17:46:52] <botifico>	 [02ssl] 07WikiTideSSLBot 036deef1c - Bot: Update SSL cert for metroidpedia.com
[17:53:04] <icinga-miraheze>	 PROBLEM - tech.exputra.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tech.exputra.com' expires in 15 day(s) (Mon 29 Jul 2024 05:40:12 PM GMT +0000).
[17:53:15] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6deef1cc9f63...2a0a0734c821
[17:53:18] <botifico>	 [02ssl] 07WikiTideSSLBot 032a0a073 - Bot: Update SSL cert for tech.exputra.com
[17:55:06] <icinga-miraheze>	 PROBLEM - schizoidnightmares.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'schizoidnightmares.wiki' expires in 15 day(s) (Mon 29 Jul 2024 05:38:48 PM GMT +0000).
[17:55:20] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2a0a0734c821...8ad6ff73b6ac
[17:55:22] <botifico>	 [02ssl] 07WikiTideSSLBot 038ad6ff7 - Bot: Update SSL cert for schizoidnightmares.wiki
[17:56:10] <icinga-miraheze>	 RECOVERY - petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.petrawiki.org' will expire on Fri 11 Oct 2024 04:28:13 PM GMT +0000.
[17:57:14] <icinga-miraheze>	 RECOVERY - en.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.petrawiki.org' will expire on Fri 11 Oct 2024 04:28:13 PM GMT +0000.
[18:02:09] <icinga-miraheze>	 RECOVERY - resources.africanvision.org - LetsEncrypt on sslhost is OK: OK - Certificate 'resources.africanvision.org' will expire on Fri 11 Oct 2024 04:33:40 PM GMT +0000.
[18:04:22] <icinga-miraheze>	 RECOVERY - wiki.zahira.me - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zahira.me' will expire on Fri 11 Oct 2024 04:34:47 PM GMT +0000.
[18:04:40] <icinga-miraheze>	 PROBLEM - crashspyrowiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'crashspyrowiki.com' expires in 15 day(s) (Mon 29 Jul 2024 05:56:27 PM GMT +0000).
[18:04:51] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8ad6ff73b6ac...d216e70da8a4
[18:04:53] <botifico>	 [02ssl] 07WikiTideSSLBot 03d216e70 - Bot: Update SSL cert for crashspyrowiki.com
[18:08:18] <icinga-miraheze>	 RECOVERY - wiki.artmechanicum.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.artmechanicum.com' will expire on Fri 11 Oct 2024 04:38:59 PM GMT +0000.
[18:10:08] <icinga-miraheze>	 PROBLEM - nintendowiki.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nintendowiki.wiki' expires in 15 day(s) (Mon 29 Jul 2024 05:38:38 PM GMT +0000).
[18:10:18] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d216e70da8a4...a01c1c753b43
[18:10:19] <botifico>	 [02ssl] 07WikiTideSSLBot 03a01c1c7 - Bot: Update SSL cert for nintendowiki.wiki
[18:15:29] <icinga-miraheze>	 RECOVERY - wiki.bluevertigo.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.bluevertigo.org' will expire on Fri 11 Oct 2024 04:46:20 PM GMT +0000.
[18:16:17] <icinga-miraheze>	 RECOVERY - metroidpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'metroidpedia.com' will expire on Fri 11 Oct 2024 04:46:44 PM GMT +0000.
[18:21:38] <icinga-miraheze>	 RECOVERY - tech.exputra.com - LetsEncrypt on sslhost is OK: OK - Certificate 'tech.exputra.com' will expire on Fri 11 Oct 2024 04:53:09 PM GMT +0000.
[18:22:12] <icinga-miraheze>	 PROBLEM - www.sekaipedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.sekaipedia.org' expires in 15 day(s) (Mon 29 Jul 2024 06:01:15 PM GMT +0000).
[18:22:24] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a01c1c753b43...fa55b4f33c49
[18:22:26] <botifico>	 [02ssl] 07WikiTideSSLBot 03fa55b4f - Bot: Update SSL cert for www.sekaipedia.org
[18:24:12] <icinga-miraheze>	 RECOVERY - schizoidnightmares.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'schizoidnightmares.wiki' will expire on Fri 11 Oct 2024 04:55:15 PM GMT +0000.
[18:27:56] <icinga-miraheze>	 PROBLEM - rarewarewiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'rarewarewiki.com' expires in 15 day(s) (Mon 29 Jul 2024 06:15:52 PM GMT +0000).
[18:28:08] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fa55b4f33c49...aa088e7791a8
[18:28:09] <botifico>	 [02ssl] 07WikiTideSSLBot 03aa088e7 - Bot: Update SSL cert for rarewarewiki.com
[18:34:20] <icinga-miraheze>	 RECOVERY - crashspyrowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'crashspyrowiki.com' will expire on Fri 11 Oct 2024 05:04:46 PM GMT +0000.
[18:36:11] <icinga-miraheze>	 PROBLEM - apeirology.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'apeirology.com' expires in 15 day(s) (Mon 29 Jul 2024 06:10:08 PM GMT +0000).
[18:36:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.37, 21.21, 18.82
[18:36:23] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/aa088e7791a8...aa6d98cdb1b5
[18:36:25] <botifico>	 [02ssl] 07WikiTideSSLBot 03aa6d98c - Bot: Update SSL cert for apeirology.com
[18:38:20] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.20, 18.63, 18.16
[18:38:44] <icinga-miraheze>	 PROBLEM - wiki.dwarvesrp.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.dwarvesrp.com' expires in 15 day(s) (Mon 29 Jul 2024 06:26:57 PM GMT +0000).
[18:38:55] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/aa6d98cdb1b5...e53bc0a4898b
[18:38:57] <botifico>	 [02ssl] 07WikiTideSSLBot 03e53bc0a - Bot: Update SSL cert for wiki.dwarvesrp.com
[18:44:21] <icinga-miraheze>	 PROBLEM - sona.pona.la - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sona.pona.la' expires in 15 day(s) (Mon 29 Jul 2024 06:14:03 PM GMT +0000).
[18:44:31] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e53bc0a4898b...cd2ea53161fe
[18:44:32] <botifico>	 [02ssl] 07WikiTideSSLBot 03cd2ea53 - Bot: Update SSL cert for sona.pona.la
[18:51:24] <icinga-miraheze>	 RECOVERY - www.sekaipedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.sekaipedia.org' will expire on Fri 11 Oct 2024 05:22:18 PM GMT +0000.
[18:56:52] <icinga-miraheze>	 RECOVERY - rarewarewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'rarewarewiki.com' will expire on Fri 11 Oct 2024 05:28:02 PM GMT +0000.
[19:02:44] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[19:04:33] <icinga-miraheze>	 PROBLEM - reviwiki.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'reviwiki.info' expires in 15 day(s) (Mon 29 Jul 2024 06:45:20 PM GMT +0000).
[19:04:44] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 70 system event log (SEL) entries present]
[19:04:47] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cd2ea53161fe...6fd807aba2d6
[19:04:48] <botifico>	 [02ssl] 07WikiTideSSLBot 036fd807a - Bot: Update SSL cert for reviwiki.info
[19:05:53] <icinga-miraheze>	 RECOVERY - apeirology.com - LetsEncrypt on sslhost is OK: OK - Certificate 'apeirology.com' will expire on Fri 11 Oct 2024 05:36:17 PM GMT +0000.
[19:07:57] <icinga-miraheze>	 RECOVERY - wiki.dwarvesrp.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dwarvesrp.com' will expire on Fri 11 Oct 2024 05:38:50 PM GMT +0000.
[19:08:23] <icinga-miraheze>	 RECOVERY - nintendowiki.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nintendowiki.wiki' will expire on Fri 11 Oct 2024 05:10:12 PM GMT +0000.
[19:08:51] <icinga-miraheze>	 PROBLEM - yokaiwatchwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'yokaiwatchwiki.com' expires in 15 day(s) (Mon 29 Jul 2024 06:37:02 PM GMT +0000).
[19:09:07] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6fd807aba2d6...67f3de57fab4
[19:09:08] <botifico>	 [02ssl] 07WikiTideSSLBot 0367f3de5 - Bot: Update SSL cert for yokaiwatchwiki.com
[19:21:49] <icinga-miraheze>	 PROBLEM - freedomplanetwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'freedomplanetwiki.com' expires in 15 day(s) (Mon 29 Jul 2024 07:07:48 PM GMT +0000).
[19:22:01] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/67f3de57fab4...b8b5a207dd85
[19:22:02] <botifico>	 [02ssl] 07WikiTideSSLBot 03b8b5a20 - Bot: Update SSL cert for freedomplanetwiki.com
[19:25:06] <icinga-miraheze>	 PROBLEM - files.pornwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'files.pornwiki.org' expires in 15 day(s) (Mon 29 Jul 2024 07:00:15 PM GMT +0000).
[19:25:17] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b8b5a207dd85...0ab295edeafd
[19:25:20] <botifico>	 [02ssl] 07WikiTideSSLBot 030ab295e - Bot: Update SSL cert for files.pornwiki.org
[19:26:12] <icinga-miraheze>	 PROBLEM - minescape.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'minescape.wiki' expires in 15 day(s) (Mon 29 Jul 2024 07:23:15 PM GMT +0000).
[19:26:24] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0ab295edeafd...b7285374ba41
[19:26:27] <botifico>	 [02ssl] 07WikiTideSSLBot 03b728537 - Bot: Update SSL cert for minescape.wiki
[19:34:05] <icinga-miraheze>	 RECOVERY - reviwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Fri 11 Oct 2024 06:04:41 PM GMT +0000.
[19:42:21] <icinga-miraheze>	 RECOVERY - sona.pona.la - LetsEncrypt on sslhost is OK: OK - Certificate 'sona.pona.la' will expire on Fri 11 Oct 2024 05:44:26 PM GMT +0000.
[19:48:21] <icinga-miraheze>	 PROBLEM - ncuwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ncuwiki.com' expires in 15 day(s) (Mon 29 Jul 2024 07:39:54 PM GMT +0000).
[19:48:34] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b7285374ba41...a30ee13eada8
[19:48:35] <botifico>	 [02ssl] 07WikiTideSSLBot 03a30ee13 - Bot: Update SSL cert for ncuwiki.com
[19:50:38] <icinga-miraheze>	 RECOVERY - freedomplanetwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'freedomplanetwiki.com' will expire on Fri 11 Oct 2024 06:21:55 PM GMT +0000.
[19:54:12] <icinga-miraheze>	 RECOVERY - files.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'files.pornwiki.org' will expire on Fri 11 Oct 2024 06:25:11 PM GMT +0000.
[19:54:54] <icinga-miraheze>	 RECOVERY - minescape.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'minescape.wiki' will expire on Fri 11 Oct 2024 06:26:18 PM GMT +0000.
[19:56:20] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.42, 20.74, 18.28
[19:58:20] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.35, 20.15, 18.40
[20:07:34] <icinga-miraheze>	 RECOVERY - yokaiwatchwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'yokaiwatchwiki.com' will expire on Fri 11 Oct 2024 06:09:00 PM GMT +0000.
[20:08:47] <icinga-miraheze>	 PROBLEM - smafallfest.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'smafallfest.org' expires in 15 day(s) (Mon 29 Jul 2024 07:46:50 PM GMT +0000).
[20:08:58] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a30ee13eada8...319cb037814f
[20:08:59] <botifico>	 [02ssl] 07WikiTideSSLBot 03319cb03 - Bot: Update SSL cert for smafallfest.org
[20:14:19] <icinga-miraheze>	 PROBLEM - wiki.limaru.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.limaru.net' expires in 15 day(s) (Mon 29 Jul 2024 07:44:49 PM GMT +0000).
[20:14:30] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/319cb037814f...1200daf74670
[20:14:31] <botifico>	 [02ssl] 07WikiTideSSLBot 031200daf - Bot: Update SSL cert for wiki.limaru.net
[20:15:48] <icinga-miraheze>	 PROBLEM - creativecommons.id - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'creativecommons.id' expires in 15 day(s) (Mon 29 Jul 2024 07:48:07 PM GMT +0000).
[20:16:01] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1200daf74670...9c990f3b1fa2
[20:16:02] <botifico>	 [02ssl] 07WikiTideSSLBot 039c990f3 - Bot: Update SSL cert for creativecommons.id
[20:17:39] <icinga-miraheze>	 PROBLEM - witchingworld.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'witchingworld.org' expires in 15 day(s) (Mon 29 Jul 2024 07:58:05 PM GMT +0000).
[20:17:50] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9c990f3b1fa2...b788fe4737ba
[20:17:51] <botifico>	 [02ssl] 07WikiTideSSLBot 03b788fe4 - Bot: Update SSL cert for witchingworld.org
[20:25:13] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.17, 21.69, 19.16
[20:27:08] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.83, 21.17, 19.22
[20:31:00] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 12.16, 18.22, 18.62
[20:31:18] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[20:33:19] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 72 system event log (SEL) entries present]
[20:38:04] <icinga-miraheze>	 RECOVERY - smafallfest.org - LetsEncrypt on sslhost is OK: OK - Certificate 'smafallfest.org' will expire on Fri 11 Oct 2024 07:08:53 PM GMT +0000.
[20:40:18] <icinga-miraheze>	 PROBLEM - elsterclopedie.marithaime.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'elsterclopedie.marithaime.nl' expires in 15 day(s) (Mon 29 Jul 2024 08:17:10 PM GMT +0000).
[20:40:33] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b788fe4737ba...3877aae80f8c
[20:40:35] <botifico>	 [02ssl] 07WikiTideSSLBot 033877aae - Bot: Update SSL cert for elsterclopedie.marithaime.nl
[20:42:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[20:43:27] <icinga-miraheze>	 PROBLEM - translate.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'translate.petrawiki.org' expires in 15 day(s) (Mon 29 Jul 2024 08:25:46 PM GMT +0000).
[20:43:38] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3877aae80f8c...3c02e1d2fd90
[20:43:39] <botifico>	 [02ssl] 07WikiTideSSLBot 033c02e1d - Bot: Update SSL cert for translate.petrawiki.org
[20:48:13] <icinga-miraheze>	 RECOVERY - ncuwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'ncuwiki.com' will expire on Fri 11 Oct 2024 06:48:29 PM GMT +0000.
[20:56:50] <icinga-miraheze>	 PROBLEM - psychevos.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'psychevos.org' expires in 15 day(s) (Mon 29 Jul 2024 08:39:10 PM GMT +0000).
[20:57:01] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3c02e1d2fd90...b6e6787491a3
[20:57:04] <botifico>	 [02ssl] 07WikiTideSSLBot 03b6e6787 - Bot: Update SSL cert for psychevos.org
[20:57:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:09:37] <icinga-miraheze>	 RECOVERY - elsterclopedie.marithaime.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'elsterclopedie.marithaime.nl' will expire on Fri 11 Oct 2024 07:40:27 PM GMT +0000.
[21:12:19] <icinga-miraheze>	 RECOVERY - wiki.limaru.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.limaru.net' will expire on Fri 11 Oct 2024 07:14:25 PM GMT +0000.
[21:12:53] <icinga-miraheze>	 RECOVERY - translate.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'translate.petrawiki.org' will expire on Fri 11 Oct 2024 07:43:32 PM GMT +0000.
[21:13:49] <icinga-miraheze>	 RECOVERY - creativecommons.id - LetsEncrypt on sslhost is OK: OK - Certificate 'creativecommons.id' will expire on Fri 11 Oct 2024 07:15:53 PM GMT +0000.
[21:15:56] <icinga-miraheze>	 RECOVERY - witchingworld.org - LetsEncrypt on sslhost is OK: OK - Certificate 'witchingworld.org' will expire on Fri 11 Oct 2024 07:17:44 PM GMT +0000.
[21:16:15] <icinga-miraheze>	 PROBLEM - acgn.sfdev.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'acgn.sfdev.eu.org' expires in 15 day(s) (Mon 29 Jul 2024 08:53:25 PM GMT +0000).
[21:16:25] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b6e6787491a3...d0b0d28c82e1
[21:16:26] <botifico>	 [02ssl] 07WikiTideSSLBot 03d0b0d28 - Bot: Update SSL cert for acgn.sfdev.eu.org
[21:26:10] <icinga-miraheze>	 RECOVERY - psychevos.org - LetsEncrypt on sslhost is OK: OK - Certificate 'psychevos.org' will expire on Fri 11 Oct 2024 07:56:55 PM GMT +0000.
[22:06:57] <botifico>	 [02mw-config] 07OAuthority closed pull request 03#5599: T12286: Add $wgRSSAllowLinkTag and $wgRSSItemMaxLength - 13https://github.com/miraheze/mw-config/pull/5599
[22:06:58] <botifico>	 [02mw-config] 07OAuthority pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/6a3affc67672...71bdd14a1a28
[22:07:01] <botifico>	 [02mw-config] 07BlankEclair 0371bdd14 - T12286: Add $wgRSSAllowLinkTag and $wgRSSItemMaxLength (#5599)
[22:07:21] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'config': True} to all
[22:07:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:07:32] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 10s
[22:07:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:08:05] <miraheze-github>	 miraheze/mw-config - OAuthority the build passed.
[22:14:44] <icinga-miraheze>	 RECOVERY - acgn.sfdev.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'acgn.sfdev.eu.org' will expire on Fri 11 Oct 2024 08:16:20 PM GMT +0000.
[22:23:55] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[22:23:56] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[22:24:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:24:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:32:50] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'config': True} to all
[22:32:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:33:01] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 11s
[22:33:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:35:17] <icinga-miraheze>	 PROBLEM - shiroko.us - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket
[22:57:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[22:59:29] <icinga-miraheze>	 PROBLEM - shiroko.us - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for shiroko.us could not be found
[23:02:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:44:32] <botifico>	 [02mw-config] 07BlankEclair opened pull request 03#5606: $wgRSSAllowLinkTag: Escape HTML - 13https://github.com/miraheze/mw-config/pull/5606
[23:45:28] <miraheze-github>	 miraheze/mw-config - BlankEclair the build passed.