[00:13:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.74, 22.20, 21.71
[00:15:48] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.97, 21.22, 21.39
[00:17:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.85, 23.84, 22.34
[00:21:01] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query.
[00:29:36] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.69, 24.40, 20.49
[00:33:26] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.72, 22.56, 20.71
[00:35:22] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.91, 20.37, 20.13
[00:39:35] <botifico>	 [02mw-config] 07Universal-Omega closed pull request 03#5621: T12370: Allow for NATURAL_SORT_KEY on rainversewiki - 13https://github.com/miraheze/mw-config/pull/5621
[00:39:36] <botifico>	 [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/679f40ef9813...1f9ff7d833b5
[00:39:39] <botifico>	 [02mw-config] 07BlankEclair 031f9ff7d - T12370: Allow for NATURAL_SORT_KEY on rainversewiki (#5621)
[00:40:34] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[00:41:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.56, 22.24, 23.96
[00:43:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.42, 24.32, 24.52
[00:50:54] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[00:53:32] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[00:53:33] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[00:53:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:53:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:55:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 17.32, 22.38, 23.89
[01:02:33] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'config': True} to all
[01:02:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:02:45] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 12s
[01:02:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:07:03] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'config': True} to all
[01:07:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:07:14] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 10s
[01:07:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:09:58] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.80, 19.05, 17.80
[01:11:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.82, 18.72, 17.84
[01:13:48] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.26, 21.95, 21.69
[01:19:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.09, 20.70, 19.11
[01:23:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.64, 18.73, 18.75
[01:27:48] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.79, 23.68, 23.42
[01:29:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.04, 24.53, 23.77
[01:35:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.99, 22.13, 20.10
[01:37:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.86, 22.40, 20.45
[01:43:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.47, 19.21, 19.78
[01:47:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.37, 22.09, 23.95
[02:00:46] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[02:01:48] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 11.08, 15.72, 19.60
[02:02:46] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 252 system event log (SEL) entries present]
[02:30:46] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.08, 22.25, 19.64
[02:34:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.81, 23.51, 20.80
[02:36:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.07, 24.23, 21.35
[02:39:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.61, 22.58, 18.91
[02:41:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.92, 21.47, 18.98
[02:43:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.06, 19.27, 18.48
[02:44:41] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 15.93, 21.86, 22.07
[02:55:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.67, 19.92, 18.62
[02:58:37] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.91, 19.12, 20.31
[02:59:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 20.21, 19.83, 18.90
[03:22:28] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.94, 21.20, 20.07
[03:25:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.70, 19.66, 18.29
[03:27:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.50, 18.94, 18.21
[03:32:24] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 13.47, 18.13, 19.55
[03:36:23] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.39, 21.18, 20.39
[03:37:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.01, 21.17, 19.39
[03:38:22] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.38, 20.49, 20.20
[03:40:21] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.97, 19.61, 19.95
[03:41:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.15, 19.31, 19.10
[03:56:47] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.22, 22.76, 20.45
[04:04:28] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.73, 23.88, 21.58
[04:08:19] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.01, 23.63, 22.08
[04:10:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.72, 23.53, 22.19
[04:12:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.23, 22.22, 21.92
[04:19:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.68, 18.24, 20.09
[04:25:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.10, 20.60, 20.46
[04:26:04] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.16, 21.92, 19.53
[04:28:01] <icinga-miraheze>	 PROBLEM - wizardia.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wizardia.wiki' expires in 15 day(s) (Thu 08 Aug 2024 03:56:51 AM GMT +0000).
[04:28:03] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.66, 20.36, 19.24
[04:28:13] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/92762f9d2429...93d26cb96bd3
[04:28:16] <botifico>	 [02ssl] 07WikiTideSSLBot 0393d26cb - Bot: Update SSL cert for wizardia.wiki
[04:31:16] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:31:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.96, 23.58, 21.56
[04:33:14] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001732617617 secs
[04:34:01] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 30.38, 25.12, 21.43
[04:35:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.98, 21.45, 21.21
[04:43:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.64, 19.40, 20.24
[04:43:58] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.41, 22.18, 22.06
[04:44:48] <icinga-miraheze>	 PROBLEM - wiki.yunachannel.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.yunachannel.com' expires in 15 day(s) (Thu 08 Aug 2024 04:23:15 AM GMT +0000).
[04:44:59] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/93d26cb96bd3...1f6d5e629861
[04:45:02] <botifico>	 [02ssl] 07WikiTideSSLBot 031f6d5e6 - Bot: Update SSL cert for wiki.yunachannel.com
[04:47:34] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns]
[04:51:55] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.96, 18.48, 20.36
[04:57:02] <icinga-miraheze>	 RECOVERY - wizardia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wizardia.wiki' will expire on Mon 21 Oct 2024 03:28:07 AM GMT +0000.
[05:02:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:04:27] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.58, 3.31, 1.42
[05:06:27] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.61, 3.01, 1.52
[05:08:31] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.72, 4.70, 2.33
[05:10:27] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.77, 3.49, 2.16
[05:12:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:12:27] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.26, 2.35, 1.90
[05:12:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.27, 20.67, 19.92
[05:14:33] <icinga-miraheze>	 RECOVERY - wiki.yunachannel.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yunachannel.com' will expire on Mon 21 Oct 2024 03:44:54 AM GMT +0000.
[05:15:49] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[05:16:47] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.02, 19.79, 18.05
[05:18:42] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.96, 21.43, 18.84
[05:24:28] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.58, 21.68, 19.78
[05:26:23] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.63, 19.86, 19.34
[05:32:41] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.40, 23.33, 23.71
[05:44:37] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 11.75, 16.06, 19.73
[05:46:32] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns]
[05:54:40] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[05:56:42] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [253 system event log (SEL) entries present]
[05:57:32] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:59:29] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -5.984306335e-05 secs
[06:02:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:14:00] <icinga-miraheze>	 PROBLEM - wiki.kirbygang.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.kirbygang.com' expires in 15 day(s) (Thu 08 Aug 2024 05:46:20 AM GMT +0000).
[06:27:27] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:29:25] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -7.212162018e-05 secs
[06:32:29] <icinga-miraheze>	 PROBLEM - wiki.coolstation.space - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.coolstation.space' expires in 15 day(s) (Thu 08 Aug 2024 06:13:14 AM GMT +0000).
[06:32:41] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1f6d5e629861...d4dc0f357c35
[06:32:43] <botifico>	 [02ssl] 07WikiTideSSLBot 03d4dc0f3 - Bot: Update SSL cert for wiki.coolstation.space
[06:47:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[07:02:12] <icinga-miraheze>	 RECOVERY - wiki.coolstation.space - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.coolstation.space' will expire on Mon 21 Oct 2024 05:32:35 AM GMT +0000.
[07:13:08] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[07:13:33] <icinga-miraheze>	 PROBLEM - internalwiki.coolstation.space - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'internalwiki.coolstation.space' expires in 15 day(s) (Thu 08 Aug 2024 06:48:44 AM GMT +0000).
[07:13:46] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d4dc0f357c35...b424907db22b
[07:13:49] <botifico>	 [02ssl] 07WikiTideSSLBot 03b424907 - Bot: Update SSL cert for internalwiki.coolstation.space
[07:22:04] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.94, 20.81, 18.05
[07:28:02] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.93, 22.94, 20.08
[07:31:07] <icinga-miraheze>	 PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - fc.songcontests.eu All nameservers failed to answer the query.
[07:34:00] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.74, 20.28, 19.77
[07:34:29] <icinga-miraheze>	 PROBLEM - equestripedia.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - equestripedia.org All nameservers failed to answer the query.
[07:35:25] <icinga-miraheze>	 PROBLEM - tech.exputra.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - tech.exputra.com All nameservers failed to answer the query.
[07:35:25] <icinga-miraheze>	 PROBLEM - tep.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - tep.wiki All nameservers failed to answer the query.
[07:35:58] <icinga-miraheze>	 PROBLEM - amps.wiki.gd - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[07:36:40] <icinga-miraheze>	 PROBLEM - gogigantic.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - gogigantic.wiki All nameservers failed to answer the query.
[07:36:56] <icinga-miraheze>	 PROBLEM - pwiki.drydraytonvillagehall.org.uk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - pwiki.drydraytonvillagehall.org.uk All nameservers failed to answer the query.
[07:37:27] <icinga-miraheze>	 PROBLEM - valsora.tep.wiki - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[07:37:29] <icinga-miraheze>	 PROBLEM - wiki.creaturathegame.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.creaturathegame.com All nameservers failed to answer the query.
[07:37:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.31, 21.36, 18.57
[07:38:02] <icinga-miraheze>	 PROBLEM - wiki.projectdiablo2.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.projectdiablo2.com All nameservers failed to answer the query.
[07:38:18] <icinga-miraheze>	 PROBLEM - wiki.candelabrem.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[07:38:23] <icinga-miraheze>	 PROBLEM - wiki.blackorchestra.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.blackorchestra.net All nameservers failed to answer the query.
[07:38:23] <icinga-miraheze>	 PROBLEM - valsora.tep.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valsora.tep.wiki All nameservers failed to answer the query.
[07:38:29] <icinga-miraheze>	 PROBLEM - wiki.secondrenaissance.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.secondrenaissance.net All nameservers failed to answer the query.
[07:39:34] <icinga-miraheze>	 PROBLEM - amps.wiki.gd - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - amps.wiki.gd All nameservers failed to answer the query.
[07:39:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.32, 19.11, 18.06
[07:40:44] <icinga-miraheze>	 PROBLEM - wiki.apico.buzz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.apico.buzz All nameservers failed to answer the query.
[07:42:17] <icinga-miraheze>	 PROBLEM - wiki.dwarvesrp.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.dwarvesrp.com All nameservers failed to answer the query.
[07:42:40] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[07:43:01] <icinga-miraheze>	 PROBLEM - wiki.danvs.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.danvs.net All nameservers failed to answer the query.
[07:43:37] <icinga-miraheze>	 PROBLEM - wiki.otir.nl - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.otir.nl All nameservers failed to answer the query.
[07:44:33] <icinga-miraheze>	 PROBLEM - tep.wiki - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[07:44:42] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [254 system event log (SEL) entries present]
[07:45:55] <icinga-miraheze>	 PROBLEM - wiki.bikinibottomdiscord.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.bikinibottomdiscord.org All nameservers failed to answer the query.
[07:46:34] <icinga-miraheze>	 PROBLEM - kb.nena.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[07:46:38] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:47:27] <icinga-miraheze>	 PROBLEM - www.hebammenwiki.de - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.hebammenwiki.de All nameservers failed to answer the query.
[07:48:36] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -4.425644875e-05 secs
[07:48:54] <icinga-miraheze>	 PROBLEM - wiki.candelabrem.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.candelabrem.com All nameservers failed to answer the query.
[07:51:21] <icinga-miraheze>	 PROBLEM - guia.cineastas.pt - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - guia.cineastas.pt All nameservers failed to answer the query.
[07:54:49] <icinga-miraheze>	 PROBLEM - wiki.starshipsailing.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.starshipsailing.net All nameservers failed to answer the query.
[08:02:04] <icinga-miraheze>	 PROBLEM - wiki.esufranchise.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.esufranchise.com All nameservers failed to answer the query.
[08:02:50] <icinga-miraheze>	 PROBLEM - wiki.make717.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.make717.org All nameservers failed to answer the query.
[08:03:58] <icinga-miraheze>	 RECOVERY - equestripedia.org - reverse DNS on sslhost is OK: SSL OK - equestripedia.org reverse DNS resolves to cp37.wikitide.net - CNAME FLAT
[08:04:20] <icinga-miraheze>	 RECOVERY - tep.wiki - reverse DNS on sslhost is OK: SSL OK - tep.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:04:48] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.75, 21.20, 20.46
[08:05:24] <icinga-miraheze>	 RECOVERY - amps.wiki.gd - LetsEncrypt on sslhost is OK: OK - Certificate 'amps.wiki.gd' will expire on Mon 09 Sep 2024 05:49:32 PM GMT +0000.
[08:06:22] <icinga-miraheze>	 RECOVERY - pwiki.drydraytonvillagehall.org.uk - reverse DNS on sslhost is OK: SSL OK - pwiki.drydraytonvillagehall.org.uk reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:06:37] <icinga-miraheze>	 RECOVERY - wiki.creaturathegame.com - reverse DNS on sslhost is OK: SSL OK - wiki.creaturathegame.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:06:55] <icinga-miraheze>	 RECOVERY - valsora.tep.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'valsora.tep.wiki' will expire on Sun 29 Sep 2024 03:32:02 PM GMT +0000.
[08:07:06] <icinga-miraheze>	 RECOVERY - wiki.candelabrem.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.candelabrem.com' will expire on Tue 17 Sep 2024 08:22:32 PM GMT +0000.
[08:07:17] <icinga-miraheze>	 RECOVERY - valsora.tep.wiki - reverse DNS on sslhost is OK: SSL OK - valsora.tep.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:07:28] <icinga-miraheze>	 RECOVERY - wiki.secondrenaissance.net - reverse DNS on sslhost is OK: SSL OK - wiki.secondrenaissance.net reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:08:02] <icinga-miraheze>	 PROBLEM - wiki.fatcatfablab.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.fatcatfablab.org All nameservers failed to answer the query.
[08:08:08] <icinga-miraheze>	 RECOVERY - amps.wiki.gd - reverse DNS on sslhost is OK: SSL OK - amps.wiki.gd reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:10:19] <icinga-miraheze>	 PROBLEM - wiki.yunachannel.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.yunachannel.com All nameservers failed to answer the query.
[08:10:46] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.58, 23.46, 21.47
[08:11:58] <icinga-miraheze>	 PROBLEM - airlineinsider.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - airlineinsider.org All nameservers failed to answer the query.
[08:12:00] <icinga-miraheze>	 RECOVERY - wiki.dwarvesrp.com - reverse DNS on sslhost is OK: SSL OK - wiki.dwarvesrp.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:12:28] <icinga-miraheze>	 RECOVERY - wiki.danvs.net - reverse DNS on sslhost is OK: SSL OK - wiki.danvs.net reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:12:40] <icinga-miraheze>	 RECOVERY - internalwiki.coolstation.space - LetsEncrypt on sslhost is OK: OK - Certificate 'internalwiki.coolstation.space' will expire on Mon 21 Oct 2024 06:13:39 AM GMT +0000.
[08:12:45] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.53, 21.63, 21.05
[08:13:15] <icinga-miraheze>	 RECOVERY - wiki.otir.nl - reverse DNS on sslhost is OK: SSL OK - wiki.otir.nl reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:13:35] <icinga-miraheze>	 RECOVERY - tep.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tep.wiki' will expire on Wed 09 Oct 2024 09:10:56 PM GMT +0000.
[08:14:45] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.33, 22.71, 21.45
[08:16:06] <icinga-miraheze>	 RECOVERY - kb.nena.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kb.nena.org' will expire on Sun 01 Sep 2024 11:12:12 AM GMT +0000.
[08:16:42] <icinga-miraheze>	 RECOVERY - www.hebammenwiki.de - reverse DNS on sslhost is OK: SSL OK - www.hebammenwiki.de reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:18:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.76, 23.29, 22.17
[08:21:06] <icinga-miraheze>	 RECOVERY - guia.cineastas.pt - reverse DNS on sslhost is OK: SSL OK - guia.cineastas.pt reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:24:15] <icinga-miraheze>	 RECOVERY - wiki.starshipsailing.net - reverse DNS on sslhost is OK: SSL OK - wiki.starshipsailing.net reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:24:42] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.29, 23.14, 22.18
[08:26:41] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 16.59, 20.63, 21.39
[08:29:20] <icinga-miraheze>	 RECOVERY - fc.songcontests.eu - reverse DNS on sslhost is OK: SSL OK - fc.songcontests.eu reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:30:38] <icinga-miraheze>	 RECOVERY - wiki.esufranchise.com - reverse DNS on sslhost is OK: SSL OK - wiki.esufranchise.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:30:40] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 29.58, 24.48, 22.68
[08:32:02] <botifico>	 [02ManageWiki] 07waki285 opened pull request 03#476: Create an alias for special pages in Japanese - 13https://github.com/miraheze/ManageWiki/pull/476
[08:32:08] <botifico>	 [02ManageWiki] 07coderabbitai[bot] commented on pull request 03#476: Create an alias for special pages in Japanese - 13https://github.com/miraheze/ManageWiki/pull/476#issuecomment-2244586056
[08:32:10] <icinga-miraheze>	 RECOVERY - wiki.make717.org - reverse DNS on sslhost is OK: SSL OK - wiki.make717.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:33:25] <icinga-miraheze>	 RECOVERY - tech.exputra.com - reverse DNS on sslhost is OK: SSL OK - tech.exputra.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:34:33] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.45, 21.83, 18.82
[08:34:40] <icinga-miraheze>	 RECOVERY - gogigantic.wiki - reverse DNS on sslhost is OK: SSL OK - gogigantic.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:36:03] <icinga-miraheze>	 RECOVERY - wiki.projectdiablo2.com - reverse DNS on sslhost is OK: SSL OK - wiki.projectdiablo2.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:36:24] <icinga-miraheze>	 RECOVERY - wiki.blackorchestra.net - reverse DNS on sslhost is OK: SSL OK - wiki.blackorchestra.net reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:36:28] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.62, 18.96, 18.13
[08:36:36] <icinga-miraheze>	 RECOVERY - wiki.fatcatfablab.org - reverse DNS on sslhost is OK: SSL OK - wiki.fatcatfablab.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:39:38] <icinga-miraheze>	 RECOVERY - wiki.yunachannel.com - reverse DNS on sslhost is OK: SSL OK - wiki.yunachannel.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:40:37] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.72, 23.31, 23.42
[08:40:42] <icinga-miraheze>	 RECOVERY - wiki.apico.buzz - reverse DNS on sslhost is OK: SSL OK - wiki.apico.buzz reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:41:24] <icinga-miraheze>	 RECOVERY - airlineinsider.org - reverse DNS on sslhost is OK: SSL OK - airlineinsider.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:43:55] <icinga-miraheze>	 RECOVERY - wiki.bikinibottomdiscord.org - reverse DNS on sslhost is OK: SSL OK - wiki.bikinibottomdiscord.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:46:55] <icinga-miraheze>	 RECOVERY - wiki.candelabrem.com - reverse DNS on sslhost is OK: SSL OK - wiki.candelabrem.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:56:32] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 13.28, 17.26, 19.92
[09:09:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.62, 19.08, 17.57
[09:13:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.17, 19.53, 18.32
[09:24:47] <icinga-miraheze>	 PROBLEM - progressbar95.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - progressbar95.wiki All nameservers failed to answer the query.
[09:24:47] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.13, 20.77, 19.24
[09:26:30] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[09:27:21] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.44, 22.61, 20.12
[09:28:28] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0009821653366 secs
[09:28:38] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.43, 19.65, 19.16
[09:31:19] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.91, 23.24, 20.99
[09:33:19] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.79, 24.41, 21.68
[09:35:18] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.93, 22.29, 21.25
[09:39:17] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 9.71, 17.28, 19.61
[09:44:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.88, 21.01, 19.76
[09:46:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.22, 22.38, 20.36
[09:48:50] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.11, 20.16, 19.79
[09:53:32] <icinga-miraheze>	 RECOVERY - progressbar95.wiki - reverse DNS on sslhost is OK: SSL OK - progressbar95.wiki reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[09:54:37] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.92, 23.29, 21.00
[09:56:32] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.22, 23.23, 21.27
[10:00:23] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.63, 19.42, 20.19
[10:07:02] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[10:09:00] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0008854568005 secs
[10:23:28] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.91, 21.68, 20.03
[10:31:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.92, 22.79, 21.77
[10:32:40] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[10:34:42] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [256 system event log (SEL) entries present]
[10:35:00] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.83, 18.97, 20.40
[11:01:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:02:50] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[11:06:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:08:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:08:59] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.074 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[11:13:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:14:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:15:18] <miraheze-github>	 miraheze/ManageWiki - waki285 the build passed.
[11:16:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.20, 3.57, 2.45
[11:18:08] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.57, 2.49, 2.18
[11:19:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:19:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.52, 22.63, 19.21
[11:43:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.24, 21.72, 22.94
[11:54:14] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[11:55:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.35, 22.20, 21.92
[11:56:22] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:58:25] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001292586327 secs
[11:59:56] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.03, 21.44, 21.80
[12:05:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.85, 18.74, 20.36
[12:11:58] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:13:56] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -8.225440979e-06 secs
[12:27:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.91, 19.81, 18.44
[12:29:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.85, 22.09, 19.47
[12:31:56] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.55, 20.54, 19.22
[12:33:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.13, 22.78, 20.17
[12:39:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.49, 21.72, 20.82
[12:45:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.26, 20.02, 20.37
[12:49:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.32, 21.08, 20.68
[12:51:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.78, 23.75, 21.72
[13:05:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.64, 22.45, 23.13
[13:19:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.86, 18.59, 20.25
[13:23:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.24, 21.32, 21.03
[13:25:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.15, 23.26, 21.82
[13:27:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.07, 23.42, 22.12
[13:29:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.04, 24.80, 22.75
[13:37:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 15.58, 23.99, 23.97
[13:40:40] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[13:42:42] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [258 system event log (SEL) entries present]
[13:47:33] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[13:51:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 13.61, 17.19, 20.26
[14:15:59] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:33:16] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:37:21] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0001313388348 secs
[14:46:15] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[15:03:39] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:05:37] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.001594185829 secs
[15:53:19] <icinga-miraheze>	 PROBLEM - index.pdcommunity.ir - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - index.pdcommunity.ir All nameservers failed to answer the query.
[16:07:48] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.69, 21.08, 15.65
[16:09:48] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.37, 20.94, 16.30
[16:11:47] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.59, 20.08, 16.55
[16:12:51] <botifico>	 [02mediawiki-repos] 07redbluegreenhat pushed 031 commit to 03T12170 [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/commit/d57c1e32e496
[16:12:52] <botifico>	 [02mediawiki-repos] 07redbluegreenhat 03d57c1e3 - T12170: Install FeaturedFeeds
[16:12:55] <botifico>	 [02mediawiki-repos] 07redbluegreenhat created branch 03T12170 - 13https://github.com/miraheze/mediawiki-repos
[16:12:59] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:13:05] <botifico>	 [02mediawiki-repos] 07redbluegreenhat opened pull request 03#28: T12170: Install FeaturedFeeds - 13https://github.com/miraheze/mediawiki-repos/pull/28
[16:13:11] <botifico>	 [02mediawiki-repos] 07coderabbitai[bot] commented on pull request 03#28: T12170: Install FeaturedFeeds - 13https://github.com/miraheze/mediawiki-repos/pull/28#issuecomment-2245665455
[16:14:57] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0005654990673 secs
[16:22:00] <icinga-miraheze>	 RECOVERY - index.pdcommunity.ir - reverse DNS on sslhost is OK: SSL OK - index.pdcommunity.ir reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:24:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.32, 22.14, 19.08
[16:26:43] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.03, 21.13, 19.12
[16:28:42] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 13.16, 18.50, 18.41
[16:32:38] <botifico>	 [02mw-config] 07redbluegreenhat pushed 031 commit to 03T12170 [+0/-0/±3] 13https://github.com/miraheze/mw-config/commit/fbcb1b3deef2
[16:32:39] <botifico>	 [02mw-config] 07redbluegreenhat 03fbcb1b3 - T12170: Install FeaturedFeeds
[16:32:41] <botifico>	 [02mw-config] 07redbluegreenhat created branch 03T12170 - 13https://github.com/miraheze/mw-config
[16:32:50] <botifico>	 [02mw-config] 07redbluegreenhat opened pull request 03#5622: T12170: Install FeaturedFeeds - 13https://github.com/miraheze/mw-config/pull/5622
[16:33:45] <miraheze-github>	 miraheze/mw-config - redbluegreenhat the build has errored.
[16:33:50] <botifico>	 [02mw-config] 07redbluegreenhat pushed 031 commit to 03T12170 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/fbcb1b3deef2...9492d8ec5ac3
[16:33:53] <botifico>	 [02mw-config] 07redbluegreenhat 039492d8e - fix
[16:33:54] <botifico>	 [02mw-config] 07redbluegreenhat synchronize pull request 03#5622: T12170: Install FeaturedFeeds - 13https://github.com/miraheze/mw-config/pull/5622
[16:34:21] <botifico>	 [02mw-config] 07github-actions[bot] pushed 031 commit to 03T12170 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9492d8ec5ac3...026e5c091a95
[16:34:24] <botifico>	 [02mw-config] 07github-actions 03026e5c0 - CI: lint code to MediaWiki standards
[16:34:27] <botifico>	 [02mw-config] 07github-actions[bot] synchronize pull request 03#5622: T12170: Install FeaturedFeeds - 13https://github.com/miraheze/mw-config/pull/5622
[16:34:52] <miraheze-github>	 miraheze/mw-config - redbluegreenhat the build passed.
[16:38:38] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 29.83, 25.07, 21.32
[16:43:14] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[16:43:46] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[16:45:12] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.000667899847 secs
[16:58:32] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.80, 23.87, 24.00
[17:00:31] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 23.84, 24.16, 24.11
[17:02:31] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 11.18, 19.61, 22.48
[17:02:38] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:02:40] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[17:04:42] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [260 system event log (SEL) entries present]
[17:07:38] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:08:29] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.89, 16.51, 20.04
[17:11:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:13:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 8.81, 4.92, 2.51
[17:15:07] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.46, 3.98, 2.44
[17:16:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:17:04] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:17:04] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.95, 3.28, 2.38
[17:19:02] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0008762478828 secs
[17:20:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:22:56] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 9.01, 5.47, 3.46
[17:23:23] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.96, 20.88, 19.68
[17:25:22] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.33, 22.90, 20.57
[17:25:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:26:49] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.87, 3.91, 3.30
[17:28:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.15, 4.16, 3.44
[17:29:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:29:21] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.35, 23.45, 21.39
[17:30:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.31, 3.72, 3.38
[17:31:56] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.51, 20.81, 17.51
[17:32:40] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.14, 3.28, 3.26
[17:33:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.95, 20.95, 18.00
[17:34:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:35:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.78, 19.84, 17.98
[17:36:25] <botifico>	 [02dns] 07MacFan4000 opened pull request 03#536: add fanojo.wiki and kagaga.jp zones - 13https://github.com/miraheze/dns/pull/536
[17:37:28] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:37:55] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:38:27] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.77, 3.91, 3.46
[17:38:30] <MacFan4000>	 Could somebody please merge https://github.com/miraheze/dns/pull/536?
[17:41:34] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0003603100777 secs
[17:42:55] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:43:17] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.71, 21.95, 21.17
[17:43:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.41, 20.43, 19.05
[17:44:27] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.76, 3.68, 3.54
[17:46:01] <botifico>	 [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b424907db22b...70fd36731c2c
[17:46:03] <botifico>	 [02ssl] 07MacFan4000 0370fd367 - add junxions.miraheze.org redirect
[17:46:27] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.82, 2.67, 3.18
[17:47:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.44, 19.81, 19.12
[17:51:22] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/ssl/compare/70fd36731c2c...8f4865064798
[17:51:23] <botifico>	 [02ssl] 07WikiTideSSLBot 038f48650 - Bot: Update SSL cert for iewiki.auroville.org
[17:51:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.80, 21.64, 20.03
[17:57:04] <botifico>	 [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8f4865064798...522c074a9c5b
[17:57:05] <botifico>	 [02ssl] 07MacFan4000 03522c074 - add iewiki.auroville.org redirect
[17:57:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.02, 18.81, 19.44
[18:05:10] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.86, 22.31, 23.72
[18:18:46] <icinga-miraheze>	 PROBLEM - iewiki.auroville.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns05.domaincontrol.com.', 'ns06.domaincontrol.com.'], 'CNAME': 'iewiki.purnamcommunity.in.'}
[18:19:46] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.44, 21.06, 19.46
[18:21:42] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.10, 20.45, 19.46
[18:23:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.08, 20.28, 21.19
[18:27:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 17.95, 21.09, 21.51
[18:27:31] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.13, 20.12, 19.72
[18:33:03] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 19.72, 18.96, 20.34
[18:37:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.94, 22.31, 21.45
[18:41:01] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:41:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.23, 23.35, 21.95
[18:43:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.84, 22.10, 21.67
[18:45:19] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001316428185 secs
[18:47:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 29.60, 25.51, 23.11
[18:51:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.78, 21.99, 22.28
[18:55:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.98, 24.05, 22.97
[18:57:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 14.87, 20.98, 22.02
[19:01:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.94, 23.20, 22.54
[19:05:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.07, 23.36, 22.82
[19:11:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.89, 23.72, 23.13
[19:31:07] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[19:31:30] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.90, 22.52, 19.77
[19:33:06] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0005086958408 secs
[19:35:23] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.42, 22.25, 20.42
[19:37:20] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 13.96, 19.68, 19.71
[19:47:00] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:53:01] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[19:54:39] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[19:55:00] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001904368401 secs
[19:55:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 10.84, 18.98, 22.87
[19:56:35] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 266 system event log (SEL) entries present]
[19:59:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 29.82, 24.04, 23.93
[20:01:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.02, 22.43, 23.37
[20:05:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.76, 23.10, 23.35
[20:07:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.62, 21.06, 22.56
[20:17:04] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:23:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.30, 21.06, 21.08
[20:25:49] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:27:48] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0008046627045 secs
[20:29:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.99, 23.56, 22.53
[20:31:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.28, 24.94, 23.15
[20:37:34] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.48, 22.68, 19.47
[20:41:27] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.43, 22.81, 20.43
[20:42:54] <botifico>	 [02dns] 07The-Voidwalker closed pull request 03#536: add fanojo.wiki and kagaga.jp zones - 13https://github.com/miraheze/dns/pull/536
[20:42:57] <botifico>	 [02dns] 07The-Voidwalker pushed 031 commit to 03master [+2/-0/±0] 13https://github.com/miraheze/dns/compare/62562ac344af...c6b8b5dd066a
[20:42:59] <botifico>	 [02dns] 07MacFan4000 03c6b8b5d - add fanojo.wiki and kagaga.jp zones (#536)
[20:42:59] <MirahezeRelay>	 <orduin> MacFan4000, done
[20:43:23] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.35, 20.29, 19.78
[21:01:53] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.61, 20.40, 19.10
[21:05:46] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 20.29, 19.86, 19.14
[21:13:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 16.61, 20.57, 23.60
[21:13:26] <icinga-miraheze>	 PROBLEM - www.dcmultiversewiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dcmultiversewiki.com' expires in 15 day(s) (Thu 08 Aug 2024 08:59:39 PM GMT +0000).
[21:15:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.79, 23.20, 24.17
[21:19:04] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[21:20:23] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.88, 19.94, 19.06
[21:23:22] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001109063625 secs
[21:24:16] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.17, 22.66, 20.30
[21:26:12] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.34, 21.27, 20.12
[21:29:52] <icinga-miraheze>	 PROBLEM - dcmultiversewiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dcmultiversewiki.com' expires in 15 day(s) (Thu 08 Aug 2024 08:59:39 PM GMT +0000).
[21:30:05] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.39, 18.54, 19.34
[21:30:26] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[21:31:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 13.66, 20.09, 23.77
[21:32:27] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 268 system event log (SEL) entries present]
[21:35:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.38, 22.85, 24.07
[21:41:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.70, 22.54, 23.76
[21:45:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.14, 23.94, 23.89
[21:45:32] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:45:41] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.65, 20.59, 19.64
[21:47:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.90, 23.77, 23.90
[21:47:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:47:38] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 12.57, 17.39, 18.58
[21:49:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.26, 23.45, 23.73
[21:51:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.21, 22.65, 23.43
[21:53:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.86, 23.77, 23.71
[22:05:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 17.33, 21.87, 23.59
[22:13:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.08, 21.62, 22.26
[22:14:14] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:15:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.61, 20.82, 21.91
[22:21:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.86, 22.79, 22.21
[22:29:25] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.63, 21.79, 19.03
[22:31:21] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.35, 20.15, 18.77
[22:38:51] <icinga-miraheze>	 PROBLEM - lgbtqia.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lgbtqia.wiki' expires in 15 day(s) (Thu 08 Aug 2024 10:24:14 PM GMT +0000).
[22:39:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 16.59, 20.80, 23.26
[22:42:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[22:44:26] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[22:45:03] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 11.83, 15.81, 20.38
[22:46:27] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 270 system event log (SEL) entries present]
[22:46:55] <icinga-miraheze>	 PROBLEM - www.lgbtqia.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lgbtqia.wiki' expires in 15 day(s) (Thu 08 Aug 2024 10:24:14 PM GMT +0000).
[22:51:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.47, 22.36, 21.69
[22:55:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.47, 22.72, 22.11
[23:03:11] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:03:11] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.33, 22.48, 21.92
[23:06:16] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.75, 3.53, 1.56
[23:06:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:08:10] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[23:08:10] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[23:09:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.05, 22.83, 22.45
[23:10:10] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 24 minutes ago with 0 failures
[23:10:11] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 52 packages available for upgrade (0 critical updates).
[23:12:16] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.54, 2.98, 2.17
[23:17:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.84, 25.03, 23.41
[23:19:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.22, 23.72, 23.16
[23:21:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.55, 24.73, 23.56
[23:27:46] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.84, 21.55, 19.96
[23:33:35] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.97, 19.95, 19.84
[23:40:26] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[23:41:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 11.69, 19.20, 22.81
[23:42:28] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 272 system event log (SEL) entries present]
[23:51:03] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.55, 22.91, 22.75
[23:54:04] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 30.06, 24.68, 21.53
[23:59:53] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.71, 22.87, 22.13