[00:01:30] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:01:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.21, 24.08, 23.87
[00:03:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.07, 23.94, 23.90
[00:09:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.48, 23.45, 23.53
[00:15:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.96, 22.83, 23.41
[00:21:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.56, 22.37, 22.84
[00:27:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.81, 22.45, 22.95
[00:29:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.22, 22.80, 23.00
[00:31:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.12, 22.86, 23.03
[00:32:37] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.78, 21.01, 19.32
[00:33:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.01, 23.03, 23.06
[00:38:24] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.56, 20.28, 19.64
[00:43:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.13, 23.15, 23.37
[00:49:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.52, 23.96, 23.52
[00:52:38] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[00:54:39] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [234 system event log (SEL) entries present]
[00:55:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:15:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.75, 21.23, 23.56
[01:21:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.54, 22.37, 22.96
[01:21:55] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 29.05, 22.86, 19.69
[01:29:37] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 15.90, 21.43, 20.88
[01:31:33] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.25, 19.95, 20.38
[01:35:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.23, 22.11, 21.19
[01:39:17] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.27, 19.62, 20.37
[02:15:30] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:17:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.34, 20.97, 23.75
[02:21:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.42, 23.41, 24.06
[02:33:02] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 13.94, 17.84, 22.99
[02:41:02] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 12.27, 15.68, 20.03
[02:47:07] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[02:49:08] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [235 system event log (SEL) entries present]
[02:51:02] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.78, 21.68, 20.09
[03:03:02] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.47, 23.75, 23.14
[03:19:02] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 12.67, 17.40, 20.21
[03:45:02] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 29.66, 19.80, 16.36
[03:49:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.78, 21.62, 23.92
[03:55:02] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.64, 17.95, 14.38
[03:55:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.70, 23.63, 23.75
[03:57:02] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.07, 17.22, 14.53
[04:01:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.32, 23.76, 23.87
[04:11:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.09, 22.31, 22.70
[04:12:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:13:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.34, 20.18, 21.86
[04:17:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:25:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.18, 22.96, 22.13
[04:27:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.20, 22.69, 22.11
[04:29:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.15, 24.46, 22.81
[04:35:55] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.33, 22.80, 19.40
[04:39:46] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 15.51, 21.26, 19.68
[04:39:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.85, 23.51, 23.70
[04:41:41] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 12.87, 18.45, 18.85
[04:54:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:57:42] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[04:59:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:59:42] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [237 system event log (SEL) entries present]
[04:59:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.20, 21.85, 21.43
[05:01:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.71, 21.04, 21.21
[05:02:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:03:33] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.55, 2.24, 0.97
[05:05:33] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.97, 2.17, 1.08
[05:07:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:07:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.30, 22.71, 21.64
[05:09:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.82, 21.99, 21.51
[05:13:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.21, 22.90, 21.92
[05:15:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.69, 23.64, 22.34
[05:19:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 30.53, 24.88, 22.93
[05:29:42] <icinga-miraheze>	 PROBLEM - db151 Backups SQL on db151 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864187 seconds old and 40860 bytes
[05:35:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.91, 23.01, 23.47
[05:47:15] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 29.73, 22.89, 19.37
[05:49:11] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 16.75, 20.59, 18.97
[05:51:06] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.51, 18.89, 18.53
[05:53:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.42, 22.21, 21.72
[05:55:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.84, 22.40, 21.89
[06:05:48] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.45, 19.58, 20.38
[06:09:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.19, 22.26, 21.19
[06:17:02] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 14.28, 21.12, 23.64
[06:23:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.01, 23.38, 23.04
[06:25:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.70, 24.04, 23.32
[06:27:02] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.27, 24.84, 24.09
[06:27:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.45, 22.40, 22.79
[06:29:55] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.99, 19.92, 16.92
[06:33:52] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.08, 19.47, 17.44
[06:37:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.94, 24.86, 23.68
[06:45:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.02, 23.09, 23.78
[06:48:12] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sel_parse: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[06:50:13] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [238 system event log (SEL) entries present]
[06:51:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.70, 23.85, 23.71
[06:53:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.11, 22.91, 23.35
[06:55:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.71, 23.54, 23.54
[07:01:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.17, 21.92, 23.02
[07:07:02] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.46, 21.34, 23.39
[07:09:02] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.87, 22.11, 23.37
[07:09:48] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.82, 17.74, 20.40
[07:11:02] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.85, 23.18, 23.65
[07:13:02] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.00, 24.38, 24.02
[07:15:02] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.13, 23.46, 23.81
[07:18:46] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.06, 20.86, 20.60
[07:20:45] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.97, 21.12, 20.68
[07:21:02] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 10.67, 14.53, 19.59
[07:22:45] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.25, 23.64, 21.65
[07:31:04] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[07:33:10] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:34:27] <icinga-miraheze>	 PROBLEM - wiki.myehs.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.myehs.eu All nameservers failed to answer the query.
[07:35:15] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[07:36:13] <icinga-miraheze>	 PROBLEM - www.gengbaike.top - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.gengbaike.top All nameservers failed to answer the query.
[07:37:19] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0009608864784 secs
[07:39:46] <icinga-miraheze>	 PROBLEM - wiki.gesamtschule-nordkirchen.de - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gesamtschule-nordkirchen.de All nameservers failed to answer the query.
[07:41:40] <icinga-miraheze>	 PROBLEM - wiki.seredrau.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seredrau.xyz All nameservers failed to answer the query.
[07:48:02] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[07:50:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.99, 21.99, 23.38
[07:51:20] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query.
[07:52:22] <icinga-miraheze>	 PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.qadrishattari.xyz. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[07:54:37] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.36, 22.20, 23.01
[07:56:37] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.46, 22.07, 22.89
[08:02:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.95, 23.91, 23.25
[08:03:54] <icinga-miraheze>	 RECOVERY - wiki.myehs.eu - reverse DNS on sslhost is OK: SSL OK - wiki.myehs.eu reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:05:58] <icinga-miraheze>	 RECOVERY - www.gengbaike.top - reverse DNS on sslhost is OK: SSL OK - www.gengbaike.top reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:08:33] <icinga-miraheze>	 RECOVERY - wiki.gesamtschule-nordkirchen.de - reverse DNS on sslhost is OK: SSL OK - wiki.gesamtschule-nordkirchen.de reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:10:51] <icinga-miraheze>	 RECOVERY - wiki.seredrau.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.seredrau.xyz reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:14:51] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[08:20:54] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:21:41] <icinga-miraheze>	 RECOVERY - wiki.qadrishattari.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.qadrishattari.xyz reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:44:40] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[08:46:41] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [239 system event log (SEL) entries present]
[08:53:14] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:55:13] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0003810822964 secs
[09:01:46] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 38.46.223.205/cpweb
[09:03:41] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[09:03:50] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[09:05:45] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0004506409168 secs
[09:27:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[09:34:13] <icinga-miraheze>	 PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.gab.pt.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[09:52:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:36:06] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[10:38:01] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0007666349411 secs
[10:46:23] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[10:48:18] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0007145404816 secs
[10:55:53] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[10:57:48] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0005142986774 secs
[11:01:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:02:10] <icinga-miraheze>	 RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[11:06:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:25:24] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.30, 20.04, 17.67
[11:29:21] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.36, 23.00, 19.50
[11:31:19] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.15, 19.65, 18.70
[11:34:10] <icinga-miraheze>	 PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query.
[12:03:20] <icinga-miraheze>	 PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[12:12:53] <botifico>	 [02landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/a822bf47bcfc...dc12b1ddf728
[12:12:54] <botifico>	 [02landing] 07translatewiki 03dc12b1d - Localisation updates from https://translatewiki.net.
[12:12:56] <botifico>	 [02CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/2deb4261fb64...208bb2b96ad9
[12:12:57] <botifico>	 [02CreateWiki] 07translatewiki 03208bb2b - Localisation updates from https://translatewiki.net.
[12:12:59] <botifico>	 [02MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/375fd6d3bc61...7592532880b6
[12:13:02] <botifico>	 [02MirahezeMagic] 07translatewiki 037592532 - Localisation updates from https://translatewiki.net.
[12:13:48] <miraheze-github>	 miraheze/landing - translatewiki the build passed.
[12:16:21] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build has errored.
[12:16:48] <miraheze-github>	 miraheze/CreateWiki - translatewiki the build has errored.
[12:19:49] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.10, 19.25, 17.29
[12:21:14] <botifico>	 [02mw-config] 07BlankEclair opened pull request 03#5621: T12370: Allow for NATURAL_SORT_KEY on rainversewiki - 13https://github.com/miraheze/mw-config/pull/5621
[12:21:47] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.93, 19.16, 17.48
[12:22:09] <miraheze-github>	 miraheze/mw-config - BlankEclair the build passed.
[12:23:51] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151
[12:23:52] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:23:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:03] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151
[12:24:04] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:24:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:25:57] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:27:42] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.22, 22.68, 19.17
[12:27:57] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0003278553486 secs
[12:29:41] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.92, 22.41, 19.53
[12:31:39] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.93, 19.89, 18.96
[12:32:26] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all
[12:32:30] <icinga-miraheze>	 RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[12:32:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:32:40] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 13s
[12:32:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:36:09] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[12:37:30] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all
[12:37:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:37:41] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 11s
[12:37:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:38:04] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.000320315361 secs
[12:45:49] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[13:14:40] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[13:22:24] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:24:19] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0003591775894 secs
[13:37:02] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[13:38:58] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0007455050945 secs
[14:01:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.02, 21.28, 23.21
[14:09:48] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.58, 17.13, 20.29
[14:14:34] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[14:16:29] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0003716051579 secs
[14:52:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:54:20] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[14:56:25] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:58:20] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.001207053661 secs
[15:48:27] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.07, 20.94, 19.39
[15:50:31] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.74, 19.95, 19.22
[16:04:10] <icinga-miraheze>	 PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[16:16:12] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.56, 22.09, 18.89
[16:18:08] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.51, 20.50, 18.69
[16:20:04] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 19.10, 20.25, 18.82
[16:21:06] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[16:23:02] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0002706944942 secs
[16:33:20] <icinga-miraheze>	 PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query.
[17:02:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:02:30] <icinga-miraheze>	 RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[17:03:33] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.71, 3.28, 1.34
[17:05:33] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.32, 2.35, 1.23
[17:07:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:18:12] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.20, 22.35, 19.41
[17:22:04] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 17.10, 21.95, 20.09
[17:23:02] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.20, 21.86, 18.53
[17:24:00] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.63, 24.82, 21.38
[17:27:02] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.25, 23.18, 19.85
[17:31:00] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.01, 22.80, 20.31
[17:33:00] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.77, 22.22, 20.41
[17:34:11] <icinga-miraheze>	 PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query.
[17:35:00] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.63, 19.54, 19.66
[17:37:32] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 16.16, 21.70, 22.82
[17:42:52] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/631eb376b786...1df7db82c399
[17:42:53] <botifico>	 [02ssl] 07WikiTideSSLBot 031df7db8 - Bot: Add SSL cert for pickaxe.showierdata.xyz
[17:43:20] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.34, 15.43, 19.63
[17:46:40] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[17:47:14] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.90, 20.63, 20.91
[17:47:41] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1df7db82c399...6efb5adbb849
[17:47:42] <botifico>	 [02ssl] 07WikiTideSSLBot 036efb5ad - Bot: Update SSL cert for wiki.msnld.com
[17:48:02] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/ssl/compare/6efb5adbb849...27ab9392215e
[17:48:05] <botifico>	 [02ssl] 07WikiTideSSLBot 0327ab939 - Bot: Update SSL cert for pokemonwiki.info
[17:48:39] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/27ab9392215e...e6f9075634f8
[17:48:42] <botifico>	 [02ssl] 07WikiTideSSLBot 03e6f9075 - Bot: Update SSL cert for wiki.fatcatfablab.org
[17:48:57] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e6f9075634f8...fd004b6fd4de
[17:48:58] <botifico>	 [02ssl] 07WikiTideSSLBot 03fd004b6 - Bot: Update SSL cert for wiki.misslessfighters.club
[17:49:17] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fd004b6fd4de...ee67a9b428f8
[17:49:18] <botifico>	 [02ssl] 07WikiTideSSLBot 03ee67a9b - Bot: Update SSL cert for www.journeytheword.wiki
[17:49:35] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ee67a9b428f8...6977f8154dae
[17:49:37] <botifico>	 [02ssl] 07WikiTideSSLBot 036977f81 - Bot: Update SSL cert for patternarchive.online
[17:50:00] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6977f8154dae...824729fcd674
[17:50:01] <botifico>	 [02ssl] 07WikiTideSSLBot 03824729f - Bot: Update SSL cert for trollpasta.com
[17:51:06] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.62, 21.58, 21.12
[17:51:27] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/824729fcd674...a8b8b068871b
[17:51:29] <botifico>	 [02ssl] 07WikiTideSSLBot 03a8b8b06 - Bot: Update SSL cert for www.christipedia.nl
[17:51:47] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a8b8b068871b...39ec8c17a026
[17:51:49] <botifico>	 [02ssl] 07WikiTideSSLBot 0339ec8c1 - Bot: Update SSL cert for tl.awiki.org
[17:52:15] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/39ec8c17a026...ce7327ad2637
[17:52:16] <botifico>	 [02ssl] 07WikiTideSSLBot 03ce7327a - Bot: Update SSL cert for dariawiki.org
[17:52:36] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ce7327ad2637...55b9e3af6b0c
[17:52:38] <botifico>	 [02ssl] 07WikiTideSSLBot 0355b9e3a - Bot: Update SSL cert for publictestwiki.com
[17:53:24] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/55b9e3af6b0c...8ea414a162a7
[17:53:26] <botifico>	 [02ssl] 07WikiTideSSLBot 038ea414a - Bot: Update SSL cert for permanentfuturelab.wiki
[17:53:44] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8ea414a162a7...d3177dbeb004
[17:53:47] <botifico>	 [02ssl] 07WikiTideSSLBot 03d3177db - Bot: Update SSL cert for isocasg.org
[17:54:04] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d3177dbeb004...8fced414cba7
[17:54:06] <botifico>	 [02ssl] 07WikiTideSSLBot 038fced41 - Bot: Update SSL cert for wiki.3point0.science
[17:54:25] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8fced414cba7...53862d1f4825
[17:54:26] <botifico>	 [02ssl] 07WikiTideSSLBot 0353862d1 - Bot: Update SSL cert for lostidols.wiki
[17:56:26] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/53862d1f4825...47ee1c6af804
[17:56:27] <botifico>	 [02ssl] 07WikiTideSSLBot 0347ee1c6 - Bot: Update SSL cert for pyramidgames.wiki
[17:59:02] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.49, 23.79, 22.87
[17:59:27] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/47ee1c6af804...57b125a3f987
[17:59:30] <botifico>	 [02ssl] 07WikiTideSSLBot 0357b125a - Bot: Update SSL cert for en.pornwiki.org
[18:00:26] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/57b125a3f987...dea178fbaa24
[18:00:28] <botifico>	 [02ssl] 07WikiTideSSLBot 03dea178f - Bot: Update SSL cert for burnout.wiki
[18:01:02] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.00, 24.35, 23.19
[18:01:05] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/dea178fbaa24...92762f9d2429
[18:01:08] <botifico>	 [02ssl] 07WikiTideSSLBot 0392762f9 - Bot: Update SSL cert for www.johanloopmans.nl
[18:03:22] <icinga-miraheze>	 RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[18:05:02] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.79, 21.71, 22.41
[18:06:11] <icinga-miraheze>	 RECOVERY - wiki.msnld.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.msnld.com' will expire on Sun 20 Oct 2024 04:47:34 PM GMT +0000.
[18:10:26] <icinga-miraheze>	 RECOVERY - wiki.misslessfighters.club - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.misslessfighters.club' will expire on Sun 20 Oct 2024 04:48:51 PM GMT +0000.
[18:11:43] <icinga-miraheze>	 RECOVERY - wiki.fatcatfablab.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fatcatfablab.org' will expire on Sun 20 Oct 2024 04:48:33 PM GMT +0000.
[18:14:42] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query.
[18:15:33] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:17:18] <icinga-miraheze>	 RECOVERY - pokemonwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'pokemonwiki.info' will expire on Sun 20 Oct 2024 04:47:56 PM GMT +0000.
[18:17:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:19:18] <icinga-miraheze>	 RECOVERY - www.trollpasta.com - LetsEncrypt on sslhost is OK: OK - Certificate 'trollpasta.com' will expire on Sun 20 Oct 2024 04:49:54 PM GMT +0000.
[18:20:32] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.76, 21.72, 21.36
[18:22:04] <icinga-miraheze>	 RECOVERY - lostidols.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'lostidols.wiki' will expire on Sun 20 Oct 2024 04:54:19 PM GMT +0000.
[18:23:52] <icinga-miraheze>	 RECOVERY - www.burnout.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'burnout.wiki' will expire on Sun 20 Oct 2024 05:00:20 PM GMT +0000.
[18:23:55] <icinga-miraheze>	 RECOVERY - www.lostidols.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'lostidols.wiki' will expire on Sun 20 Oct 2024 04:54:19 PM GMT +0000.
[18:25:41] <icinga-miraheze>	 RECOVERY - wiki.3point0.science - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.3point0.science' will expire on Sun 20 Oct 2024 04:53:57 PM GMT +0000.
[18:25:41] <icinga-miraheze>	 RECOVERY - www.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.pornwiki.org' will expire on Sun 20 Oct 2024 04:59:19 PM GMT +0000.
[18:25:46] <icinga-miraheze>	 RECOVERY - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Sun 20 Oct 2024 04:53:18 PM GMT +0000.
[18:25:52] <icinga-miraheze>	 RECOVERY - www.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Sun 20 Oct 2024 04:52:30 PM GMT +0000.
[18:25:53] <icinga-miraheze>	 RECOVERY - patternarchive.online - LetsEncrypt on sslhost is OK: OK - Certificate 'patternarchive.online' will expire on Sun 20 Oct 2024 04:49:30 PM GMT +0000.
[18:26:30] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.48, 23.26, 22.56
[18:26:39] <icinga-miraheze>	 RECOVERY - en.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.pornwiki.org' will expire on Sun 20 Oct 2024 04:59:19 PM GMT +0000.
[18:27:10] <icinga-miraheze>	 RECOVERY - publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Sun 20 Oct 2024 04:52:30 PM GMT +0000.
[18:27:26] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.45, 21.03, 19.15
[18:27:41] <icinga-miraheze>	 RECOVERY - tl.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'tl.awiki.org' will expire on Sun 20 Oct 2024 04:51:42 PM GMT +0000.
[18:28:29] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.88, 24.13, 22.94
[18:29:21] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.84, 21.78, 19.65
[18:29:49] <icinga-miraheze>	 RECOVERY - christipedia.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Sun 20 Oct 2024 04:51:22 PM GMT +0000.
[18:30:13] <icinga-miraheze>	 RECOVERY - www.dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Sun 20 Oct 2024 04:52:09 PM GMT +0000.
[18:30:24] <icinga-miraheze>	 RECOVERY - dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Sun 20 Oct 2024 04:52:09 PM GMT +0000.
[18:30:29] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.43, 21.79, 22.20
[18:31:16] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.05, 19.89, 19.19
[18:36:27] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.47, 22.84, 22.28
[18:37:21] <icinga-miraheze>	 RECOVERY - johanloopmans.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.johanloopmans.nl' will expire on Sun 20 Oct 2024 05:00:59 PM GMT +0000.
[18:37:49] <icinga-miraheze>	 RECOVERY - pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.pornwiki.org' will expire on Sun 20 Oct 2024 04:59:19 PM GMT +0000.
[18:37:52] <icinga-miraheze>	 RECOVERY - trollpasta.com - LetsEncrypt on sslhost is OK: OK - Certificate 'trollpasta.com' will expire on Sun 20 Oct 2024 04:49:54 PM GMT +0000.
[18:38:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.39, 20.55, 19.72
[18:40:25] <icinga-miraheze>	 RECOVERY - www.pyramidgames.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'pyramidgames.wiki' will expire on Sun 20 Oct 2024 04:56:18 PM GMT +0000.
[18:40:26] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 17.14, 21.58, 22.02
[18:40:54] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.16, 18.80, 19.21
[18:41:33] <icinga-miraheze>	 RECOVERY - pyramidgames.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'pyramidgames.wiki' will expire on Sun 20 Oct 2024 04:56:18 PM GMT +0000.
[18:42:12] <icinga-miraheze>	 RECOVERY - www.christipedia.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Sun 20 Oct 2024 04:51:22 PM GMT +0000.
[18:42:36] <icinga-miraheze>	 RECOVERY - www.johanloopmans.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.johanloopmans.nl' will expire on Sun 20 Oct 2024 05:00:59 PM GMT +0000.
[18:44:35] <icinga-miraheze>	 RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp36.wikitide.net - CNAME OK
[18:44:37] <icinga-miraheze>	 RECOVERY - isocasg.org - LetsEncrypt on sslhost is OK: OK - Certificate 'isocasg.org' will expire on Sun 20 Oct 2024 04:53:38 PM GMT +0000.
[18:45:01] <icinga-miraheze>	 RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Sun 20 Oct 2024 04:53:18 PM GMT +0000.
[18:46:18] <icinga-miraheze>	 RECOVERY - journeytheword.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.journeytheword.wiki' will expire on Sun 20 Oct 2024 04:49:12 PM GMT +0000.
[18:46:27] <icinga-miraheze>	 RECOVERY - burnout.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'burnout.wiki' will expire on Sun 20 Oct 2024 05:00:20 PM GMT +0000.
[18:47:42] <icinga-miraheze>	 RECOVERY - www.journeytheword.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.journeytheword.wiki' will expire on Sun 20 Oct 2024 04:49:12 PM GMT +0000.
[18:47:58] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns]
[18:48:23] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.06, 21.49, 21.27
[18:55:53] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importDump.php --wiki=spartakuswiki /home/macfan/spartakus_pages_full.xml --no-updates (START)
[18:55:57] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importDump.php --wiki=spartakuswiki /home/macfan/spartakus_pages_full.xml --no-updates (END - exit=256)
[18:55:58] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=spartakuswiki (START)
[18:55:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:55:59] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=spartakuswiki (END - exit=0)
[18:56:00] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/initSiteStats.php --wiki=spartakuswiki --update (END - exit=2)
[18:56:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:56:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:56:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:56:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:56:21] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.31, 23.55, 22.69
[18:57:28] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importDump.php --wiki=spartakuswiki /home/macfan/spartakus_pages_full.xml --no-updates (START)
[18:57:30] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importDump.php --wiki=spartakuswiki /home/macfan/spartakus_pages_full.xml --no-updates (END - exit=256)
[18:57:31] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=spartakuswiki (START)
[18:57:32] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=spartakuswiki (END - exit=2)
[18:57:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:57:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:57:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:57:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:59:49] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php --wiki=spartakuswiki /srv/mediawiki/1.41/vendor/mwstake/mediawiki-component-commonwebapis/sql/mws_category_index.sql (END - exit=0)
[18:59:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:59:54] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importDump.php --wiki=spartakuswiki /home/macfan/spartakus_pages_full.xml --no-updates (START)
[19:00:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:10:16] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.63, 18.79, 20.40
[19:12:38] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importDump.php --wiki=spartakuswiki /home/macfan/spartakus_pages_full.xml --no-updates (END - exit=0)
[19:12:39] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=spartakuswiki (START)
[19:12:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:12:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:15:58] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=spartakuswiki (END - exit=0)
[19:15:59] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/initSiteStats.php --wiki=spartakuswiki --update (END - exit=0)
[19:16:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:16:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:20:12] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.46, 20.26, 20.15
[19:23:46] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importDump.php --wiki=cyberpediawiki /home/macfan/deltarune_pages_full.xml --no-updates (START)
[19:23:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:24:11] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.05, 23.12, 21.24
[19:28:10] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.99, 22.60, 21.55
[19:30:09] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.28, 24.85, 22.50
[19:32:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[19:39:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.62, 19.76, 18.33
[19:43:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.01, 20.04, 18.76
[19:46:04] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 13.79, 19.97, 22.85
[19:46:21] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[19:48:21] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 246 system event log (SEL) entries present]
[19:52:02] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 29.56, 23.89, 23.34
[20:04:38] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importDump.php --wiki=cyberpediawiki /home/macfan/deltarune_pages_full.xml --no-updates (END - exit=0)
[20:04:39] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=cyberpediawiki (START)
[20:04:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:04:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:05:58] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 17.80, 22.70, 23.83
[20:14:01] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[20:21:49] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[20:22:24] <icinga-miraheze>	 PROBLEM - wiki.kscucf.org - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket
[20:23:48] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001283615828 secs
[20:23:52] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.37, 23.23, 22.30
[20:24:49] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 30.29, 23.46, 19.75
[20:28:39] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.16, 23.95, 20.85
[20:30:34] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.23, 24.23, 21.34
[20:34:25] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.93, 23.50, 21.64
[20:36:20] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.86, 24.48, 22.17
[20:37:48] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.05, 23.87, 23.93
[20:39:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.29, 24.23, 24.01
[20:43:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.57, 23.85, 23.94
[20:44:01] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.49, 22.40, 22.51
[20:45:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 23.79, 24.25, 24.10
[20:46:48] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:48:51] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=cyberpediawiki (END - exit=0)
[20:48:53] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/initSiteStats.php --wiki=cyberpediawiki --update (END - exit=0)
[20:48:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:49:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:55:48] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.21, 22.66, 23.67
[20:59:56] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.36, 17.99, 20.23
[21:06:04] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:07:48] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.63, 21.85, 22.01
[21:08:02] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001287937164 secs
[21:09:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.09, 21.28, 21.78
[21:13:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.38, 22.93, 22.15
[21:15:01] <icinga-miraheze>	 PROBLEM - wiki.kscucf.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.kscucf.org could not be found
[21:15:09] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[21:21:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.48, 21.90, 20.06
[21:29:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.22, 19.25, 19.79
[21:33:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.14, 22.28, 20.85
[21:39:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.21, 23.06, 21.83
[21:45:57] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:47:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.41, 18.62, 20.25
[21:51:48] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 17.73, 19.88, 23.46
[21:53:28] <MirahezeLSBot>	 !log [void@mwtask181] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'Cargo'} to all
[21:53:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:53:43] <MirahezeLSBot>	 !log [void@mwtask181] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'Cargo'} to all - SUCCESS in 14s
[21:53:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.38, 21.33, 23.53
[21:53:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:55:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.23, 20.39, 22.91
[21:55:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.87, 21.89, 21.17
[21:59:44] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[22:01:44] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 248 system event log (SEL) entries present]
[22:01:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.67, 19.18, 20.29
[22:03:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.93, 20.46, 21.41
[22:07:48] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.99, 22.18, 22.07
[22:13:54] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:35:48] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.29, 22.87, 22.06
[22:39:48] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.34, 23.61, 22.64
[22:47:48] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.60, 23.72, 22.81
[22:49:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 30.14, 23.50, 20.67
[22:55:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.65, 22.78, 21.61
[23:02:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:03:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 13.10, 17.09, 19.58
[23:05:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.67, 21.20, 23.42
[23:07:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:09:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.59, 24.10, 24.05
[23:11:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.11, 4.47, 2.32
[23:11:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.72, 22.06, 23.32
[23:13:09] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.09, 3.75, 2.33
[23:15:06] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.75, 2.70, 2.11
[23:17:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:17:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.92, 22.76, 22.97
[23:26:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.73, 20.59, 19.68
[23:28:29] <botifico>	 [02python-functions] 07dependabot[bot] pushed 031 commit to 03dependabot/pip/dot-github/setuptools-71.1.0 [+0/-0/±1] 13https://github.com/miraheze/python-functions/commit/91eb10fe257d
[23:28:32] <botifico>	 [02python-functions] 07dependabot[bot] 0391eb10f - Bump setuptools from 71.0.1 to 71.1.0 in /.github
[23:28:34] <botifico>	 [02python-functions] 07dependabot[bot] created branch 03dependabot/pip/dot-github/setuptools-71.1.0 - 13https://github.com/miraheze/python-functions
[23:28:35] <botifico>	 [02python-functions] 07dependabot[bot] opened pull request 03#47: Bump setuptools from 71.0.1 to 71.1.0 in /.github - 13https://github.com/miraheze/python-functions/pull/47
[23:28:36] <botifico>	 [02python-functions] 07dependabot[bot] labeled pull request 03#47: Bump setuptools from 71.0.1 to 71.1.0 in /.github - 13https://github.com/miraheze/python-functions/pull/47
[23:28:39] <botifico>	 [02python-functions] 07dependabot[bot] labeled pull request 03#47: Bump setuptools from 71.0.1 to 71.1.0 in /.github - 13https://github.com/miraheze/python-functions/pull/47
[23:28:42] <botifico>	 [02python-functions] 07dependabot[bot] pushed 031 commit to 03dependabot/pip/dot-github/pytest-8.3.1 [+0/-0/±1] 13https://github.com/miraheze/python-functions/commit/f2fd0e32900d
[23:28:44] <botifico>	 [02python-functions] 07dependabot[bot] 03f2fd0e3 - Bump pytest from 8.2.2 to 8.3.1 in /.github
[23:28:47] <botifico>	 [02python-functions] 07dependabot[bot] created branch 03dependabot/pip/dot-github/pytest-8.3.1 - 13https://github.com/miraheze/python-functions
[23:28:49] <botifico>	 [02python-functions] 07dependabot[bot] opened pull request 03#48: Bump pytest from 8.2.2 to 8.3.1 in /.github - 13https://github.com/miraheze/python-functions/pull/48
[23:28:51] <botifico>	 [02python-functions] 07dependabot[bot] labeled pull request 03#48: Bump pytest from 8.2.2 to 8.3.1 in /.github - 13https://github.com/miraheze/python-functions/pull/48
[23:28:54] <botifico>	 [02python-functions] 07dependabot[bot] labeled pull request 03#48: Bump pytest from 8.2.2 to 8.3.1 in /.github - 13https://github.com/miraheze/python-functions/pull/48
[23:28:57] <botifico>	 [02python-functions] 07coderabbitai[bot] commented on pull request 03#47: Bump setuptools from 71.0.1 to 71.1.0 in /.github - 13https://github.com/miraheze/python-functions/pull/47#issuecomment-2243978389
[23:29:00] <botifico>	 [02python-functions] 07coderabbitai[bot] commented on pull request 03#48: Bump pytest from 8.2.2 to 8.3.1 in /.github - 13https://github.com/miraheze/python-functions/pull/48#issuecomment-2243978412
[23:30:28] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.29, 19.86, 19.57
[23:32:32] <miraheze-github>	 miraheze/python-functions - dependabot[bot] the build passed.
[23:33:39] <miraheze-github>	 miraheze/python-functions - dependabot[bot] the build passed.
[23:36:15] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.22, 20.84, 19.83
[23:42:01] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.18, 21.75, 20.91
[23:49:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.42, 19.10, 20.12
[23:53:48] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.52, 20.94, 23.76