[00:02:09] <icinga-miraheze>	 PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 13.30, 8.04, 3.49
[00:02:38] <icinga-miraheze>	 PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 16.56, 9.91, 4.20
[00:08:09] <icinga-miraheze>	 PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 1.33, 3.78, 3.11
[00:08:27] <icinga-miraheze>	 PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.46, 2.69, 1.77
[00:09:52] <MirahezeLSBot>	 !log [@mwtask141] starting deploy of {'config': True} to all
[00:09:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:10:03] <MirahezeLSBot>	 !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 10s
[00:10:09] <icinga-miraheze>	 RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 2.49, 3.24, 2.99
[00:10:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:10:27] <icinga-miraheze>	 RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.16, 2.80, 1.93
[00:14:06] <icinga-miraheze>	 PROBLEM - mw131 Puppet on mw131 is UNKNOWN: NRPE: Unable to read output
[00:18:45] <icinga-miraheze>	 PROBLEM - cp33 Current Load on cp33 is WARNING: LOAD WARNING - total load average: 2.88, 3.57, 2.30
[00:20:38] <icinga-miraheze>	 PROBLEM - cp22 Current Load on cp22 is WARNING: LOAD WARNING - total load average: 0.73, 2.65, 3.66
[00:20:46] <icinga-miraheze>	 RECOVERY - cp33 Current Load on cp33 is OK: LOAD OK - total load average: 1.71, 2.83, 2.18
[00:22:38] <icinga-miraheze>	 RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 0.21, 1.83, 3.23
[00:27:51] <icinga-miraheze>	 RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK
[00:31:03] <icinga-miraheze>	 PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query.
[00:40:05] <icinga-miraheze>	 RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[01:01:03] <icinga-miraheze>	 PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'}
[01:13:37] <icinga-miraheze>	 PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query.
[01:43:03] <icinga-miraheze>	 RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK
[02:29:05] <icinga-miraheze>	 PROBLEM - cp32 Current Load on cp32 is CRITICAL: LOAD CRITICAL - total load average: 3.89, 4.24, 2.43
[02:31:03] <icinga-miraheze>	 PROBLEM - cp32 Current Load on cp32 is WARNING: LOAD WARNING - total load average: 2.55, 3.45, 2.35
[02:33:02] <icinga-miraheze>	 RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 1.69, 2.85, 2.27
[03:01:48] <icinga-miraheze>	 RECOVERY - db121 Backups SQL on db121 is OK: FILE_AGE OK: /var/log/sql-backup.log is 106 seconds old and 0 bytes
[03:45:02] <icinga-miraheze>	 PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.83, 7.43, 6.33
[03:47:02] <icinga-miraheze>	 RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.27, 6.23, 6.03
[03:58:49] <botifico>	 [02miraheze/ErrorPages] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/actions/checkout-4 [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/commit/fe5139eb76d4
[03:58:51] <botifico>	 [02miraheze/ErrorPages] 07dependabot[bot] 03fe5139e - Bump actions/checkout from 3 to 4
[03:58:54] <botifico>	 [02ErrorPages] 07dependabot[bot] created branch 03dependabot/github_actions/actions/checkout-4 - 13https://github.com/miraheze/ErrorPages
[03:58:56] <botifico>	 [02ErrorPages] 07dependabot[bot] opened pull request 03#38: Bump actions/checkout from 3 to 4 - 13https://github.com/miraheze/ErrorPages/pull/38
[03:58:58] <botifico>	 [02ErrorPages] 07dependabot[bot] labeled pull request 03#38: Bump actions/checkout from 3 to 4 - 13https://github.com/miraheze/ErrorPages/pull/38
[03:59:46] <miraheze-github>	 miraheze/ErrorPages - dependabot[bot] the build passed.
[04:05:00] <icinga-miraheze>	 PROBLEM - cp32 Current Load on cp32 is CRITICAL: LOAD CRITICAL - total load average: 5.54, 6.51, 3.83
[04:08:58] <icinga-miraheze>	 RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 0.92, 3.34, 3.13
[04:51:39] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query.
[05:20:20] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK
[05:37:34] <icinga-miraheze>	 PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.4996365011 secs
[05:54:01] <icinga-miraheze>	 PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: CRITICAL - load average: 6.15, 3.74, 2.48
[05:56:00] <icinga-miraheze>	 RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 2.77, 3.31, 2.48
[06:01:03] <icinga-miraheze>	 PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query.
[06:11:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.99, 3.44, 3.09
[06:13:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.75, 3.96, 3.33
[06:18:42] <icinga-miraheze>	 PROBLEM - db101 Backups SQL on db101 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864135 seconds old and 49257 bytes
[06:29:11] <icinga-miraheze>	 PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[06:39:00] <botifico>	 [02miraheze/landing] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/actions/checkout-4 [+0/-0/±1] 13https://github.com/miraheze/landing/commit/d31f4dd30269
[06:39:01] <botifico>	 [02miraheze/landing] 07dependabot[bot] 03d31f4dd - Bump actions/checkout from 3 to 4
[06:39:04] <botifico>	 [02landing] 07dependabot[bot] created branch 03dependabot/github_actions/actions/checkout-4 - 13https://github.com/miraheze/landing
[06:39:06] <botifico>	 [02landing] 07dependabot[bot] opened pull request 03#119: Bump actions/checkout from 3 to 4 - 13https://github.com/miraheze/landing/pull/119
[06:39:07] <botifico>	 [02landing] 07dependabot[bot] labeled pull request 03#119: Bump actions/checkout from 3 to 4 - 13https://github.com/miraheze/landing/pull/119
[06:39:53] <miraheze-github>	 miraheze/landing - dependabot[bot] the build passed.
[06:53:22] <botifico>	 [02miraheze/puppet] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/actions/checkout-4 [+0/-0/±3] 13https://github.com/miraheze/puppet/commit/bd93964a4659
[06:53:23] <botifico>	 [02miraheze/puppet] 07dependabot[bot] 03bd93964 - build(deps): bump actions/checkout from 3 to 4
[06:53:26] <botifico>	 [02puppet] 07dependabot[bot] opened pull request 03#3399: build(deps): bump actions/checkout from 3 to 4 - 13https://github.com/miraheze/puppet/pull/3399
[06:53:29] <botifico>	 [02puppet] 07dependabot[bot] created branch 03dependabot/github_actions/actions/checkout-4 - 13https://github.com/miraheze/puppet
[06:53:30] <botifico>	 [02puppet] 07dependabot[bot] labeled pull request 03#3399: build(deps): bump actions/checkout from 3 to 4 - 13https://github.com/miraheze/puppet/pull/3399
[06:53:31] <botifico>	 [02puppet] 07dependabot[bot] labeled pull request 03#3399: build(deps): bump actions/checkout from 3 to 4 - 13https://github.com/miraheze/puppet/pull/3399
[06:57:51] <icinga-miraheze>	 RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK
[06:58:41] <botifico>	 [02miraheze/phabricator-extensions] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/actions/checkout-4 [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/commit/95f41ba866cc
[06:58:43] <botifico>	 [02miraheze/phabricator-extensions] 07dependabot[bot] 0395f41ba - Bump actions/checkout from 3 to 4
[06:58:45] <botifico>	 [02phabricator-extensions] 07dependabot[bot] created branch 03dependabot/github_actions/actions/checkout-4 - 13https://github.com/miraheze/phabricator-extensions
[06:58:47] <botifico>	 [02phabricator-extensions] 07dependabot[bot] opened pull request 03#38: Bump actions/checkout from 3 to 4 - 13https://github.com/miraheze/phabricator-extensions/pull/38
[06:58:50] <botifico>	 [02phabricator-extensions] 07dependabot[bot] labeled pull request 03#38: Bump actions/checkout from 3 to 4 - 13https://github.com/miraheze/phabricator-extensions/pull/38
[06:58:51] <botifico>	 [02phabricator-extensions] 07dependabot[bot] labeled pull request 03#38: Bump actions/checkout from 3 to 4 - 13https://github.com/miraheze/phabricator-extensions/pull/38
[07:03:19] <miraheze-github>	 miraheze/phabricator-extensions - dependabot[bot] the build has errored.
[07:27:01] <icinga-miraheze>	 RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK
[07:31:01] <icinga-miraheze>	 PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical]
[07:31:03] <icinga-miraheze>	 PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'}
[08:19:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.89, 3.75, 3.95
[08:31:04] <icinga-miraheze>	 RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.19, 2.76, 3.30
[09:10:10] <icinga-miraheze>	 PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[09:15:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.64, 3.69, 2.98
[09:17:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.99, 3.65, 3.05
[09:19:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.10, 3.78, 3.17
[09:39:35] <icinga-miraheze>	 RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK
[10:10:58] <icinga-miraheze>	 PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[10:20:58] <icinga-miraheze>	 PROBLEM - wiki.hsdfz.org.cn - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.hsdfz.org.cn' expires in 15 day(s) (Thu 21 Sep 2023 10:04:25 GMT +0000).
[10:23:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.47, 3.58, 3.96
[10:27:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.17, 3.73, 3.92
[10:37:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.28, 3.68, 3.96
[10:39:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.77, 3.91, 4.00
[10:40:23] <icinga-miraheze>	 PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query.
[10:41:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.73, 3.85, 3.97
[10:43:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.03, 4.03, 4.03
[10:47:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.53, 3.85, 3.96
[10:49:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.18, 3.94, 3.98
[10:51:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.83, 3.84, 3.93
[10:55:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.08, 3.92, 3.95
[10:55:24] <icinga-miraheze>	 PROBLEM - beamngmodding.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'beamngmodding.com' expires in 15 day(s) (Thu 21 Sep 2023 10:24:39 GMT +0000).
[10:57:59] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/38215c7d06ee...57ad8db07a67
[10:58:00] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot 0357ad8db - Bot: Update SSL cert for beamngmodding.com
[10:59:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.55, 3.90, 3.95
[11:01:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.01, 3.85, 3.92
[11:09:48] <icinga-miraheze>	 RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK
[11:17:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.45, 3.83, 3.99
[11:24:48] <icinga-miraheze>	 RECOVERY - beamngmodding.com - LetsEncrypt on sslhost is OK: OK - Certificate 'beamngmodding.com' will expire on Mon 04 Dec 2023 09:57:52 GMT +0000.
[11:27:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.18, 3.49, 3.69
[11:29:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.55, 3.51, 3.67
[11:39:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.40, 3.67, 3.66
[11:41:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.37, 3.52, 3.60
[11:53:04] <icinga-miraheze>	 RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.83, 3.21, 3.38
[12:07:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.62, 4.16, 3.66
[12:29:10] <icinga-miraheze>	 PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[12:57:51] <icinga-miraheze>	 RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK
[13:38:50] <icinga-miraheze>	 PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query.
[14:01:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.29, 3.77, 4.00
[14:08:15] <icinga-miraheze>	 PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[14:09:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.59, 3.89, 3.93
[14:11:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.53, 3.70, 3.86
[14:13:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.17, 3.82, 3.88
[14:15:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.78, 3.83, 3.88
[14:36:38] <icinga-miraheze>	 PROBLEM - cp32 Current Load on cp32 is CRITICAL: LOAD CRITICAL - total load average: 5.19, 4.50, 2.42
[14:37:40] <icinga-miraheze>	 RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK
[14:38:37] <icinga-miraheze>	 RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 1.13, 3.16, 2.18
[14:39:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.24, 4.07, 3.79
[14:39:55] <icinga-miraheze>	 PROBLEM - cp33 Current Load on cp33 is CRITICAL: LOAD CRITICAL - total load average: 2.56, 4.07, 2.55
[14:41:55] <icinga-miraheze>	 RECOVERY - cp33 Current Load on cp33 is OK: LOAD OK - total load average: 1.34, 3.05, 2.35
[14:45:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.77, 3.92, 3.83
[14:51:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.02, 3.68, 3.72
[14:53:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.92, 3.74, 3.73
[14:55:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.51, 4.01, 3.83
[15:01:26] <icinga-miraheze>	 PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query qadrishattari.xyz. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[15:25:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.23, 3.60, 3.92
[15:27:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.29, 3.75, 3.93
[15:29:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.37, 3.73, 3.91
[15:30:38] <icinga-miraheze>	 RECOVERY - wiki.qadrishattari.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.qadrishattari.xyz reverse DNS resolves to cp23.miraheze.org - CNAME OK
[15:31:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.01, 3.80, 3.91
[15:33:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 4.00, 3.77, 3.88
[15:35:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.20, 4.29, 4.06
[16:11:07] <icinga-miraheze>	 PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.21, 2.73, 1.33
[16:13:07] <icinga-miraheze>	 RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.27, 2.46, 1.40
[16:25:09] <icinga-miraheze>	 PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.54, 3.24, 2.75
[16:29:03] <icinga-miraheze>	 RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.74, 3.01, 2.77
[16:44:09] <icinga-miraheze>	 PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 1.91, 3.54, 2.31
[16:46:09] <icinga-miraheze>	 RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 1.07, 2.66, 2.14
[16:59:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.29, 3.64, 3.97
[17:07:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.06, 3.86, 3.93
[17:25:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.40, 3.82, 4.00
[17:53:04] <icinga-miraheze>	 RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.12, 3.21, 3.40
[18:09:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 6.98, 5.12, 4.07
[18:22:13] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query.
[18:57:02] <icinga-miraheze>	 PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.76, 7.14, 5.79
[18:59:02] <icinga-miraheze>	 PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.94, 7.47, 6.03
[19:01:02] <icinga-miraheze>	 RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.39, 6.57, 5.88
[19:31:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.46, 3.60, 3.96
[19:35:43] <icinga-miraheze>	 PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query.
[19:39:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.83, 3.90, 3.91
[19:41:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.95, 3.87, 3.90
[19:47:34] <icinga-miraheze>	 PROBLEM - cp32 NTP time on cp32 is CRITICAL: NTP CRITICAL: Offset 0.5399252474 secs
[19:49:40] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[19:51:04] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.07, 3.80, 3.83
[20:05:09] <icinga-miraheze>	 PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[20:36:23] <icinga-miraheze>	 PROBLEM - cp32 Current Load on cp32 is WARNING: LOAD WARNING - total load average: 3.86, 3.83, 2.34
[20:38:23] <icinga-miraheze>	 RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 1.57, 2.94, 2.19
[20:49:39] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK
[20:58:23] <icinga-miraheze>	 PROBLEM - cp32 Current Load on cp32 is WARNING: LOAD WARNING - total load average: 1.59, 3.77, 2.67
[20:58:41] <botifico>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/cccf2799b530
[20:58:44] <botifico>	 [02miraheze/puppet] 07paladox 03cccf279 - varnish: remove converting thumbs to contain -thumbnail
[20:58:55] <botifico>	 [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet
[20:58:56] <botifico>	 [02puppet] 07paladox opened pull request 03#3400: varnish: remove converting thumbs to contain -thumbnail - 13https://github.com/miraheze/puppet/pull/3400
[20:58:59] <botifico>	 [02puppet] 07paladox edited pull request 03#3400: varnish: remove converting thumbs to contain -thumbnail - 13https://github.com/miraheze/puppet/pull/3400
[20:59:30] <botifico>	 [02puppet] 07paladox closed pull request 03#3400: varnish: remove converting thumbs to contain -thumbnail - 13https://github.com/miraheze/puppet/pull/3400
[20:59:32] <botifico>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a817ebd5dfce...4307037b4398
[20:59:33] <botifico>	 [02miraheze/puppet] 07paladox 034307037 - varnish: remove converting thumbs to contain -thumbnail (#3400)
[20:59:35] <botifico>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4
[20:59:38] <botifico>	 [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet
[21:00:23] <icinga-miraheze>	 RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 0.81, 2.71, 2.40
[21:02:01] <icinga-miraheze>	 PROBLEM - cp33 Current Load on cp33 is WARNING: LOAD WARNING - total load average: 2.93, 3.70, 2.37
[21:02:56] <icinga-miraheze>	 PROBLEM - wiki.widwwa.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.widwwa.co.uk' expires in 15 day(s) (Thu 21 Sep 2023 20:43:25 GMT +0000).
[21:03:42] <icinga-miraheze>	 PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 12.81, 10.03, 7.18
[21:03:49] <icinga-miraheze>	 PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.81, 6.08, 5.00
[21:03:57] <icinga-miraheze>	 PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: CRITICAL - NGINX Error Rate is 64%
[21:03:59] <icinga-miraheze>	 RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK
[21:04:01] <icinga-miraheze>	 PROBLEM - graylog131 Current Load on graylog131 is WARNING: WARNING - load average: 3.93, 3.04, 2.35
[21:04:01] <icinga-miraheze>	 RECOVERY - cp33 Current Load on cp33 is OK: LOAD OK - total load average: 1.25, 2.77, 2.18
[21:04:25] <icinga-miraheze>	 PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 22.92, 13.44, 8.28
[21:04:38] <icinga-miraheze>	 PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 20.69, 11.99, 7.38
[21:04:42] <icinga-miraheze>	 PROBLEM - swiftproxy131 Current Load on swiftproxy131 is CRITICAL: CRITICAL - load average: 21.06, 11.76, 5.61
[21:04:46] <icinga-miraheze>	 PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 61%
[21:05:21] <icinga-miraheze>	 PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 16.65, 11.04, 6.93
[21:05:35] <icinga-miraheze>	 PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 16.04, 9.90, 6.07
[21:05:57] <icinga-miraheze>	 PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 44%
[21:06:46] <icinga-miraheze>	 PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is WARNING: WARNING - NGINX Error Rate is 59%
[21:07:35] <icinga-miraheze>	 PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 10.79, 9.86, 6.52
[21:07:40] <botifico>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4307037b4398...f555b87db078
[21:07:42] <botifico>	 [02miraheze/puppet] 07paladox 03f555b87 - Update default.vcl
[21:07:49] <icinga-miraheze>	 RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.04, 6.68, 5.57
[21:08:01] <icinga-miraheze>	 PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: CRITICAL - load average: 4.28, 3.65, 2.76
[21:08:46] <icinga-miraheze>	 RECOVERY - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is OK: OK - NGINX Error Rate is 28%
[21:09:35] <icinga-miraheze>	 RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 6.68, 9.22, 6.74
[21:09:48] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/57ad8db07a67...f6f5aebb004b
[21:09:50] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot 03f6f5aeb - Bot: Update SSL cert for wiki.widwwa.co.uk
[21:09:56] <icinga-miraheze>	 RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 10%
[21:10:00] <icinga-miraheze>	 PROBLEM - graylog131 Current Load on graylog131 is WARNING: WARNING - load average: 2.95, 3.48, 2.81
[21:10:38] <icinga-miraheze>	 PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 5.68, 11.53, 9.14
[21:11:20] <icinga-miraheze>	 PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 7.01, 10.64, 8.45
[21:11:42] <icinga-miraheze>	 PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 6.86, 11.89, 10.01
[21:12:01] <icinga-miraheze>	 RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 2.12, 2.96, 2.70
[21:12:38] <icinga-miraheze>	 RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 6.50, 9.85, 8.82
[21:12:42] <icinga-miraheze>	 PROBLEM - swiftproxy131 Current Load on swiftproxy131 is WARNING: WARNING - load average: 2.75, 7.97, 6.89
[21:13:20] <icinga-miraheze>	 RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 4.61, 8.52, 7.94
[21:13:42] <icinga-miraheze>	 RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 5.66, 10.00, 9.56
[21:14:25] <icinga-miraheze>	 PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 6.82, 10.59, 10.07
[21:14:42] <icinga-miraheze>	 RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 1.39, 5.84, 6.25
[21:16:25] <icinga-miraheze>	 RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 5.40, 8.83, 9.48
[21:17:01] <icinga-miraheze>	 RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK
[21:21:01] <icinga-miraheze>	 PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical]
[21:28:23] <icinga-miraheze>	 PROBLEM - cp32 Current Load on cp32 is CRITICAL: LOAD CRITICAL - total load average: 4.82, 5.97, 3.43
[21:32:23] <icinga-miraheze>	 RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 0.91, 3.10, 2.82
[21:41:44] <botifico>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_40 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/0bfe72842740...653515218ae7
[21:41:46] <botifico>	 [02miraheze/mediawiki] 07paladox 036535152 - Update Math
[21:42:44] <MirahezeLSBot>	 !log [paladox@test131] starting deploy of {'world': True, 'l10n': True, 'force': True} to all
[21:42:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:45:15] <botifico>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/dd64574677e8...bbe83c573c6e
[21:45:18] <botifico>	 [02miraheze/mediawiki] 07paladox 03bbe83c5 - Update Math
[21:45:51] <MirahezeLSBot>	 !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True} to all
[21:45:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:50:53] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/f6f5aebb004b...3007f55857d9
[21:50:54] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot 033007f55 - Bot: Add SSL cert for wiki.fediverse.kr
[21:52:04] <botifico>	 [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3007f55857d9...79a1a0e0157f
[21:52:05] <botifico>	 [02miraheze/ssl] 07MacFan4000 0379a1a0e - Update certs.yaml
[21:53:52] <botifico>	 [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/79a1a0e0157f...edafa7348ad9
[21:53:54] <botifico>	 [02miraheze/ssl] 07MacFan4000 03edafa73 - add wiki.fediverse.kr redirect
[21:54:53] <miraheze-github>	 miraheze/ssl - MacFan4000 the build has errored.
[21:55:49] <MirahezeLSBot>	 !log [paladox@test131] finished deploy of {'world': True, 'l10n': True, 'force': True} to all - SUCCESS in 784s
[21:55:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:55:54] <botifico>	 [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/edafa7348ad9...d89871026ca8
[21:55:56] <botifico>	 [02miraheze/ssl] 07MacFan4000 03d898710 - Update redirects.yaml
[21:56:05] <MirahezeLSBot>	 !log [paladox@mwtask141] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 613s
[21:56:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:01:49] <icinga-miraheze>	 RECOVERY - wiki.widwwa.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.widwwa.co.uk' will expire on Mon 04 Dec 2023 20:09:40 GMT +0000.
[22:21:40] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query.
[22:46:03] <icinga-miraheze>	 PROBLEM - cp33 Current Load on cp33 is CRITICAL: LOAD CRITICAL - total load average: 2.61, 5.39, 3.41
[22:50:03] <icinga-miraheze>	 RECOVERY - cp33 Current Load on cp33 is OK: LOAD OK - total load average: 0.66, 2.88, 2.84
[22:50:19] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[22:57:11] <icinga-miraheze>	 PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.4998576343 secs
[23:03:09] <icinga-miraheze>	 PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 14.69, 7.41, 3.34
[23:11:09] <icinga-miraheze>	 RECOVERY - db112 Current Load on db112 is OK: OK - load average: 0.75, 5.70, 4.83
[23:20:03] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK
[23:33:18] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.18, 3.46, 3.92
[23:33:47] <botifico>	 [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d89871026ca8...e9aa44de9c03
[23:33:50] <botifico>	 [02miraheze/ssl] 07MacFan4000 03e9aa44d - Update certs.yaml
[23:34:24] <botifico>	 [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/e9aa44de9c03...c1eed5847597
[23:34:27] <botifico>	 [02miraheze/ssl] 07MacFan4000 03c1eed58 - Delete certificates/mediawiki.ecfin.eu.crt
[23:39:19] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.66, 3.83, 3.90
[23:41:18] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.87, 3.91, 3.93
[23:49:19] <botifico>	 [02dns] 07MacFan4000 opened pull request 03#435: Add two zones + www.widedata.org - 13https://github.com/miraheze/dns/pull/435
[23:49:40] <MacFan4000>	 paladox: can you please merge ^?
[23:53:02] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.06, 3.75, 3.81
[23:53:43] <botifico>	 [02dns] 07paladox closed pull request 03#435: Add two zones + www.widedata.org - 13https://github.com/miraheze/dns/pull/435
[23:53:45] <botifico>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+2/-0/±1] 13https://github.com/miraheze/dns/compare/1373aa0138b8...dd0aae058259
[23:53:47] <botifico>	 [02miraheze/dns] 07MacFan4000 03dd0aae0 - Add two zones + www.widedata.org (#435)
[23:54:59] <icinga-miraheze>	 PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.61, 3.69, 3.78
[23:59:17] <icinga-miraheze>	 PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL