[00:02:14] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 11.90, 8.72, 3.81 [00:03:14] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 6.80, 7.89, 3.87 [00:08:14] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 0.58, 3.06, 2.78 [00:09:14] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 1.25, 3.08, 2.95 [00:09:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.78, 3.86, 3.73 [00:13:14] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 4.94, 4.08, 3.39 [00:13:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.40, 3.77, 3.75 [00:21:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.30, 3.91, 3.77 [00:23:14] PROBLEM - cp22 Current Load on cp22 is WARNING: LOAD WARNING - total load average: 0.85, 3.40, 3.98 [00:23:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.12, 3.56, 3.65 [00:27:14] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 0.37, 1.89, 3.24 [00:29:19] PROBLEM - cp33 NTP time on cp33 is CRITICAL: NTP CRITICAL: Offset 0.871927917 secs [00:31:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.70, 3.82, 3.68 [00:33:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.22, 3.64, 3.63 [00:35:39] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.08, 3.85, 3.71 [00:37:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.20, 3.60, 3.63 [00:39:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.00, 3.99, 3.76 [00:40:19] PROBLEM - arcanewaterswiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for arcanewaterswiki.org could not be found [00:53:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.58, 3.55, 3.98 [01:09:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.36, 3.95, 3.72 [01:11:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.66, 3.91, 3.74 [01:17:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.37, 3.95, 3.79 [01:21:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.06, 3.79, 3.79 [01:33:35] PROBLEM - arcanewaterswiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address arcanewaterswiki.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:35:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.50, 2.90, 3.34 [01:39:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.79, 4.21, 3.75 [01:45:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.95, 3.98, 3.80 [01:57:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.04, 3.82, 3.75 [01:59:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.61, 3.77, 3.74 [02:09:39] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.39, 4.26, 3.92 [02:17:39] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.66, 3.93, 3.93 [02:23:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.13, 3.88, 3.91 [02:25:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.39, 3.74, 3.86 [02:29:39] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.11, 3.85, 3.88 [02:33:39] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.80, 3.87, 3.88 [02:39:39] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 7.31, 4.75, 4.15 [02:57:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.45, 3.74, 3.95 [02:59:39] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.33, 3.91, 3.98 [03:01:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.63, 3.93, 3.99 [03:01:51] RECOVERY - db142 Backups SQL on db142 is OK: FILE_AGE OK: /var/log/sql-backup.log is 110 seconds old and 0 bytes [03:04:08] PROBLEM - mon141 Backups Grafana on mon141 is WARNING: FILE_AGE WARNING: /var/log/grafana-backup.log is 864228 seconds old and 93 bytes [03:05:40] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.88, 4.13, 4.04 [03:37:39] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.79, 3.41, 3.96 [03:39:39] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.33, 3.90, 4.06 [03:47:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.69, 3.34, 3.84 [04:09:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.62, 3.76, 3.63 [04:18:52] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [04:47:41] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [05:20:33] PROBLEM - wiki.consid.vn - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query consid.vn. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:35:04] PROBLEM - tudien.como.vn - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - tudien.como.vn All nameservers failed to answer the query. [05:43:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.26, 3.53, 3.93 [05:50:19] RECOVERY - wiki.consid.vn - reverse DNS on sslhost is OK: SSL OK - wiki.consid.vn reverse DNS resolves to cp22.miraheze.org - CNAME OK [05:53:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.63, 2.66, 3.39 [06:05:04] PROBLEM - tudien.como.vn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tudien.como.vn could not be found [06:09:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.84, 3.31, 3.05 [06:11:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.97, 3.12, 3.01 [06:18:42] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: CRITICAL - load average: 4.45, 3.22, 2.46 [06:19:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.76, 3.74, 3.36 [06:20:42] RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 1.62, 2.57, 2.32 [06:28:14] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 7.19, 8.55, 4.04 [06:31:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.31, 3.99, 3.64 [06:34:14] PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 2.27, 3.69, 3.17 [06:36:14] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 1.13, 2.76, 2.89 [07:11:11] PROBLEM - cloud10 Puppet on cloud10 is UNKNOWN: NRPE: Unable to read output [07:13:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.20, 3.16, 3.92 [07:19:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.41, 2.51, 3.36 [07:23:31] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [07:27:31] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:39:11] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:52:33] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [07:56:02] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [08:01:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.72, 3.18, 2.85 [08:03:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.71, 3.09, 2.87 [10:23:31] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:52:32] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [11:12:59] [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/e49ba62520e9...d81610d6616a [11:13:01] [02miraheze/ImportDump] 07translatewiki 03d81610d - Localisation updates from https://translatewiki.net. [11:13:03] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/bcb70af274ca...e0e0984aec90 [11:13:06] [02miraheze/MirahezeMagic] 07translatewiki 03e0e0984 - Localisation updates from https://translatewiki.net. [11:19:41] miraheze/MirahezeMagic - translatewiki the build has errored. [11:46:25] PROBLEM - wiki.danvs.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.danvs.net All nameservers failed to answer the query. [12:05:05] PROBLEM - tudien.como.vn - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - tudien.como.vn All nameservers failed to answer the query. [12:25:16] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.28, 3.56, 3.10 [12:36:53] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.96, 3.21, 3.22 [12:40:48] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.47, 3.89, 3.50 [12:42:44] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.89, 3.88, 3.55 [12:48:33] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.15, 3.31, 3.39 [12:52:28] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.07, 3.59, 3.47 [12:54:24] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.53, 3.46, 3.43 [12:58:16] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.97, 3.26, 3.36 [13:35:05] PROBLEM - tudien.como.vn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tudien.como.vn could not be found [14:14:26] RECOVERY - wiki.danvs.net - reverse DNS on sslhost is OK: SSL OK - wiki.danvs.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [14:16:29] PROBLEM - miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.wiki' expires in 15 day(s) (Wed 06 Sep 2023 13:51:26 GMT +0000). [17:02:44] [02miraheze/ManageWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.16.1 [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/commit/5a566bedea03 [17:02:45] [02miraheze/ManageWiki] 07dependabot[bot] 035a566be - Bump stylelint-config-wikimedia from 0.15.0 to 0.16.1 [17:02:47] [02ManageWiki] 07dependabot[bot] opened pull request 03#421: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.1 - 13https://github.com/miraheze/ManageWiki/pull/421 [17:02:49] [02ManageWiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.16.1 - 13https://github.com/miraheze/ManageWiki [17:02:50] [02ManageWiki] 07dependabot[bot] labeled pull request 03#421: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.1 - 13https://github.com/miraheze/ManageWiki/pull/421 [17:02:52] [02ManageWiki] 07dependabot[bot] labeled pull request 03#421: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.1 - 13https://github.com/miraheze/ManageWiki/pull/421 [17:02:55] [02ManageWiki] 07dependabot[bot] commented on pull request 03#412: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.0 - 13https://github.com/miraheze/ManageWiki/pull/412#issuecomment-1686708895 [17:02:58] [02ManageWiki] 07dependabot[bot] closed pull request 03#412: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.0 - 13https://github.com/miraheze/ManageWiki/pull/412 [17:02:59] [02miraheze/ManageWiki] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.16.0 [17:03:00] [02ManageWiki] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.16.0 - 13https://github.com/miraheze/ManageWiki [17:07:38] miraheze/ManageWiki - dependabot[bot] the build has errored. [18:09:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.21, 3.28, 2.53 [18:21:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.73, 3.69, 3.34 [18:23:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.18, 3.96, 3.49 [18:33:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.09, 3.90, 3.77 [18:35:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.05, 3.95, 3.80 [18:45:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.48, 3.88, 3.94 [18:47:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.20, 4.35, 4.10 [18:57:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.83, 3.69, 3.93 [19:03:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.43, 2.56, 3.31 [19:03:40] PROBLEM - prometheus131 Current Load on prometheus131 is CRITICAL: CRITICAL - load average: 7.73, 3.83, 1.78 [19:11:31] PROBLEM - prometheus131 Current Load on prometheus131 is WARNING: WARNING - load average: 3.86, 3.98, 2.74 [19:13:28] PROBLEM - prometheus131 Current Load on prometheus131 is CRITICAL: CRITICAL - load average: 4.51, 4.30, 3.01 [19:22:18] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.68, 3.24, 3.15 [19:23:21] RECOVERY - prometheus131 Current Load on prometheus131 is OK: OK - load average: 1.26, 3.35, 3.33 [19:37:48] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.99, 3.39, 3.38 [19:41:43] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.62, 3.69, 3.52 [20:09:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.81, 2.82, 3.26 [20:31:45] PROBLEM - cp32 NTP time on cp32 is CRITICAL: NTP CRITICAL: Offset 0.8394051492 secs [21:17:50] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [21:21:45] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [21:27:31] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:56:02] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [22:02:35] [02miraheze/CreateWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.16.1 [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/commit/fb9b172f050f [22:02:38] [02miraheze/CreateWiki] 07dependabot[bot] 03fb9b172 - Bump stylelint-config-wikimedia from 0.15.0 to 0.16.1 [22:02:40] [02CreateWiki] 07dependabot[bot] opened pull request 03#436: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.1 - 13https://github.com/miraheze/CreateWiki/pull/436 [22:02:41] [02CreateWiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.16.1 - 13https://github.com/miraheze/CreateWiki [22:02:44] [02CreateWiki] 07dependabot[bot] labeled pull request 03#436: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.1 - 13https://github.com/miraheze/CreateWiki/pull/436 [22:02:46] [02CreateWiki] 07dependabot[bot] labeled pull request 03#436: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.1 - 13https://github.com/miraheze/CreateWiki/pull/436 [22:02:49] [02CreateWiki] 07dependabot[bot] commented on pull request 03#425: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.0 - 13https://github.com/miraheze/CreateWiki/pull/425#issuecomment-1687104510 [22:02:51] [02CreateWiki] 07dependabot[bot] closed pull request 03#425: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.0 - 13https://github.com/miraheze/CreateWiki/pull/425 [22:02:54] [02miraheze/CreateWiki] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.16.0 [22:02:55] [02CreateWiki] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.16.0 - 13https://github.com/miraheze/CreateWiki [22:08:05] miraheze/CreateWiki - dependabot[bot] the build has errored. [23:04:30] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 14.12, 7.81, 3.62 [23:08:30] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.19, 6.60, 4.27