[00:00:50] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 2.74, 4.46, 3.11 [00:01:22] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:02:17] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.72, 4.73, 3.72 [00:02:32] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.67, 3.30, 3.16 [00:03:31] RECOVERY - isv.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Tue 05 Sep 2023 22:02:13 GMT +0000. [00:04:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.75, 4.65, 3.65 [00:04:46] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.16, 3.06, 2.89 [00:08:16] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.40, 3.54, 3.62 [00:10:15] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.20, 3.90, 3.73 [00:12:15] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.78, 3.40, 3.56 [00:12:19] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 6.98, 3.90, 2.05 [00:12:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.66, 3.68, 3.67 [00:16:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.11, 2.82, 3.31 [00:18:15] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.11, 2.68, 3.21 [00:18:21] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.16, 3.90, 2.86 [00:20:20] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.44, 2.70, 2.54 [00:21:12] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:22:14] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.02, 3.92, 3.63 [00:23:10] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [00:24:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.76, 3.93, 3.67 [00:28:37] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.52, 6.82, 4.76 [00:30:12] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.86, 2.61, 3.17 [00:32:37] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.39, 8.44, 5.83 [00:34:31] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.75, 3.80, 2.87 [00:34:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.40, 3.79, 3.33 [00:36:11] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.66, 4.03, 3.62 [00:36:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.01, 3.58, 3.28 [00:36:32] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.58, 3.25, 2.78 [00:36:34] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.33, 7.68, 6.23 [00:38:11] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.38, 3.62, 3.51 [00:38:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.87, 3.01, 3.12 [00:40:27] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.03, 3.74, 3.09 [00:42:10] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.16, 3.04, 3.32 [00:42:30] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.17, 3.51, 3.09 [00:42:31] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.80, 6.65, 6.26 [00:44:29] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.81, 3.25, 3.05 [00:46:18] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 12.44, 5.14, 2.67 [00:47:50] PROBLEM - cp22 APT on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:52:23] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 13.56, 9.01, 7.00 [00:52:27] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.03, 3.68, 3.26 [00:53:06] RECOVERY - cp22 APT on cp22 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [00:54:16] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.90, 3.84, 3.15 [00:56:06] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.45, 3.13, 2.91 [00:56:16] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.80, 2.75, 2.83 [00:56:25] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.06, 3.56, 3.32 [00:58:06] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.20, 2.40, 2.67 [00:58:25] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.23, 3.20, 3.23 [01:00:06] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [01:00:19] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.31, 7.90, 7.50 [01:06:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.05, 4.28, 6.11 [01:29:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.57, 6.74, 7.86 [01:30:36] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.44, 3.43, 3.19 [01:32:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.94, 2.94, 3.05 [01:33:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.40, 7.27, 7.79 [01:36:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.69, 5.04, 3.77 [01:38:34] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1063159704 secs [01:42:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.53, 3.61, 3.56 [01:43:50] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.38, 6.76, 4.25 [01:44:34] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08442252874 secs [01:46:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.21, 3.03, 3.33 [01:47:48] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.78, 7.54, 5.28 [01:48:12] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 22.66, 11.45, 5.79 [01:49:47] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 0.99, 5.16, 4.67 [01:56:32] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.95, 3.87, 3.58 [01:57:22] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:58:32] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.55, 4.03, 3.70 [01:58:44] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.61, 2.41, 1.55 [01:59:40] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [02:00:12] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.51, 7.54, 6.90 [02:00:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.41, 3.93, 3.71 [02:00:42] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.46, 3.07, 1.89 [02:01:42] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [02:02:12] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 1.58, 5.56, 6.26 [02:02:40] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.21, 2.33, 1.76 [02:03:49] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.81, 3.93, 3.36 [02:04:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.15, 3.70, 3.63 [02:06:32] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.41, 3.30, 3.49 [02:08:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.72, 2.83, 3.30 [02:09:09] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [02:09:48] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.04, 3.28, 3.30 [02:11:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.09, 7.39, 7.95 [02:15:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.01, 7.37, 7.79 [02:16:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.23, 4.54, 3.75 [02:17:48] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 10.29, 5.13, 3.85 [02:19:23] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [02:20:32] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.00, 3.38, 3.46 [02:21:22] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [02:22:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.99, 2.92, 3.28 [02:29:48] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.85, 3.45, 3.95 [02:31:04] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [02:35:48] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.81, 2.36, 3.29 [02:51:41] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.22, 5.34, 2.96 [02:53:41] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.18, 4.80, 3.08 [02:55:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.17, 7.09, 7.92 [02:57:14] [02mw-config] 07Universal-Omega commented on pull request 03#5248: rm Agent Isai and Universal Omega from MirahezeStaffAccessIDs per resignations - 13https://github.com/miraheze/mw-config/pull/5248#issuecomment-1581814560 [02:59:23] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.48, 7.79, 8.01 [02:59:25] [02mw-config] 07AgentIsai closed pull request 03#5248: rm Agent Isai and Universal Omega from MirahezeStaffAccessIDs per resignations - 13https://github.com/miraheze/mw-config/pull/5248 [02:59:28] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4708e1d9765c...2dcaf9724d32 [02:59:29] [02miraheze/mw-config] 07brandon-wm 032dcaf97 - rm Agent Isai and Universal Omega from MirahezeStaffAccessIDs per resignations (#5248) [02:59:56] PROBLEM - swiftproxy131 HTTP on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [03:00:33] miraheze/mw-config - AgentIsai the build passed. [03:00:54] RECOVERY - db112 Backups SQL on db112 is OK: FILE_AGE OK: /var/log/sql-backup.log is 52 seconds old and 365 bytes [03:01:41] RECOVERY - db112 Backups SQL on db112 is OK: FILE_AGE OK: /var/log/sql-backup.log is 98 seconds old and 365 bytes [03:01:53] RECOVERY - swiftproxy131 HTTP on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.023 second response time [03:03:42] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.80, 4.39, 3.41 [03:05:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.26, 3.64, 3.02 [03:05:42] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.96, 3.41, 3.17 [03:07:41] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.48, 3.07, 3.06 [03:08:24] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [03:09:09] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.65, 3.82, 3.30 [03:10:38] !log [@mwtask141] starting deploy of {'config': True} to all [03:10:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:10:57] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 18s [03:11:05] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.04, 4.19, 3.48 [03:11:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:16:15] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.2693148851 secs [03:19:32] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [03:21:31] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [03:23:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.78, 3.56, 3.12 [03:25:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.18, 3.03, 2.97 [03:26:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.03, 3.16, 3.96 [03:26:34] !log [@test131] starting deploy of {'config': True} to all [03:26:35] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [03:26:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:26:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:32:16] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.14, 2.89, 1.77 [03:33:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.21, 7.18, 7.94 [03:36:16] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.21, 3.69, 2.39 [03:36:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.77, 2.52, 3.26 [03:38:16] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.02, 2.82, 2.23 [03:39:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.08, 7.37, 7.68 [03:43:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.80, 7.85, 7.82 [03:45:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.27, 8.23, 7.98 [03:47:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.46, 8.00, 7.93 [03:49:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.72, 8.34, 8.07 [03:53:30] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.20, 4.09, 3.14 [03:55:23] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:57:29] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.71, 3.56, 3.16 [03:59:29] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.42, 4.19, 3.45 [04:00:26] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [04:02:38] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.84, 4.22, 3.37 [04:03:28] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.12, 3.25, 3.25 [04:04:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.52, 3.62, 3.25 [04:08:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.26, 2.64, 2.97 [04:12:15] PROBLEM - scum-wiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for scum-wiki.com could not be found [04:13:59] [02CreateWiki] 07joritochip closed pull request 03#392: DRAFT: Add tag categorization (T8791) - 13https://github.com/miraheze/CreateWiki/pull/392 [04:14:15] [02ManageWiki] 07joritochip closed pull request 03#390: DRAFT: Tags on ManageWiki/core (T8791) - 13https://github.com/miraheze/ManageWiki/pull/390 [04:16:00] RECOVERY - scum-wiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'scum-wiki.com' will expire on Mon 24 Jul 2023 20:30:24 GMT +0000. [04:17:25] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.22, 3.99, 3.46 [04:19:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 4.78, 6.40, 7.82 [04:19:25] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.45, 3.50, 3.35 [04:21:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.18, 7.45, 8.05 [04:21:24] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.19, 3.23, 3.28 [04:25:50] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.32, 6.21, 3.20 [04:26:16] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.004132151604 secs [04:30:16] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.73, 2.64, 1.59 [04:31:46] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.31, 5.68, 4.14 [04:31:52] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:32:16] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.52, 4.70, 2.49 [04:34:18] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [04:36:15] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.2045190632 secs [04:36:16] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.21, 3.27, 2.45 [04:52:36] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:53:30] [02MirahezeMagic] 07Universal-Omega closed pull request 03#407: Create MirahezeSurveyResults special page - 13https://github.com/miraheze/MirahezeMagic/pull/407 [04:53:40] [02MirahezeMagic] 07Universal-Omega closed pull request 03#398: Try using qmegas/memcache-search for clearing cache - 13https://github.com/miraheze/MirahezeMagic/pull/398 [04:53:48] [02puppet] 07Universal-Omega closed pull request 03#3136: mediawiki: add MirahezeMagic composer - 13https://github.com/miraheze/puppet/pull/3136 [04:54:00] [02puppet] 07Universal-Omega closed pull request 03#2942: mediawiki: rewrite ^/page to ^/wiki/page - 13https://github.com/miraheze/puppet/pull/2942 [04:54:22] [02ImportDump] 07Universal-Omega closed pull request 03#36: Add an onSpecialRequestImportDumpModifyFormFields hook - 13https://github.com/miraheze/ImportDump/pull/36 [04:54:34] [02WikiDiscover] 07Universal-Omega closed pull request 03#88: API: try to extend ApiQueryGeneratorBase and add PHPUnit test - 13https://github.com/miraheze/WikiDiscover/pull/88 [04:54:40] [02WikiDiscover] 07Universal-Omega closed pull request 03#96: Move adding WikiDiscoverDescription from ManageWiki to WikiDiscover - 13https://github.com/miraheze/WikiDiscover/pull/96 [04:54:59] [02CreateWiki] 07Universal-Omega closed pull request 03#390: Pass $cwdb to CreateWikiCreation hook - 13https://github.com/miraheze/CreateWiki/pull/390 [04:55:05] [02CreateWiki] 07Universal-Omega closed pull request 03#325: Use tempnam for database lists - 13https://github.com/miraheze/CreateWiki/pull/325 [04:55:12] [02CreateWiki] 07Universal-Omega closed pull request 03#328: CreateWikiJson: only decode when actually used - 13https://github.com/miraheze/CreateWiki/pull/328 [04:55:20] [02CreateWiki] 07Universal-Omega closed pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [04:55:29] [02CreateWiki] 07Universal-Omega closed pull request 03#354: Support partial blocks for requesting wikis - 13https://github.com/miraheze/CreateWiki/pull/354 [04:55:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.53, 10.07, 11.78 [04:55:36] [02CreateWiki] 07Universal-Omega closed pull request 03#361: T9874: add support for warning bureaucrats who haven't edited their new wiki - 13https://github.com/miraheze/CreateWiki/pull/361 [04:55:51] [02CreateWiki] 07Universal-Omega closed pull request 03#411: Remove the need for CreateWikiLogFormatter - 13https://github.com/miraheze/CreateWiki/pull/411 [04:56:10] [02ManageWiki] 07Universal-Omega closed pull request 03#408: Add some hooks to ManageWiki to allow further control - 13https://github.com/miraheze/ManageWiki/pull/408 [04:56:17] [02ManageWiki] 07Universal-Omega closed pull request 03#407: Remove some dependency on CreateWiki - 13https://github.com/miraheze/ManageWiki/pull/407 [04:56:27] [02ManageWiki] 07Universal-Omega closed pull request 03#377: Remove ContentHandlerForModelID hook - 13https://github.com/miraheze/ManageWiki/pull/377 [04:56:45] [02mw-config] 07Universal-Omega closed pull request 03#5206: Replace using 'var' in ManageWikiExtensions, only use 'name' - 13https://github.com/miraheze/mw-config/pull/5206 [04:56:56] [02mw-config] 07Universal-Omega closed pull request 03#5152: Try using redis for session cache - 13https://github.com/miraheze/mw-config/pull/5152 [04:57:06] [02mw-config] 07Universal-Omega closed pull request 03#5099: Namespace and rename MirahezeFunctions - 13https://github.com/miraheze/mw-config/pull/5099 [04:57:13] [02mw-config] 07Universal-Omega closed pull request 03#4907: Include disabled extensions in cache - 13https://github.com/miraheze/mw-config/pull/4907 [04:57:40] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [04:58:35] [02landing] 07Universal-Omega closed pull request 03#115: Add volunteer.php page - 13https://github.com/miraheze/landing/pull/115 [04:58:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.37, 3.57, 3.14 [04:58:58] [02IncidentReporting] 07Universal-Omega closed pull request 03#57: T9999: add `$wgIncidentReportingInactiveServices` - 13https://github.com/miraheze/IncidentReporting/pull/57 [04:59:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.41, 6.95, 7.81 [04:59:24] [02RottenLinks] 07Universal-Omega closed pull request 03#65: Add caching to responses to prevent multiple unnecessary requests - 13https://github.com/miraheze/RottenLinks/pull/65 [04:59:27] [02RottenLinks] 07Universal-Omega reopened pull request 03#65: Add caching to responses to prevent multiple unnecessary requests - 13https://github.com/miraheze/RottenLinks/pull/65 [04:59:32] [02RottenLinks] 07Universal-Omega closed pull request 03#65: Add caching to responses to prevent multiple unnecessary requests - 13https://github.com/miraheze/RottenLinks/pull/65 [04:59:38] [02RottenLinks] 07Universal-Omega closed pull request 03#66: updateExternalLinks: batch insertions - 13https://github.com/miraheze/RottenLinks/pull/66 [05:00:14] [02phabricator-extensions] 07Universal-Omega closed pull request 03#34: Support Swift file storage for Phabricator - 13https://github.com/miraheze/phabricator-extensions/pull/34 [05:00:47] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.51, 3.10, 3.01 [05:01:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.23, 7.72, 7.98 [05:01:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.05, 11.44, 11.83 [05:01:55] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1616538167 secs [05:03:04] miraheze/RottenLinks - Universal-Omega the build passed. [05:07:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.28, 11.76, 11.97 [05:09:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.29, 12.25, 12.09 [05:21:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.85, 11.24, 11.78 [05:26:08] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:27:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.67, 11.56, 11.59 [05:28:22] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [05:29:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.08, 10.24, 11.11 [05:30:15] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.01537081599 secs [05:30:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.31, 10.74, 11.87 [05:31:55] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.09072247148 secs [05:35:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.53, 7.06, 7.88 [05:35:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.71, 12.03, 11.49 [05:36:48] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.23, 3.59, 3.16 [05:37:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.16, 11.58, 11.39 [05:40:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.01, 11.95, 11.66 [05:40:38] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.16, 3.01, 3.04 [05:42:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.79, 11.18, 11.40 [05:45:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.53, 7.84, 7.80 [05:47:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.05, 11.65, 11.43 [05:49:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.84, 11.24, 11.30 [05:55:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.12, 11.36, 11.24 [05:56:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.29, 10.82, 10.82 [05:57:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.96, 11.71, 11.38 [06:00:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.35, 11.52, 11.21 [06:01:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.12, 11.23, 11.22 [06:02:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.37, 12.07, 11.41 [06:07:01] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.13, 4.02, 2.92 [06:09:01] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.56, 3.09, 2.71 [06:11:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.38, 10.94, 11.51 [06:11:41] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 12.25, 6.09, 3.00 [06:15:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.96, 11.58, 11.59 [06:15:41] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.22, 5.52, 3.62 [06:16:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.26, 11.54, 11.75 [06:17:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.86, 7.06, 7.88 [06:18:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.32, 12.22, 11.99 [06:19:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.55, 11.47, 11.58 [06:20:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.54, 11.80, 11.86 [06:21:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.48, 7.75, 7.92 [06:24:09] PROBLEM - cloud14 Puppet on cloud14 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [06:25:16] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:25:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.27, 11.73, 11.56 [06:27:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.26, 7.75, 7.91 [06:27:30] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [06:27:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.32, 10.36, 11.08 [06:28:47] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 3.97, 4.60, 2.73 [06:29:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.10, 8.24, 8.09 [06:30:27] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:32:25] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [06:32:44] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.25, 3.35, 2.69 [06:34:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.39, 10.64, 10.66 [06:36:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.31, 10.43, 10.61 [06:38:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.54, 10.83, 10.71 [06:42:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.25, 11.77, 11.15 [06:50:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.07, 11.02, 10.96 [06:51:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.90, 12.10, 10.94 [06:52:09] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:54:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.26, 11.70, 11.33 [06:56:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.61, 11.92, 11.43 [06:58:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.58, 11.57, 11.35 [06:58:18] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.56, 2.64, 1.89 [07:00:17] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.54, 2.24, 1.84 [07:01:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.61, 11.67, 11.58 [07:03:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.83, 7.37, 7.98 [07:03:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.39, 12.27, 11.83 [07:03:34] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.68, 3.30, 2.70 [07:04:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.60, 11.43, 11.17 [07:06:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.98, 11.29, 11.14 [07:07:31] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.00, 3.20, 2.81 [07:09:23] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.75, 8.03, 8.04 [07:10:16] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.30, 11.83, 11.35 [07:11:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.92, 4.14, 3.28 [07:13:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.10, 11.46, 11.95 [07:14:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.98, 11.31, 11.42 [07:16:33] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.03, 9.06, 5.66 [07:21:19] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:21:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.78, 7.38, 9.80 [07:21:59] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:22:06] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:22:15] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.38, 7.26, 9.60 [07:22:30] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.17, 3.20, 2.01 [07:22:50] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:23:02] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw121 [07:23:13] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 1 backends are down. mw121 [07:23:15] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.433 second response time [07:23:42] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:23:52] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 6.40, 3.82, 2.38 [07:25:02] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [07:25:14] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [07:28:25] PROBLEM - mon141 Current Load on mon141 is CRITICAL: CRITICAL - load average: 5.05, 3.87, 2.27 [07:29:21] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:29:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.88, 7.46, 7.99 [07:29:33] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:29:52] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [07:29:52] PROBLEM - mwtask141 Disk Space on mwtask141 is WARNING: DISK WARNING - free space: / 6867 MB (9% inode=89%); [07:30:07] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.017 second response time [07:30:25] RECOVERY - mon141 Current Load on mon141 is OK: OK - load average: 2.11, 3.17, 2.21 [07:31:08] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 3 backends are down. mw121 mw122 mw141 [07:33:35] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.361 second response time [07:33:35] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [07:33:53] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [07:34:04] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.282 second response time [07:34:13] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.568 second response time [07:34:23] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [07:34:23] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.579 second response time [07:34:25] PROBLEM - mon141 Current Load on mon141 is CRITICAL: CRITICAL - load average: 6.07, 4.47, 2.97 [07:35:01] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [07:35:09] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.379 second response time [07:35:14] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [07:35:17] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.490 second response time [07:35:37] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.164 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [07:36:23] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 16% [07:37:02] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [07:37:07] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [07:37:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.17, 5.82, 7.59 [07:37:51] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [07:37:52] PROBLEM - mwtask141 Disk Space on mwtask141 is CRITICAL: DISK CRITICAL - free space: / 3789 MB (5% inode=89%); [07:38:25] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:38:25] PROBLEM - mon141 Current Load on mon141 is WARNING: WARNING - load average: 2.33, 3.79, 3.06 [07:38:50] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:39:13] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [07:39:22] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 3.31, 4.82, 6.52 [07:39:32] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:40:20] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:40:23] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.218 second response time [07:40:25] RECOVERY - mon141 Current Load on mon141 is OK: OK - load average: 1.39, 3.00, 2.86 [07:40:46] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.310 second response time [07:41:31] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.275 second response time [07:41:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.95, 8.01, 8.02 [07:42:17] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.50, 8.33, 4.72 [07:43:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.15, 6.83, 6.98 [07:43:52] PROBLEM - puppet141 Current Load on puppet141 is WARNING: WARNING - load average: 3.20, 3.98, 3.78 [07:48:16] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.23, 3.89, 3.79 [07:51:42] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:51:52] RECOVERY - puppet141 Current Load on puppet141 is OK: OK - load average: 1.81, 2.68, 3.26 [07:52:16] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.83, 2.27, 3.15 [07:56:18] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 1.06, 2.75, 3.90 [07:59:14] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [08:00:18] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.55, 2.16, 3.36 [08:01:56] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [08:03:55] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.018 second response time [08:04:52] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 1.78, 2.14, 3.80 [08:06:17] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 1.39, 2.98, 7.64 [08:06:20] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [08:08:52] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.34, 1.62, 3.19 [08:10:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.93, 2.95, 6.52 [08:17:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.42, 7.24, 7.86 [08:21:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.16, 7.68, 7.88 [08:24:05] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.12, 6.83, 5.91 [08:26:01] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:26:04] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.62, 4.86, 5.30 [08:28:56] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [08:29:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.34, 10.29, 9.26 [08:31:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.50, 9.73, 9.17 [08:35:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.61, 7.43, 7.84 [08:39:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.16, 7.74, 7.82 [08:41:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.72, 7.69, 7.80 [08:43:23] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.30, 7.88, 7.85 [08:59:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.63, 7.19, 7.98 [09:03:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.18, 7.73, 7.99 [09:04:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.88, 10.55, 9.44 [09:06:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.86, 10.51, 9.55 [09:08:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.47, 10.09, 9.51 [09:12:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.89, 11.38, 10.09 [09:18:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.18, 10.82, 10.10 [09:20:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.85, 11.74, 10.67 [09:22:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.98, 11.73, 10.81 [09:26:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.76, 9.75, 9.89 [09:26:16] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.50, 11.69, 10.92 [09:28:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.96, 11.51, 10.94 [09:29:11] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.49, 2.74, 2.10 [09:31:09] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.14, 2.20, 1.98 [09:34:15] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.06, 8.79, 9.90 [09:45:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.10, 7.14, 7.90 [09:47:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.23, 7.39, 7.89 [09:58:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.87, 11.72, 10.36 [10:00:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.56, 11.23, 10.32 [10:04:49] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.95, 9.19, 9.73 [10:08:26] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 6664 bytes in 0.044 second response time [10:12:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.44, 10.23, 9.90 [10:12:52] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.10, 3.08, 2.16 [10:14:33] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.01, 9.42, 9.62 [10:14:52] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.45, 3.21, 2.33 [10:16:52] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.73, 2.60, 2.20 [10:24:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.11, 9.42, 8.90 [10:26:02] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 10.17, 9.77, 9.10 [10:26:45] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [10:29:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.31, 7.07, 7.94 [10:30:26] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.465 second response time [10:30:45] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [10:31:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.34, 7.70, 8.07 [10:35:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.77, 11.86, 10.34 [10:35:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.12, 11.56, 10.40 [10:37:37] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.99, 12.33, 10.69 [10:37:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.21, 11.20, 10.40 [10:38:26] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 314 bytes in 0.093 second response time [10:39:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.22, 10.77, 10.30 [10:41:49] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.81, 13.05, 11.16 [10:45:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.33, 9.73, 9.97 [10:45:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.74, 11.10, 10.83 [10:55:05] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:55:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.51, 11.70, 11.06 [10:57:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.20, 11.45, 11.06 [10:57:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.49, 11.58, 10.67 [10:59:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.08, 11.53, 10.77 [11:01:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.40, 11.55, 11.17 [11:01:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.63, 11.93, 10.98 [11:03:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.72, 11.31, 11.13 [11:05:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.46, 11.84, 11.34 [11:06:22] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [11:09:44] [02miraheze/ErrorPages] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/ErrorPages/compare/4349eaab8bd7...121376e91314 [11:09:47] [02miraheze/ErrorPages] 07translatewiki 03121376e - Localisation updates from https://translatewiki.net. [11:09:50] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/1f993cc82cc9...c311853499f7 [11:09:52] [02miraheze/CreateWiki] 07translatewiki 03c311853 - Localisation updates from https://translatewiki.net. [11:09:53] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/compare/9b9b9319ceac...d09dca2dc849 [11:09:56] [02miraheze/ManageWiki] 07translatewiki 03d09dca2 - Localisation updates from https://translatewiki.net. [11:09:57] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/landing/compare/e1c1404f32cb...a41d661acc19 [11:10:00] [02miraheze/landing] 07translatewiki 03a41d661 - Localisation updates from https://translatewiki.net. [11:10:03] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+1/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/eda7502c8380...884fe3a85a0c [11:10:04] [02miraheze/MirahezeMagic] 07translatewiki 03884fe3a - Localisation updates from https://translatewiki.net. [11:10:05] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/WikiDiscover/compare/e3fe1b016e28...832f6bbc8241 [11:10:07] [02miraheze/WikiDiscover] 07translatewiki 03832f6bb - Localisation updates from https://translatewiki.net. [11:11:17] miraheze/ErrorPages - translatewiki the build passed. [11:11:54] miraheze/landing - translatewiki the build passed. [11:13:49] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_landing],Exec[git_pull_ErrorPages] [11:17:03] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.40, 4.46, 2.70 [11:17:54] miraheze/WikiDiscover - translatewiki the build passed. [11:19:03] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.67, 3.37, 2.52 [11:19:16] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [11:21:14] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [11:22:23] miraheze/MirahezeMagic - translatewiki the build passed. [11:22:45] miraheze/CreateWiki - translatewiki the build passed. [11:27:04] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:29:16] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [11:49:16] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [11:51:14] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [11:59:52] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [12:01:41] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.67, 6.26, 3.45 [12:03:41] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.88, 6.19, 3.81 [12:13:10] miraheze/ManageWiki - translatewiki the build passed. [12:41:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.74, 7.13, 7.97 [12:45:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.14, 7.59, 7.98 [12:49:49] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [12:51:48] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [12:56:33] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:04:40] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [13:05:05] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:27:58] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [13:29:23] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.67, 3.62, 2.39 [13:31:21] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.16, 2.73, 2.21 [13:33:41] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [13:36:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.28, 3.03, 2.45 [13:38:32] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.84, 3.38, 2.66 [13:40:05] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:40:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.04, 2.89, 2.56 [13:44:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.94, 3.42, 2.89 [13:46:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.79, 2.83, 2.74 [13:56:04] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:56:55] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.21, 3.04, 2.42 [13:58:54] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.08, 2.59, 2.33 [14:01:11] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [14:03:03] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [14:27:05] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.09, 3.07, 2.51 [14:29:01] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.43, 3.34, 2.69 [14:30:09] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:30:56] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.02, 2.87, 2.59 [14:36:43] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.53, 4.41, 3.36 [14:38:05] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [14:40:34] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.17, 3.31, 3.17 [14:44:26] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:12] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:46:29] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.234 second response time [14:49:05] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [14:51:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0d07f1bc1954...feafa9b24781 [14:51:36] [02miraheze/ssl] 07MirahezeSSLBot 03feafa9b - Bot: Update SSL cert for miraheze.wiki [14:53:52] RECOVERY - mwtask141 Disk Space on mwtask141 is OK: DISK OK - free space: / 8662 MB (12% inode=90%); [14:59:31] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [15:01:45] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:02:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.59, 3.14, 2.55 [15:04:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.58, 2.78, 2.48 [15:13:51] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.49, 3.46, 2.92 [15:15:49] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.50, 3.04, 2.82 [15:28:16] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.17, 4.45, 2.98 [15:28:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.17, 3.44, 3.15 [15:30:16] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.09, 3.94, 2.98 [15:30:32] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.95, 4.25, 3.50 [15:32:10] RECOVERY - miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Wed 06 Sep 2023 13:51:26 GMT +0000. [15:32:16] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.70, 3.14, 2.81 [15:32:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.57, 3.28, 3.24 [15:40:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.16, 3.97, 3.49 [15:42:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.23, 3.24, 3.28 [15:44:00] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.68, 2.34, 1.64 [15:45:34] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1374991536 secs [15:46:00] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.80, 2.14, 1.66 [15:49:34] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09611195326 secs [15:58:16] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.02, 4.29, 2.95 [16:00:16] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.46, 3.16, 2.70 [16:03:05] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:11:41] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.14, 5.34, 2.83 [16:13:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.54, 3.56, 3.25 [16:13:41] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.70, 4.26, 2.75 [16:15:15] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.69, 3.20, 3.15 [16:25:23] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:27:34] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [16:45:41] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.67, 5.79, 3.39 [16:47:35] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 8.80, 5.12, 2.58 [16:47:41] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 6.69, 6.16, 3.83 [16:51:37] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.96, 3.71, 2.57 [16:55:00] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.64, 3.28, 2.58 [16:55:35] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.41, 3.14, 2.60 [16:56:59] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.65, 2.64, 2.43 [17:02:41] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.70, 3.82, 3.10 [17:04:40] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.70, 3.71, 3.14 [17:05:17] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.41, 4.09, 3.00 [17:06:40] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.16, 3.86, 3.27 [17:09:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.14, 3.43, 3.00 [17:11:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.30, 2.75, 2.80 [17:16:38] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.43, 3.48, 3.47 [17:18:36] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.41, 2.88, 3.26 [17:19:04] !log [macfan@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@aLL.miraheze.org [17:19:08] !log [macfan@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [17:19:09] !log [macfan@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [17:19:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:19:17] !log [macfan@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [17:19:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:19:31] !log [macfan@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 14s [17:19:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:19:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:19:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:22:37] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.28, 4.13, 3.25 [17:26:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.72, 3.64, 3.25 [17:32:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.62, 2.86, 3.14 [17:48:48] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [17:50:47] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [17:57:48] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.12, 3.91, 2.87 [17:59:48] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.90, 3.11, 2.70 [18:01:49] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [18:10:52] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.42, 2.88, 2.17 [18:12:52] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.05, 2.92, 2.27 [18:26:00] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:28:50] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.46, 2.99, 2.21 [18:28:53] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [18:29:30] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:29:55] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [18:30:48] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.91, 2.64, 2.18 [18:31:10] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [18:31:28] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [18:32:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [18:49:10] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.43, 3.09, 2.50 [18:50:29] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.481 second response time [18:51:05] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.98, 2.85, 2.47 [18:53:48] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.60, 4.17, 3.08 [18:56:15] !log [macfan@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php (START) [18:56:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:48] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.70, 2.93, 2.82 [18:59:17] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [19:02:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [19:09:48] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.59, 3.98, 3.10 [19:11:48] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.07, 3.37, 2.97 [19:17:05] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.48, 3.73, 2.86 [19:19:00] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.73, 2.94, 2.67 [19:26:16] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:28:17] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.08, 3.11, 2.10 [19:28:55] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [19:29:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [19:29:37] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=solarawiki (START) [19:29:42] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=solarawiki (END - exit=0) [19:29:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:30:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:30:16] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.72, 2.54, 2.02 [19:33:28] !log [macfan@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php (START) [19:33:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:56] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:46:57] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.64, 3.33, 2.84 [19:47:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.93, 7.44, 8.00 [19:48:36] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.19, 6.14, 4.20 [19:49:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 7.51, 7.75, 8.06 [19:49:40] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [19:50:35] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 6.53, 6.54, 4.60 [19:50:48] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.08, 3.81, 3.12 [19:51:39] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [19:52:43] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.66, 3.26, 2.99 [20:10:21] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.55, 2.80, 1.69 [20:12:19] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.90, 2.47, 1.71 [20:21:38] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.11, 3.58, 2.71 [20:24:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [20:27:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.97, 3.16, 2.85 [20:27:50] !log [macfan@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php (END - exit=0) [20:27:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:36:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.72, 5.06, 3.49 [20:40:11] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [20:40:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.17, 3.74, 3.33 [20:44:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.79, 2.85, 3.08 [20:53:05] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [20:54:23] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:57:04] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:12] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [20:59:09] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.958 second response time [20:59:50] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 16.94, 9.75, 5.87 [21:01:44] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 10.47, 9.10, 7.46 [21:03:44] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 8.83, 9.02, 7.64 [21:03:47] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 2.71, 6.92, 5.66 [21:05:46] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.63, 5.64, 5.35 [21:08:05] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [21:09:49] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [21:11:26] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.85, 3.42, 2.96 [21:13:25] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.28, 2.62, 2.72 [21:24:34] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:29:39] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [21:29:55] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 26.51, 13.61, 6.29 [21:31:51] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:33:48] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [21:35:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.49, 7.01, 7.92 [21:39:05] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.01, 4.36, 3.13 [21:39:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.55, 7.18, 7.73 [21:41:44] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.29, 2.47, 3.82 [21:43:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.80, 7.56, 7.76 [21:44:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.14, 3.28, 3.06 [21:45:41] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.91, 1.95, 3.32 [21:49:23] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.10, 8.48, 8.04 [21:56:30] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:58:45] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [21:59:53] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [22:00:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 4.00, 3.25, 2.72 [22:01:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.51, 7.42, 7.81 [22:02:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.49, 2.58, 2.54 [22:05:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.40, 7.97, 7.95 [22:16:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.39, 4.49, 3.48 [22:17:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.08, 7.28, 7.99 [22:19:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 7.96, 7.60, 8.01 [22:22:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.29, 3.75, 3.52 [22:26:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.48, 3.27, 3.38 [22:27:16] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:27:50] PROBLEM - mail121 Current Load on mail121 is WARNING: WARNING - load average: 1.80, 1.72, 0.93 [22:28:59] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.16, 3.78, 2.20 [22:29:15] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [22:29:49] RECOVERY - mail121 Current Load on mail121 is OK: OK - load average: 1.52, 1.58, 0.97 [22:30:57] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.09, 2.72, 2.00 [22:32:28] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [22:35:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.22, 3.61, 3.43 [22:37:14] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.10, 3.07, 3.25 [22:38:19] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.27, 7.10, 4.99 [22:39:44] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.72, 3.51, 1.98 [22:41:07] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.91, 5.05, 4.11 [22:45:39] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.57, 3.24, 2.46 [22:46:14] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 3.71, 7.21, 6.33 [22:48:13] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.55, 5.72, 5.88 [22:49:04] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [22:50:43] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.84, 3.93, 3.98 [22:51:03] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [22:52:48] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:55:05] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [22:56:01] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.29, 4.07, 3.04 [22:58:00] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.48, 3.46, 2.95 [23:00:00] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.01, 2.90, 2.80 [23:00:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.36, 2.19, 3.18 [23:08:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.36, 4.82, 3.91 [23:10:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.33, 3.86, 3.67 [23:10:57] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.00, 4.07, 3.24 [23:12:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.61, 3.44, 3.11 [23:14:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.94, 3.15, 3.40 [23:16:55] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.08, 3.06, 3.06 [23:20:12] MacFan4000: is it just me or are XML dumps not working? [23:20:45] hmm, i thought i fixed them error [23:21:02] Is mwtask141 out of space? [23:21:24] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.31, 3.36, 3.41 [23:22:05] I cleared space and reset /srv/mediawiki/cache on mwtask and then ran everything manually [23:22:39] Hmm. Maybe it just takes over an hour? [23:23:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.80, 4.13, 3.73 [23:24:41] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:25:14] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.07, 3.41, 3.51 [23:27:10] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.57, 3.05, 3.38 [23:29:55] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [23:30:42] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=devwiki (START) [23:31:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:31:27] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=devwiki (END - exit=0) [23:31:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:32:37] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [23:33:41] CosmicAlpha: just ran devwiki manually [23:33:54] Thanks! [23:33:54] it seems that the DataDump jobs aren [23:34:06] aren't being run for some reason [23:34:08] Can you run dpl3wiki also? [23:34:56] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=dpl3wiki (START) [23:35:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:35:23] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=dpl3wiki (END - exit=0) [23:35:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:36:48] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.86, 3.81, 3.54 [23:38:44] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.10, 4.76, 3.92 [23:40:39] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.11, 3.88, 3.71 [23:48:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.68, 2.68, 3.20 [23:49:06] Thanks MacFan4000 [23:53:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.08, 10.15, 11.68 [23:55:48] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.10, 3.43, 2.75 [23:56:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.76, 3.10, 3.10 [23:57:48] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.61, 2.72, 2.57 [23:58:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.33, 2.67, 2.93 [23:59:17] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK