[01:00:19] RECOVERY - phorge1 Backups Phorge Static on phorge1 is OK: FILE_AGE OK: /var/log/phorge-backup.log is 18 seconds old and 0 bytes [01:43:00] PROBLEM - lophocmatngu.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lophocmatngu.wiki could not be found [05:03:37] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 7.24, 5.78, 4.58 [05:05:37] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: LOAD OK - total load average: 4.54, 5.23, 4.53 [07:14:44] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [07:44:43] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [10:04:07] PROBLEM - puppet1 NTP time on puppet1 is WARNING: NTP WARNING: Offset 0.1330408454 secs [10:08:07] RECOVERY - puppet1 NTP time on puppet1 is OK: NTP OK: Offset 0.05276852846 secs [13:26:27] PROBLEM - bast1 APT on bast1 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [13:27:14] PROBLEM - bots1 APT on bots1 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [13:28:04] PROBLEM - db1 APT on db1 is CRITICAL: APT CRITICAL: 26 packages available for upgrade (1 critical updates). [13:28:13] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [13:29:09] PROBLEM - jobchron1 APT on jobchron1 is CRITICAL: APT CRITICAL: 47 packages available for upgrade (1 critical updates). [13:29:12] PROBLEM - prometheus1 APT on prometheus1 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [13:29:14] PROBLEM - cp4 APT on cp4 is CRITICAL: APT CRITICAL: 29 packages available for upgrade (1 critical updates). [13:29:15] PROBLEM - os1 APT on os1 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [13:29:19] PROBLEM - cp1 APT on cp1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:29:30] PROBLEM - cp2 APT on cp2 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:30:44] PROBLEM - swiftproxy1 APT on swiftproxy1 is CRITICAL: APT CRITICAL: 26 packages available for upgrade (1 critical updates). [13:31:05] PROBLEM - swiftobject1 APT on swiftobject1 is CRITICAL: APT CRITICAL: 26 packages available for upgrade (1 critical updates). [13:32:00] PROBLEM - mem1 APT on mem1 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [13:32:21] PROBLEM - ldap1 APT on ldap1 is CRITICAL: APT CRITICAL: 26 packages available for upgrade (1 critical updates). [13:33:05] PROBLEM - mon1 APT on mon1 is CRITICAL: APT CRITICAL: 63 packages available for upgrade (1 critical updates). [13:33:09] PROBLEM - phorge1 APT on phorge1 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (1 critical updates). [13:34:12] PROBLEM - swiftac1 APT on swiftac1 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (1 critical updates). [13:34:36] PROBLEM - cp6 APT on cp6 is CRITICAL: APT CRITICAL: 29 packages available for upgrade (1 critical updates). [13:34:48] PROBLEM - matomo1 APT on matomo1 is CRITICAL: APT CRITICAL: 50 packages available for upgrade (1 critical updates). [13:35:33] PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:41:18] PROBLEM - cp5 APT on cp5 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates). [14:11:01] RECOVERY - lophocmatngu.wiki - reverse DNS on sslhost is OK: SSL OK - lophocmatngu.wiki reverse DNS resolves to cp3.wikitide.net - CNAME FLAT [19:17:47] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mediawiki [19:18:53] PROBLEM - swiftproxy1 Current Load on swiftproxy1 is CRITICAL: LOAD CRITICAL - total load average: 17.60, 16.97, 7.69 [19:19:47] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [19:24:43] RECOVERY - swiftproxy1 Current Load on swiftproxy1 is OK: LOAD OK - total load average: 0.20, 5.42, 5.35 [20:14:44] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [20:38:38] PROBLEM - phorge1 issue-tracker.wikitide.org HTTPS on phorge1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:38:53] PROBLEM - matomo1 HTTPS on matomo1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [20:38:59] PROBLEM - phorge1 phorge-static.wikitide.org HTTPS on phorge1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:39:22] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:40:38] RECOVERY - phorge1 issue-tracker.wikitide.org HTTPS on phorge1 is OK: HTTP OK: HTTP/1.1 200 OK - 111384 bytes in 0.979 second response time [20:41:05] RECOVERY - phorge1 phorge-static.wikitide.org HTTPS on phorge1 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 58195 bytes in 8.693 second response time [20:41:19] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 3.161 second response time [20:42:51] RECOVERY - matomo1 HTTPS on matomo1 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.608 second response time [20:44:43] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [22:06:07] PROBLEM - puppet1 NTP time on puppet1 is WARNING: NTP WARNING: Offset 0.1032067239 secs [22:12:07] RECOVERY - puppet1 NTP time on puppet1 is OK: NTP OK: Offset 0.0897822082 secs [22:41:10] [02WikiTideOrg/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03update [+0/-0/±1] 13https://github.com/WikiTideOrg/mediawiki-repos/commit/70c7df5f7455 [22:41:12] [02WikiTideOrg/mediawiki-repos] 07Universal-Omega 0370c7df5 - Add MessageCachePerformance and stop using forks for Miraheze extensions [22:41:14] [02mediawiki-repos] 07Universal-Omega created branch 03update - 13https://github.com/WikiTideOrg/mediawiki-repos [22:41:17] [02mediawiki-repos] 07Universal-Omega opened pull request 03#1: Add MessageCachePerformance and stop using forks for Miraheze extensions - 13https://github.com/WikiTideOrg/mediawiki-repos/pull/1 [22:42:33] [02mediawiki-repos] 07Universal-Omega edited pull request 03#1: Add MessageCachePerformance and stop using forks for Miraheze extensions - 13https://github.com/WikiTideOrg/mediawiki-repos/pull/1 [22:44:55] [02mediawiki-repos] 07Universal-Omega edited pull request 03#1: Add MessageCachePerformance and stop using forks for Miraheze extensions - 13https://github.com/WikiTideOrg/mediawiki-repos/pull/1 [22:48:16] [02mediawiki-repos] 07Universal-Omega edited pull request 03#1: Add MessageCachePerformance and stop using forks for Miraheze extensions - 13https://github.com/WikiTideOrg/mediawiki-repos/pull/1 [22:51:11] [02mediawiki-repos] 07Universal-Omega edited pull request 03#1: Add MessageCachePerformance and stop using forks for Miraheze extensions - 13https://github.com/WikiTideOrg/mediawiki-repos/pull/1 [22:51:22] [02mediawiki-repos] 07Universal-Omega edited pull request 03#1: Add MessageCachePerformance and stop using forks for Miraheze extensions - 13https://github.com/WikiTideOrg/mediawiki-repos/pull/1 [22:51:25] [02mediawiki-repos] 07Universal-Omega edited pull request 03#1: Add MessageCachePerformance and stop using forks for Miraheze extensions - 13https://github.com/WikiTideOrg/mediawiki-repos/pull/1 [22:52:15] [02mediawiki-repos] 07Universal-Omega edited pull request 03#1: Add MessageCachePerformance and stop using forks for Miraheze extensions - 13https://github.com/WikiTideOrg/mediawiki-repos/pull/1 [22:52:19] [02mediawiki-repos] 07Universal-Omega edited pull request 03#1: Add MessageCachePerformance and stop using forks for Miraheze extensions - 13https://github.com/WikiTideOrg/mediawiki-repos/pull/1 [22:53:00] [02mediawiki-repos] 07Universal-Omega edited pull request 03#1: Add MessageCachePerformance and stop using forks for Miraheze extensions - 13https://github.com/WikiTideOrg/mediawiki-repos/pull/1 [22:53:02] [02mediawiki-repos] 07Universal-Omega edited pull request 03#1: Add MessageCachePerformance and stop using forks for Miraheze extensions - 13https://github.com/WikiTideOrg/mediawiki-repos/pull/1 [23:00:43] [02mediawiki-repos] 07Universal-Omega edited pull request 03#1: Add MessageCachePerformance and stop using forks for Miraheze extensions - 13https://github.com/WikiTideOrg/mediawiki-repos/pull/1 [23:00:45] [02mediawiki-repos] 07Universal-Omega edited pull request 03#1: Add MessageCachePerformance and stop using forks for Miraheze extensions - 13https://github.com/WikiTideOrg/mediawiki-repos/pull/1 [23:01:42] [02mediawiki-repos] 07Universal-Omega synchronize pull request 03#1: Add MessageCachePerformance and stop using forks for Miraheze extensions - 13https://github.com/WikiTideOrg/mediawiki-repos/pull/1 [23:01:43] [02WikiTideOrg/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03update [+0/-0/±1] 13https://github.com/WikiTideOrg/mediawiki-repos/compare/70c7df5f7455...4554ce2e7d86 [23:01:45] [02WikiTideOrg/mediawiki-repos] 07Universal-Omega 034554ce2 - Update GlobalNewFiles repo [23:02:39] [02mediawiki-repos] 07Universal-Omega edited pull request 03#1: Add MessageCachePerformance and stop using forks for Miraheze extensions - 13https://github.com/WikiTideOrg/mediawiki-repos/pull/1 [23:02:41] [02mediawiki-repos] 07Universal-Omega edited pull request 03#1: Add MessageCachePerformance and stop using forks for Miraheze extensions - 13https://github.com/WikiTideOrg/mediawiki-repos/pull/1 [23:14:07] PROBLEM - puppet1 NTP time on puppet1 is WARNING: NTP WARNING: Offset 0.1009750366 secs [23:16:14] [02mediawiki-repos] 07Universal-Omega edited pull request 03#1: Add MessageCachePerformance and stop using forks for Miraheze extensions - 13https://github.com/WikiTideOrg/mediawiki-repos/pull/1 [23:16:16] [02mediawiki-repos] 07Universal-Omega edited pull request 03#1: Add MessageCachePerformance and stop using forks for Miraheze extensions - 13https://github.com/WikiTideOrg/mediawiki-repos/pull/1 [23:20:07] RECOVERY - puppet1 NTP time on puppet1 is OK: NTP OK: Offset 0.09170436859 secs