[00:02:06] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.79, 3.12, 2.94 [00:04:06] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.98, 3.13, 2.97 [00:08:04] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.80, 3.81, 3.39 [00:10:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.25, 4.86, 3.86 [00:16:01] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.72, 3.92, 3.93 [00:19:59] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.06, 3.22, 3.58 [00:21:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.27, 2.96, 3.45 [00:22:22] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [00:23:58] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.27, 2.38, 3.18 [01:07:42] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.06, 4.11, 3.14 [01:09:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.36, 3.46, 3.02 [01:11:41] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 20.08, 9.51, 5.25 [01:21:37] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.36, 3.08, 3.85 [01:29:33] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.08, 2.71, 3.40 [02:11:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/dc520f67f1ba [02:11:32] [02miraheze/puppet] 07paladox 03dc520f6 - swift: fix disable_rsync script [02:11:35] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [02:11:37] [02puppet] 07paladox opened pull request 03#3266: swift: fix disable_rsync script - 13https://github.com/miraheze/puppet/pull/3266 [02:12:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.97, 6.89, 4.29 [02:18:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.24, 3.53, 3.54 [02:20:15] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.07, 3.85, 3.65 [02:22:15] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.78, 3.04, 3.38 [02:28:13] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.08, 4.37, 3.80 [02:30:12] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.77, 3.46, 3.54 [02:32:12] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.98, 2.96, 3.34 [02:36:12] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.75, 4.90, 3.99 [02:46:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.22, 3.12, 3.68 [02:50:07] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.97, 2.63, 3.35 [03:01:55] RECOVERY - db131 Backups SQL on db131 is OK: FILE_AGE OK: /var/log/sql-backup.log is 113 seconds old and 275 bytes [03:03:50] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.12, 4.79, 3.16 [03:05:45] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.50, 3.79, 2.99 [03:07:40] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.90, 3.16, 2.85 [03:08:00] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.79, 3.54, 3.07 [03:11:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.27, 4.32, 3.48 [03:15:56] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.11, 3.61, 3.42 [03:17:56] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.55, 2.93, 3.19 [03:25:52] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.85, 5.10, 3.65 [03:27:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.21, 3.98, 3.41 [03:33:31] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.46, 2.85, 3.12 [03:39:46] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.50, 4.40, 3.42 [03:41:45] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.94, 3.66, 3.28 [03:43:44] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.22, 4.07, 3.51 [03:45:45] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.33, 3.06, 3.20 [04:07:36] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.75, 3.32, 2.91 [04:09:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.34, 4.21, 3.29 [04:11:34] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.54, 3.21, 3.03 [04:27:29] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.56, 4.02, 3.03 [04:29:29] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.80, 3.11, 2.82 [04:37:26] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.27, 4.36, 3.35 [04:41:49] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.41, 3.85, 2.86 [04:43:44] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.64, 3.25, 2.75 [04:51:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.43, 3.14, 3.98 [04:55:19] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.48, 3.28, 2.93 [04:57:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.59, 2.33, 3.32 [04:57:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.37, 2.64, 2.74 [05:09:13] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.38, 3.43, 3.25 [05:11:13] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.60, 2.83, 3.05 [05:13:56] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Sun 23 Jul 2023 15:38:14 GMT +0000. [05:25:49] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.19, 3.34, 2.48 [05:27:44] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.00, 2.78, 2.37 [05:39:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.48, 3.25, 2.47 [05:45:58] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address mrlove.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:53:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.54, 3.53, 3.71 [05:57:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.19, 2.42, 3.23 [06:28:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 11.39, 7.44, 4.16 [06:32:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.22, 3.96, 3.48 [06:40:07] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.35, 2.72, 3.15 [07:15:08] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 19.54, 13.42, 6.95 [07:22:58] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1467241943 secs [07:27:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.12, 2.12, 3.74 [07:30:58] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09760847688 secs [07:31:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.64, 1.39, 3.07 [07:43:56] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Sun 23 Jul 2023 15:38:14 GMT +0000. [08:07:47] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.49, 3.56, 2.44 [08:11:46] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.31, 3.59, 2.71 [08:13:45] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.11, 2.67, 2.48 [08:15:56] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address mrlove.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:37:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.77, 3.57, 2.41 [08:41:33] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 10.56, 6.44, 3.75 [08:44:54] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Sun 23 Jul 2023 15:38:14 GMT +0000. [08:49:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.74, 2.93, 3.29 [09:15:56] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address mrlove.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [09:38:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.43, 2.67, 1.74 [09:40:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.21, 3.04, 2.01 [09:44:09] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.63, 3.85, 2.60 [09:46:08] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.55, 2.99, 2.44 [10:23:36] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [10:52:42] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:10:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.05, 4.85, 2.73 [11:21:48] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [11:26:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.99, 2.64, 3.95 [11:30:33] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.53, 1.51, 3.18 [11:36:30] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.40, 3.45, 3.39 [11:46:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.66, 3.34, 3.95 [11:50:25] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.54, 1.84, 3.19 [11:59:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/22b93f3b825b...f4f0366ab1d1 [11:59:08] [02miraheze/puppet] 07paladox 03f4f0366 - memcached_exporter: update to 0.13.0 [12:11:00] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.31, 3.60, 2.90 [12:13:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.29, 3.45, 2.94 [12:17:00] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.06, 3.65, 3.13 [12:19:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.58, 3.65, 3.20 [12:21:02] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 31060 MB (23% inode=99%); [12:23:00] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.31, 3.92, 3.40 [12:25:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.96, 3.91, 3.46 [12:39:00] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.03, 3.77, 3.60 [12:45:11] PROBLEM - lophocmatngu.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address lophocmatngu.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [12:49:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.74, 3.93, 3.82 [13:03:00] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.03, 3.64, 3.69 [13:03:14] PROBLEM - lophocmatngu.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lophocmatngu.wiki could not be found [13:05:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.37, 3.54, 3.65 [13:08:01] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.93, 3.60, 2.30 [13:10:00] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.67, 3.61, 2.47 [13:11:00] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.88, 3.07, 3.39 [13:11:59] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.11, 3.01, 2.38 [13:25:19] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.55, 2.78, 1.76 [13:27:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.38, 2.19, 1.66 [13:43:54] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.96, 6.70, 3.75 [13:47:53] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.55, 3.66, 3.15 [13:49:52] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.85, 2.96, 2.95 [14:06:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/8cf20ec71e2f [14:06:53] [02miraheze/puppet] 07paladox 038cf20ec - Fix [14:06:55] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [14:06:58] [02puppet] 07paladox opened pull request 03#3267: Fix - 13https://github.com/miraheze/puppet/pull/3267 [14:14:49] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.05, 3.11, 1.93 [14:16:49] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.05, 2.31, 1.78 [14:21:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8cf20ec71e2f...3867b4cd9903 [14:21:03] [02miraheze/puppet] 07paladox 033867b4c - Update storage.pp [14:21:04] [02puppet] 07paladox synchronize pull request 03#3267: Fix - 13https://github.com/miraheze/puppet/pull/3267 [14:21:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/3867b4cd9903...c83c7678cdd5 [14:21:53] [02miraheze/puppet] 07paladox 03c83c767 - Create rsync_footer [14:21:56] [02puppet] 07paladox synchronize pull request 03#3267: Fix - 13https://github.com/miraheze/puppet/pull/3267 [14:23:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c83c7678cdd5...fef2bb9085be [14:23:10] [02miraheze/puppet] 07paladox 03fef2bb9 - Update storage.pp [14:23:11] [02puppet] 07paladox synchronize pull request 03#3267: Fix - 13https://github.com/miraheze/puppet/pull/3267 [14:23:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fef2bb9085be...61ee4a78fa30 [14:23:51] [02miraheze/puppet] 07paladox 0361ee4a7 - Update server.pp [14:23:52] [02puppet] 07paladox synchronize pull request 03#3267: Fix - 13https://github.com/miraheze/puppet/pull/3267 [14:26:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/61ee4a78fa30...35f70cdea2a8 [14:26:22] [02miraheze/puppet] 07paladox 0335f70cd - Update server.pp [14:26:25] [02puppet] 07paladox synchronize pull request 03#3267: Fix - 13https://github.com/miraheze/puppet/pull/3267 [14:27:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/35f70cdea2a8...fc0ecc133857 [14:27:19] [02miraheze/puppet] 07paladox 03fc0ecc1 - Update disable_rsync.py [14:27:21] [02puppet] 07paladox synchronize pull request 03#3267: Fix - 13https://github.com/miraheze/puppet/pull/3267 [14:30:13] [02puppet] 07paladox synchronize pull request 03#3267: Fix - 13https://github.com/miraheze/puppet/pull/3267 [14:30:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fc0ecc133857...cfc2cc4963ed [14:30:17] [02miraheze/puppet] 07paladox 03cfc2cc4 - Update disable_rsync.py [14:30:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cfc2cc4963ed...26d69074bb8a [14:30:27] [02miraheze/puppet] 07paladox 0326d6907 - Update disable_rsync.py [14:30:30] [02puppet] 07paladox synchronize pull request 03#3267: Fix - 13https://github.com/miraheze/puppet/pull/3267 [14:31:11] [02puppet] 07paladox synchronize pull request 03#3267: Fix - 13https://github.com/miraheze/puppet/pull/3267 [14:31:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/26d69074bb8a...65ba01748d74 [14:31:14] [02miraheze/puppet] 07paladox 0365ba017 - Update storage.pp [14:34:56] [02puppet] 07paladox synchronize pull request 03#3267: Fix - 13https://github.com/miraheze/puppet/pull/3267 [14:34:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/65ba01748d74...668633437a7f [14:35:00] [02miraheze/puppet] 07paladox 036686334 - Update disable_rsync.py [14:36:27] [02puppet] 07paladox edited pull request 03#3267: swift: Fix disabling rsync - 13https://github.com/miraheze/puppet/pull/3267 [14:36:32] [02puppet] 07paladox closed pull request 03#3267: swift: Fix disabling rsync - 13https://github.com/miraheze/puppet/pull/3267 [14:36:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±3] 13https://github.com/miraheze/puppet/compare/f4f0366ab1d1...81ec2bea3be8 [14:36:34] [02miraheze/puppet] 07paladox 0381ec2be - swift: Fix disabling rsync (#3267) [14:36:37] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [14:36:40] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [14:43:29] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.85, 4.28, 2.90 [14:44:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/81ec2bea3be8...0beb02ef451a [14:44:43] [02miraheze/puppet] 07paladox 030beb02e - swift: fix syntax error in disable_rsync [14:49:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.55, 3.83, 3.37 [14:51:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.70, 3.40, 3.26 [14:56:35] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 14.54, 14.27, 9.32 [14:58:31] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 6.38, 11.26, 8.79 [15:00:26] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 5.46, 9.35, 8.38 [15:09:46] [02puppet] 07paladox closed pull request 03#3266: swift: fix disable_rsync script - 13https://github.com/miraheze/puppet/pull/3266 [15:09:47] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [15:09:48] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [15:10:17] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.17, 4.32, 3.49 [15:10:41] [02puppet] 07paladox synchronize pull request 03#3265: mariadb: increase open table cache to 150k - 13https://github.com/miraheze/puppet/pull/3265 [15:10:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b85477557f92...8dd0bb88d0c9 [15:10:47] [02miraheze/puppet] 07paladox 038dd0bb8 - Update mw.cnf.erb [15:10:49] [02puppet] 07paladox edited pull request 03#3265: mariadb: increase open table cache to 500k - 13https://github.com/miraheze/puppet/pull/3265 [15:12:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.10, 3.83, 3.40 [15:14:15] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.41, 5.28, 3.99 [15:20:14] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.72, 3.11, 3.54 [15:22:13] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.50, 2.27, 3.18 [15:40:25] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f716dd09ad21 [15:40:26] [02miraheze/mediawiki] 07paladox 03f716dd0 - Use InnoDB for search index [15:40:27] [02mediawiki] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mediawiki [15:40:29] [02mediawiki] 07paladox opened pull request 03#15813: Use InnoDB for search index - 13https://github.com/miraheze/mediawiki/pull/15813 [15:51:49] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/f716dd09ad21...750c1cd4413f [15:51:50] [02miraheze/mediawiki] 07paladox 03750c1cd - Update tables.json [15:51:53] [02mediawiki] 07paladox synchronize pull request 03#15813: Use InnoDB for search index - 13https://github.com/miraheze/mediawiki/pull/15813 [15:52:20] [02mediawiki] 07paladox synchronize pull request 03#15813: Use InnoDB for search index - 13https://github.com/miraheze/mediawiki/pull/15813 [15:52:21] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/750c1cd4413f...a8ded4b071f6 [15:52:24] [02miraheze/mediawiki] 07paladox 03a8ded4b - Update tables-generated.sql [16:00:59] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 15.82, 6.22, 3.61 [16:04:48] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 11.21, 15.45, 9.72 [16:04:56] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 49.96, 24.09, 12.07 [16:06:35] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 17.42, 19.51, 11.44 [16:06:44] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 5.73, 11.96, 9.11 [16:06:59] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 1 backends are down. mw132 [16:08:19] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 1 backends are down. mw132 [16:08:39] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 4.38, 9.59, 8.58 [16:08:58] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/a8ded4b071f6...d0da72adbbaa [16:08:59] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 15 backends are healthy [16:08:59] [02miraheze/mediawiki] 07paladox 03d0da72a - Update tables-generated.sql [16:09:03] [02mediawiki] 07paladox synchronize pull request 03#15813: Use InnoDB for search index - 13https://github.com/miraheze/mediawiki/pull/15813 [16:10:19] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 15 backends are healthy [16:10:48] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 5.43, 11.60, 10.35 [16:12:21] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 5.79, 10.30, 9.80 [16:12:46] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 4.90, 9.34, 9.67 [16:14:17] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 4.67, 8.32, 9.13 [16:16:49] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 2.93, 4.15, 2.34 [16:16:56] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.31, 2.78, 3.89 [16:18:49] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.91, 3.00, 2.13 [16:20:54] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.19, 2.24, 3.39 [16:56:40] PROBLEM - wiki.wikimedia.cat - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.40604567527771 seconds [17:06:36] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.44, 2.96, 2.39 [17:12:36] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.43, 3.88, 3.25 [17:14:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.27, 2.94, 2.97 [17:25:40] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:30:30] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.63, 6.15, 4.14 [17:40:28] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.70, 3.60, 3.76 [17:43:56] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Sun 23 Jul 2023 15:38:14 GMT +0000. [17:44:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.81, 2.18, 3.14 [17:53:44] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:54:46] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [18:07:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.44, 6.61, 5.65 [18:09:00] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/d0da72adbbaa...94ca7fd06d3f [18:09:02] [02miraheze/mediawiki] 07paladox 0394ca7fd - Update tables.json [18:09:05] [02mediawiki] 07paladox synchronize pull request 03#15813: Use InnoDB for search index - 13https://github.com/miraheze/mediawiki/pull/15813 [18:09:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.08, 6.67, 5.79 [18:11:25] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 3.36, 5.57, 5.50 [18:15:56] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address mrlove.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [18:22:40] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [18:30:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.84, 7.61, 6.41 [18:32:25] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [18:32:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.25, 6.84, 6.26 [18:34:23] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 3.73, 5.65, 5.88 [18:37:04] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.13, 3.73, 3.03 [18:43:01] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.08, 3.59, 3.26 [18:45:01] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.22, 2.80, 3.01 [18:51:36] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [18:52:10] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.53, 2.95, 2.36 [18:54:04] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.63, 2.41, 2.23 [18:57:55] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.55, 3.14, 2.56 [18:59:50] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.03, 2.79, 2.50 [19:01:21] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.zulaclub.net. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:02:46] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 10.57, 10.83, 6.86 [19:04:42] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.76, 8.97, 6.64 [19:13:56] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Sun 23 Jul 2023 15:38:14 GMT +0000. [19:30:26] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [19:45:56] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address mrlove.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [20:07:28] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.52, 3.19, 2.17 [20:09:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.64, 3.73, 2.53 [20:11:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.27, 4.16, 2.81 [20:15:25] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.49, 2.92, 2.62 [20:24:31] RECOVERY - wiki.wikimedia.cat - reverse DNS on sslhost is OK: SSL OK - wiki.wikimedia.cat reverse DNS resolves to cp22.miraheze.org - CNAME OK [20:31:18] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.29, 4.64, 3.05 [20:33:17] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.81, 3.84, 2.95 [20:35:17] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.72, 4.02, 3.10 [20:37:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.83, 3.61, 3.06 [20:41:15] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.85, 5.04, 3.72 [20:55:10] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 28.55, 20.48, 11.18 [20:55:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.18, 2.95, 3.94 [20:57:35] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw132 [20:59:06] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 4.93, 11.61, 9.60 [20:59:35] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 15 backends are healthy [21:01:03] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 4.92, 9.43, 9.04 [21:01:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.10, 2.26, 3.34 [21:28:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/75727d0ee58c [21:28:39] [02miraheze/puppet] 07paladox 0375727d0 - matomo: make archiver concurrent [21:28:41] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [21:28:44] [02puppet] 07paladox opened pull request 03#3268: matomo: make archiver concurrent - 13https://github.com/miraheze/puppet/pull/3268 [21:29:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/75727d0ee58c...e1c8f24a1cd2 [21:29:35] [02miraheze/puppet] 07paladox 03e1c8f24 - Update init.pp [21:29:37] [02puppet] 07paladox synchronize pull request 03#3268: matomo: make archiver concurrent - 13https://github.com/miraheze/puppet/pull/3268 [21:39:11] [02puppet] 07paladox synchronize pull request 03#3268: matomo: make archiver concurrent - 13https://github.com/miraheze/puppet/pull/3268 [21:39:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e1c8f24a1cd2...a0fbe3ea17b8 [21:39:14] [02miraheze/puppet] 07paladox 03a0fbe3e - Update init.pp [21:40:19] [02puppet] 07paladox closed pull request 03#3268: matomo: make archiver concurrent - 13https://github.com/miraheze/puppet/pull/3268 [21:40:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0beb02ef451a...64096cce219a [21:40:22] [02miraheze/puppet] 07paladox 0364096cc - matomo: make archiver concurrent (#3268) [21:40:24] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [21:40:25] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [21:45:58] PROBLEM - matomo121 Puppet on matomo121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:49:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/7ebb1f436808 [21:49:13] [02miraheze/puppet] 07paladox 037ebb1f4 - matomo: fix syntax error and remove matomo-archiver job [21:49:16] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [21:49:17] [02puppet] 07paladox opened pull request 03#3269: matomo: fix syntax error and remove matomo-archiver job - 13https://github.com/miraheze/puppet/pull/3269 [21:49:27] [02puppet] 07paladox closed pull request 03#3269: matomo: fix syntax error and remove matomo-archiver job - 13https://github.com/miraheze/puppet/pull/3269 [21:49:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/64096cce219a...47327e9f2c9c [21:49:36] [02miraheze/puppet] 07paladox 0347327e9 - matomo: fix syntax error and remove matomo-archiver job (#3269) [21:49:37] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [21:49:42] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [21:49:58] RECOVERY - matomo121 Puppet on matomo121 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [21:50:35] Should thr logs bot really be making this many posts? [21:50:38] [02puppet] 07paladox closed pull request 03#3265: mariadb: increase open table cache to 500k - 13https://github.com/miraheze/puppet/pull/3265 [21:50:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/47327e9f2c9c...8e46846329fb [21:50:42] [02miraheze/puppet] 07paladox 038e46846 - mariadb: increase open table cache to 500k (#3265) [21:50:45] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [21:50:47] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [21:54:12] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:23:05] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [22:27:19] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.82, 3.42, 2.92 [22:29:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.52, 2.72, 2.73 [22:29:52] Yes [22:30:20] Its about on par with everything wmf logs [22:36:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.56, 3.22, 2.91 [22:38:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.98, 4.03, 3.23 [22:44:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.15, 3.58, 3.41 [22:46:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.40, 2.80, 3.14 [23:08:22] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.07, 3.38, 2.83 [23:10:22] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.09, 3.08, 2.77 [23:16:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.11, 4.68, 3.62 [23:18:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.89, 3.89, 3.47 [23:22:17] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.65, 2.78, 3.11 [23:36:12] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.49, 4.01, 3.15 [23:38:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.53, 3.40, 3.03 [23:42:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.49, 3.44, 3.18 [23:44:08] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.93, 2.89, 3.01