[00:01:39] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 27.60, 14.03, 5.68 [00:03:36] RECOVERY - cp23 APT on cp23 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [00:09:31] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.31, 3.27, 3.64 [00:11:29] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.72, 2.44, 3.28 [00:13:30] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp23.miraheze.org - NS RECORDS OK [00:25:31] PROBLEM - cp33 APT on cp33 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [00:27:31] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:29:44] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [00:31:29] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 8.22, 4.87, 3.01 [00:33:49] RECOVERY - cp23 APT on cp23 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [00:36:34] PROBLEM - cp32 APT on cp32 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [00:38:43] RECOVERY - cp32 APT on cp32 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:39:00] [02dns] 07MacFan4000 opened pull request 03#359: remove 3 domains that are either expired or no longer pointed to us - 13https://github.com/miraheze/dns/pull/359 [00:39:01] [url] Page not found · GitHub · GitHub | github.com [00:39:30] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.69, 3.82, 3.91 [00:39:55] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.98, 3.57, 2.45 [00:41:29] [02mediawiki] 07dependabot[bot] opened pull request 03#9658: Bump extensions/Translate from `d65a9a7` to `466f16f` - 13https://github.com/miraheze/mediawiki/pull/9658 [00:41:30] [url] Page not found · GitHub · GitHub | github.com [00:41:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Translate-466f16f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b833c3152479 [00:41:32] [02miraheze/mediawiki] 07dependabot[bot] 03b833c31 - Bump extensions/Translate from `d65a9a7` to `466f16f` [00:41:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Translate-466f16f - 13https://github.com/miraheze/mediawiki [00:41:34] ... [00:41:42] [02mediawiki] 07dependabot[bot] opened pull request 03#9659: Bump extensions/SecureLinkFixer from `6574e73` to `a6dc143` - 13https://github.com/miraheze/mediawiki/pull/9659 [00:41:42] [url] Page not found · GitHub · GitHub | github.com [00:41:43] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/SecureLinkFixer-a6dc143 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c5a220d45c31 [00:41:45] [02miraheze/mediawiki] 07dependabot[bot] 03c5a220d - Bump extensions/SecureLinkFixer from `6574e73` to `a6dc143` [00:41:46] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/SecureLinkFixer-a6dc143 - 13https://github.com/miraheze/mediawiki [00:41:47] [url] Page not found · GitHub · GitHub | github.com [00:41:52] [02mediawiki] 07dependabot[bot] opened pull request 03#9660: Bump skins/Evelution from `a09de5f` to `6afe36f` - 13https://github.com/miraheze/mediawiki/pull/9660 [00:41:53] [url] Page not found · GitHub · GitHub | github.com [00:41:53] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Evelution-6afe36f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/28f143328414 [00:41:55] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.85, 2.57, 2.22 [00:41:55] [02miraheze/mediawiki] 07dependabot[bot] 0328f1433 - Bump skins/Evelution from `a09de5f` to `6afe36f` [00:41:56] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Evelution-6afe36f - 13https://github.com/miraheze/mediawiki [00:41:57] ... [00:42:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#9658: Bump extensions/Translate from `d65a9a7` to `466f16f` - 13https://github.com/miraheze/mediawiki/pull/9658 [00:42:04] ... [00:42:05] [02mediawiki] 07github-actions[bot] labeled pull request 03#9658: Bump extensions/Translate from `d65a9a7` to `466f16f` - 13https://github.com/miraheze/mediawiki/pull/9658 [00:42:15] [02mediawiki] 07github-actions[bot] labeled pull request 03#9659: Bump extensions/SecureLinkFixer from `6574e73` to `a6dc143` - 13https://github.com/miraheze/mediawiki/pull/9659 [00:42:16] [02mediawiki] 07github-actions[bot] labeled pull request 03#9660: Bump skins/Evelution from `a09de5f` to `6afe36f` - 13https://github.com/miraheze/mediawiki/pull/9660 [00:43:28] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.58, 2.06, 3.16 [00:45:03] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [00:46:11] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 13.85, 5.47, 2.39 [00:49:12] RECOVERY - cp22 APT on cp22 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [00:49:19] [02dns] 07Universal-Omega closed pull request 03#359: remove 3 domains that are either expired or no longer pointed to us - 13https://github.com/miraheze/dns/pull/359 [00:49:20] [url] Page not found · GitHub · GitHub | github.com [00:49:21] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-3/±0] 13https://github.com/miraheze/dns/compare/9cd12f15480c...50c52fb1823e [00:49:21] [url] Comparing 9cd12f15480c...50c52fb1823e · miraheze/dns · GitHub | github.com [00:49:22] [02miraheze/dns] 07MacFan4000 0350c52fb - remove 3 domains that are either expired or no longer pointed to us (#359) [00:50:18] [02mediawiki] 07Universal-Omega closed pull request 03#9658: Bump extensions/Translate from `d65a9a7` to `466f16f` - 13https://github.com/miraheze/mediawiki/pull/9658 [00:50:19] [url] Page not found · GitHub · GitHub | github.com [00:50:20] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/c40e48dedfea...4fd2acd78107 [00:50:21] [url] Comparing c40e48dedfea...4fd2acd78107 · miraheze/mediawiki · GitHub | github.com [00:50:21] [02miraheze/mediawiki] 07dependabot[bot] 034fd2acd - Bump extensions/Translate from `d65a9a7` to `466f16f` (#9658) [00:50:23] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/extensions/Translate-466f16f [00:50:24] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/extensions/Translate-466f16f - 13https://github.com/miraheze/mediawiki [00:50:25] [url] Page not found · GitHub · GitHub | github.com [00:50:30] [02mediawiki] 07Universal-Omega closed pull request 03#9659: Bump extensions/SecureLinkFixer from `6574e73` to `a6dc143` - 13https://github.com/miraheze/mediawiki/pull/9659 [00:50:30] [url] Page not found · GitHub · GitHub | github.com [00:50:31] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/4fd2acd78107...a702ef36a2e5 [00:50:32] [url] Comparing 4fd2acd78107...a702ef36a2e5 · miraheze/mediawiki · GitHub | github.com [00:50:33] [02miraheze/mediawiki] 07dependabot[bot] 03a702ef3 - Bump extensions/SecureLinkFixer from `6574e73` to `a6dc143` (#9659) [00:50:34] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/extensions/SecureLinkFixer-a6dc143 [00:50:36] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/extensions/SecureLinkFixer-a6dc143 - 13https://github.com/miraheze/mediawiki [00:50:36] [url] Page not found · GitHub · GitHub | github.com [00:51:00] [02mediawiki] 07Universal-Omega closed pull request 03#9660: Bump skins/Evelution from `a09de5f` to `6afe36f` - 13https://github.com/miraheze/mediawiki/pull/9660 [00:51:01] ... [00:51:02] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/skins/Evelution-6afe36f - 13https://github.com/miraheze/mediawiki [00:51:02] [url] Page not found · GitHub · GitHub | github.com [00:51:03] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/a702ef36a2e5...38092cbb770c [00:51:04] [url] Comparing a702ef36a2e5...38092cbb770c · miraheze/mediawiki · GitHub | github.com [00:51:05] [02miraheze/mediawiki] 07dependabot[bot] 0338092cb - Bump skins/Evelution from `a09de5f` to `6afe36f` (#9660) [00:51:06] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/skins/Evelution-6afe36f [00:52:09] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 0.37, 3.60, 2.83 [00:54:09] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.34, 2.49, 2.51 [01:09:29] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8c46c8c7f3e7...2faa8ba89238 [01:09:29] [url] Comparing 8c46c8c7f3e7...2faa8ba89238 · miraheze/puppet · GitHub | github.com [01:09:30] [02miraheze/puppet] 07Universal-Omega 032faa8ba - site: remove cp30/31 [01:10:24] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/puppet/commit/1cc220b83249 [01:10:25] [02miraheze/puppet] 07Universal-Omega 031cc220b - Remove cp30 [01:10:27] [02puppet] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/puppet [01:10:27] [url] Page not found · GitHub · GitHub | github.com [01:10:30] [02puppet] 07Universal-Omega opened pull request 03#2873: Remove cp30/cp31 - 13https://github.com/miraheze/puppet/pull/2873 [01:10:30] [url] Page not found · GitHub · GitHub | github.com [01:11:00] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/1cc220b83249...bec5980e9c3e [01:11:00] [url] Comparing 1cc220b83249...bec5980e9c3e · miraheze/puppet · GitHub | github.com [01:11:01] [02miraheze/puppet] 07Universal-Omega 03bec5980 - Remove cp31 [01:11:03] [02puppet] 07Universal-Omega synchronize pull request 03#2873: Remove cp30/cp31 - 13https://github.com/miraheze/puppet/pull/2873 [01:11:03] [url] Page not found · GitHub · GitHub | github.com [01:13:17] [02mediawiki] 07dependabot[bot] opened pull request 03#9661: Bump extensions/TorBlock from `d2d130e` to `b96f791` - 13https://github.com/miraheze/mediawiki/pull/9661 [01:13:18] [url] Page not found · GitHub · GitHub | github.com [01:13:19] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_39/extensions/TorBlock-b96f791 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a0a9fbc74ebc [01:13:20] [02miraheze/mediawiki] 07dependabot[bot] 03a0a9fbc - Bump extensions/TorBlock from `d2d130e` to `b96f791` [01:13:22] [02mediawiki] 07dependabot[bot] labeled pull request 03#9661: Bump extensions/TorBlock from `d2d130e` to `b96f791` - 13https://github.com/miraheze/mediawiki/pull/9661 [01:13:23] ... [01:13:23] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_39/extensions/TorBlock-b96f791 - 13https://github.com/miraheze/mediawiki [01:13:24] [url] Page not found · GitHub · GitHub | github.com [01:13:26] [02mediawiki] 07dependabot[bot] opened pull request 03#9662: Bump extensions/DiscussionTools from `73299b2` to `404f693` - 13https://github.com/miraheze/mediawiki/pull/9662 [01:13:27] ... [01:13:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_39/extensions/DiscussionTools-404f693 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/feb76df98874 [01:13:29] [02miraheze/mediawiki] 07dependabot[bot] 03feb76df - Bump extensions/DiscussionTools from `73299b2` to `404f693` [01:13:31] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_39/extensions/DiscussionTools-404f693 - 13https://github.com/miraheze/mediawiki [01:13:31] ... [01:13:32] [02mediawiki] 07dependabot[bot] labeled pull request 03#9662: Bump extensions/DiscussionTools from `73299b2` to `404f693` - 13https://github.com/miraheze/mediawiki/pull/9662 [01:13:33] [url] Page not found · GitHub · GitHub | github.com [01:13:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#9661: Bump extensions/TorBlock from `d2d130e` to `b96f791` - 13https://github.com/miraheze/mediawiki/pull/9661 [01:13:38] [url] Page not found · GitHub · GitHub | github.com [01:13:39] [02mediawiki] 07dependabot[bot] opened pull request 03#9663: Bump extensions/CheckUser from `aef880c` to `333d31b` - 13https://github.com/miraheze/mediawiki/pull/9663 [01:13:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_39/extensions/CheckUser-333d31b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7fa716b74b72 [01:13:42] [02miraheze/mediawiki] 07dependabot[bot] 037fa716b - Bump extensions/CheckUser from `aef880c` to `333d31b` [01:13:43] [02mediawiki] 07dependabot[bot] labeled pull request 03#9663: Bump extensions/CheckUser from `aef880c` to `333d31b` - 13https://github.com/miraheze/mediawiki/pull/9663 [01:13:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_39/extensions/CheckUser-333d31b - 13https://github.com/miraheze/mediawiki [01:13:46] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_39/extensions/SecureLinkFixer-9059c67 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/868050c83b88 [01:13:48] [02miraheze/mediawiki] 07dependabot[bot] 03868050c - Bump extensions/SecureLinkFixer from `ef73b22` to `9059c67` [01:13:49] [02mediawiki] 07dependabot[bot] opened pull request 03#9664: Bump extensions/SecureLinkFixer from `ef73b22` to `9059c67` - 13https://github.com/miraheze/mediawiki/pull/9664 [01:13:51] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_39/extensions/SecureLinkFixer-9059c67 - 13https://github.com/miraheze/mediawiki [01:13:52] [02mediawiki] 07dependabot[bot] labeled pull request 03#9664: Bump extensions/SecureLinkFixer from `ef73b22` to `9059c67` - 13https://github.com/miraheze/mediawiki/pull/9664 [01:13:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#9662: Bump extensions/DiscussionTools from `73299b2` to `404f693` - 13https://github.com/miraheze/mediawiki/pull/9662 [01:13:55] [02mediawiki] 07dependabot[bot] opened pull request 03#9665: Bump extensions/Translate from `d65a9a7` to `466f16f` - 13https://github.com/miraheze/mediawiki/pull/9665 [01:13:57] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_39/extensions/Translate-466f16f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9f354357f079 [01:13:58] [02miraheze/mediawiki] 07dependabot[bot] 039f35435 - Bump extensions/Translate from `d65a9a7` to `466f16f` [01:14:00] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_39/extensions/Translate-466f16f - 13https://github.com/miraheze/mediawiki [01:14:01] [02mediawiki] 07dependabot[bot] labeled pull request 03#9665: Bump extensions/Translate from `d65a9a7` to `466f16f` - 13https://github.com/miraheze/mediawiki/pull/9665 [01:14:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#9663: Bump extensions/CheckUser from `aef880c` to `333d31b` - 13https://github.com/miraheze/mediawiki/pull/9663 [01:14:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#9663: Bump extensions/CheckUser from `aef880c` to `333d31b` - 13https://github.com/miraheze/mediawiki/pull/9663 [01:14:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_39/skins/Evelution-6afe36f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4efe995cdcfe [01:14:07] [02miraheze/mediawiki] 07dependabot[bot] 034efe995 - Bump skins/Evelution from `a09de5f` to `6afe36f` [01:14:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_39/skins/Evelution-6afe36f - 13https://github.com/miraheze/mediawiki [01:14:10] [02mediawiki] 07dependabot[bot] opened pull request 03#9666: Bump skins/Evelution from `a09de5f` to `6afe36f` - 13https://github.com/miraheze/mediawiki/pull/9666 [01:14:12] [02mediawiki] 07dependabot[bot] labeled pull request 03#9666: Bump skins/Evelution from `a09de5f` to `6afe36f` - 13https://github.com/miraheze/mediawiki/pull/9666 [01:14:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#9664: Bump extensions/SecureLinkFixer from `ef73b22` to `9059c67` - 13https://github.com/miraheze/mediawiki/pull/9664 [01:14:26] [02mediawiki] 07github-actions[bot] labeled pull request 03#9666: Bump skins/Evelution from `a09de5f` to `6afe36f` - 13https://github.com/miraheze/mediawiki/pull/9666 [01:14:28] [02mediawiki] 07github-actions[bot] labeled pull request 03#9665: Bump extensions/Translate from `d65a9a7` to `466f16f` - 13https://github.com/miraheze/mediawiki/pull/9665 [01:14:29] [02mediawiki] 07github-actions[bot] labeled pull request 03#9665: Bump extensions/Translate from `d65a9a7` to `466f16f` - 13https://github.com/miraheze/mediawiki/pull/9665 [01:16:54] [02mediawiki] 07Universal-Omega closed pull request 03#9666: Bump skins/Evelution from `a09de5f` to `6afe36f` - 13https://github.com/miraheze/mediawiki/pull/9666 [01:16:55] [url] Page not found · GitHub · GitHub | github.com [01:16:56] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/4249f54593d9...6f1dc1f986db [01:16:57] [url] Comparing 4249f54593d9...6f1dc1f986db · miraheze/mediawiki · GitHub | github.com [01:16:57] [02miraheze/mediawiki] 07dependabot[bot] 036f1dc1f - Bump skins/Evelution from `a09de5f` to `6afe36f` (#9666) [01:16:59] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_39/skins/Evelution-6afe36f [01:17:00] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_39/skins/Evelution-6afe36f - 13https://github.com/miraheze/mediawiki [01:17:01] [url] Page not found · GitHub · GitHub | github.com [01:17:06] [02mediawiki] 07Universal-Omega closed pull request 03#9665: Bump extensions/Translate from `d65a9a7` to `466f16f` - 13https://github.com/miraheze/mediawiki/pull/9665 [01:17:06] ... [01:17:07] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/6f1dc1f986db...9f04c24c44f2 [01:17:08] [url] Comparing 6f1dc1f986db...9f04c24c44f2 · miraheze/mediawiki · GitHub | github.com [01:17:09] [02miraheze/mediawiki] 07dependabot[bot] 039f04c24 - Bump extensions/Translate from `d65a9a7` to `466f16f` (#9665) [01:17:10] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_39/extensions/Translate-466f16f [01:17:12] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_39/extensions/Translate-466f16f - 13https://github.com/miraheze/mediawiki [01:17:12] [url] Page not found · GitHub · GitHub | github.com [01:17:17] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/9f04c24c44f2...0754667581dc [01:17:18] [url] Comparing 9f04c24c44f2...0754667581dc · miraheze/mediawiki · GitHub | github.com [01:17:19] [02miraheze/mediawiki] 07dependabot[bot] 030754667 - Bump extensions/SecureLinkFixer from `ef73b22` to `9059c67` (#9664) [01:17:20] [02mediawiki] 07Universal-Omega closed pull request 03#9664: Bump extensions/SecureLinkFixer from `ef73b22` to `9059c67` - 13https://github.com/miraheze/mediawiki/pull/9664 [01:17:21] [url] Page not found · GitHub · GitHub | github.com [01:17:22] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_39/extensions/SecureLinkFixer-9059c67 [01:17:23] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_39/extensions/SecureLinkFixer-9059c67 - 13https://github.com/miraheze/mediawiki [01:17:23] [url] Page not found · GitHub · GitHub | github.com [01:17:37] [02mediawiki] 07Universal-Omega closed pull request 03#9663: Bump extensions/CheckUser from `aef880c` to `333d31b` - 13https://github.com/miraheze/mediawiki/pull/9663 [01:17:37] [url] Page not found · GitHub · GitHub | github.com [01:17:38] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_39/extensions/CheckUser-333d31b [01:17:40] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/0754667581dc...e09421cfa0d3 [01:17:40] [url] Comparing 0754667581dc...e09421cfa0d3 · miraheze/mediawiki · GitHub | github.com [01:17:41] [02miraheze/mediawiki] 07dependabot[bot] 03e09421c - Bump extensions/CheckUser from `aef880c` to `333d31b` (#9663) [01:17:43] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_39/extensions/CheckUser-333d31b - 13https://github.com/miraheze/mediawiki [01:17:43] [url] Page not found · GitHub · GitHub | github.com [01:17:51] [02mediawiki] 07Universal-Omega closed pull request 03#9662: Bump extensions/DiscussionTools from `73299b2` to `404f693` - 13https://github.com/miraheze/mediawiki/pull/9662 [01:17:52] ... [01:17:53] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_39/extensions/DiscussionTools-404f693 [01:17:54] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/e09421cfa0d3...0376cf8ed7cc [01:17:55] [url] Comparing e09421cfa0d3...0376cf8ed7cc · miraheze/mediawiki · GitHub | github.com [01:17:56] [02miraheze/mediawiki] 07dependabot[bot] 030376cf8 - Bump extensions/DiscussionTools from `73299b2` to `404f693` (#9662) [01:17:57] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_39/extensions/DiscussionTools-404f693 - 13https://github.com/miraheze/mediawiki [01:17:58] [url] Page not found · GitHub · GitHub | github.com [01:18:10] [02mediawiki] 07Universal-Omega closed pull request 03#9661: Bump extensions/TorBlock from `d2d130e` to `b96f791` - 13https://github.com/miraheze/mediawiki/pull/9661 [01:18:10] ... [01:18:11] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_39/extensions/TorBlock-b96f791 - 13https://github.com/miraheze/mediawiki [01:18:12] [url] Page not found · GitHub · GitHub | github.com [01:18:13] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/0376cf8ed7cc...f551cc002535 [01:18:13] [url] Comparing 0376cf8ed7cc...f551cc002535 · miraheze/mediawiki · GitHub | github.com [01:18:14] [02miraheze/mediawiki] 07dependabot[bot] 03f551cc0 - Bump extensions/TorBlock from `d2d130e` to `b96f791` (#9661) [01:18:16] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_39/extensions/TorBlock-b96f791 [01:21:32] [02miraheze/puppet] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [01:21:34] [02puppet] 07Universal-Omega closed pull request 03#2873: Remove cp30/cp31 - 13https://github.com/miraheze/puppet/pull/2873 [01:21:34] [url] Page not found · GitHub · GitHub | github.com [01:21:35] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-2/±0] 13https://github.com/miraheze/puppet/compare/2faa8ba89238...be40d10f6313 [01:21:36] [url] Comparing 2faa8ba89238...be40d10f6313 · miraheze/puppet · GitHub | github.com [01:21:37] [02miraheze/puppet] 07Universal-Omega 03be40d10 - Remove cp30/cp31 (#2873) [01:21:38] [02puppet] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/puppet [01:21:39] [url] Page not found · GitHub · GitHub | github.com [02:04:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.80, 3.00, 3.85 [02:08:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 9.45, 4.56, 4.16 [02:10:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.13, 3.88, 3.96 [02:12:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.89, 5.73, 4.69 [02:44:40] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [02:46:09] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 6.79, 4.18, 2.06 [02:48:49] RECOVERY - cp22 APT on cp22 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [02:52:09] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.24, 3.20, 2.56 [03:00:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.07, 3.93 [03:02:54] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/bb4405012497 [03:02:55] [02miraheze/dns] 07Universal-Omega 03bb44050 - Rename ca data centre to us [03:02:57] [02dns] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/dns [03:02:57] [url] Page not found · GitHub · GitHub | github.com [03:02:58] [02dns] 07Universal-Omega opened pull request 03#360: Rename ca data centre to us - 13https://github.com/miraheze/dns/pull/360 [03:02:59] ... [03:12:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.70, 2.79, 3.37 [03:18:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.77, 3.29, 3.33 [03:20:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.66, 3.10, 3.26 [03:53:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.93, 3.30 [03:55:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.88, 3.14, 3.08 [04:06:57] PROBLEM - cp32 APT on cp32 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [04:07:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 16.36, 7.91, 3.67 [04:12:40] PROBLEM - cp32 PowerDNS Recursor on cp32 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:14:37] RECOVERY - cp32 PowerDNS Recursor on cp32 is OK: DNS OK: 0.249 seconds response time. miraheze.org returns 108.175.15.182,2607:f1c0:1800:26f::1,2607:f1c0:1800:8100::1,74.208.203.152 [04:14:40] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [04:15:32] RECOVERY - cp32 APT on cp32 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:16:09] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.82, 3.29, 1.68 [04:16:41] RECOVERY - cp22 APT on cp22 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [04:18:09] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.36, 2.46, 1.57 [04:31:55] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.22, 1.36, 3.90 [04:35:55] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.66, 1.33, 3.28 [04:36:29] PROBLEM - wiki.kachow.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.kachow.org' expires in 15 day(s) (Wed 12 Oct 2022 04:17:08 GMT +0000). [04:38:06] PROBLEM - cp32 APT on cp32 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [04:39:55] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.97, 5.10, 4.30 [04:42:18] RECOVERY - cp32 APT on cp32 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:42:59] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f99abcbe60fd...8d19247c0921 [04:43:00] [url] Comparing f99abcbe60fd...8d19247c0921 · miraheze/ssl · GitHub | github.com [04:43:01] [02miraheze/ssl] 07MirahezeSSLBot 038d19247 - Bot: Update SSL cert for wiki.kachow.org [04:45:03] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [04:45:55] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.15, 3.02, 3.80 [04:47:03] RECOVERY - cp22 APT on cp22 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [04:49:55] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.63, 1.80, 3.12 [04:53:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.98, 3.45, 2.95 [04:55:04] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.08, 4.45, 3.34 [04:56:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.37, 3.52, 3.12 [04:58:53] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.63, 3.22, 3.06 [05:05:47] RECOVERY - wiki.kachow.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kachow.org' will expire on Sun 25 Dec 2022 03:42:54 GMT +0000. [05:19:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.16, 4.34, 3.32 [05:20:33] [02mw-config] 07Universal-Omega synchronize pull request 03#4907: Include disabled extensions in cache - 13https://github.com/miraheze/mw-config/pull/4907 [05:20:34] [url] Page not found · GitHub · GitHub | github.com [05:20:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-1/±0] 13https://github.com/miraheze/mw-config/compare/36d3fde839f6...d7df36c5c243 [05:20:36] [url] Comparing 36d3fde839f6...d7df36c5c243 · miraheze/mw-config · GitHub | github.com [05:20:36] [02miraheze/mw-config] 07Universal-Omega 03d7df36c - Fix some weird accidental rename [05:21:44] miraheze/mw-config - Universal-Omega the build passed. [05:21:45] [02mw-config] 07Universal-Omega synchronize pull request 03#4907: Include disabled extensions in cache - 13https://github.com/miraheze/mw-config/pull/4907 [05:21:46] [url] Page not found · GitHub · GitHub | github.com [05:21:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d7df36c5c243...dccb3a4f218d [05:21:47] [url] Comparing d7df36c5c243...dccb3a4f218d · miraheze/mw-config · GitHub | github.com [05:21:48] [02miraheze/mw-config] 07Universal-Omega 03dccb3a4 - Update MirahezeFunctions.php [05:22:43] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/dccb3a4f218d...57f76d9a113e [05:22:44] [url] Comparing dccb3a4f218d...57f76d9a113e · miraheze/mw-config · GitHub | github.com [05:22:44] [02miraheze/mw-config] 07Universal-Omega 0357f76d9 - Update LocalSettings.php [05:22:46] [02mw-config] 07Universal-Omega synchronize pull request 03#4907: Include disabled extensions in cache - 13https://github.com/miraheze/mw-config/pull/4907 [05:22:47] [url] Page not found · GitHub · GitHub | github.com [05:22:58] miraheze/mw-config - Universal-Omega the build passed. [05:23:49] miraheze/mw-config - Universal-Omega the build passed. [05:28:27] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.79, 1.57, 1.25 [05:29:29] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [05:30:27] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.14, 1.38, 1.22 [05:31:27] RECOVERY - cp23 APT on cp23 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [05:33:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.27, 3.78, 3.81 [05:37:04] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 11.40, 5.41, 4.29 [05:41:06] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [05:46:10] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 6.65, 3.99, 2.12 [05:48:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.96, 2.98, 3.67 [05:49:18] PROBLEM - swiftobject112 NTP time on swiftobject112 is WARNING: NTP WARNING: Offset -0.1131268144 secs [05:50:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 0.90, 2.26, 3.32 [05:51:42] RECOVERY - cp22 APT on cp22 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [05:52:09] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.95, 3.13, 2.45 [05:55:32] RECOVERY - swiftobject112 NTP time on swiftobject112 is OK: NTP OK: Offset -0.0989202261 secs [05:56:41] PROBLEM - ping6 on cp33 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 106.23 ms [05:58:42] PROBLEM - ping6 on cp33 is WARNING: PING WARNING - Packet loss = 0%, RTA = 105.50 ms [06:06:02] PROBLEM - swiftobject112 NTP time on swiftobject112 is WARNING: NTP WARNING: Offset -0.1156661808 secs [06:07:18] PROBLEM - cp32 APT on cp32 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:07:55] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.09, 2.75, 1.59 [06:08:27] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.07, 1.59, 1.31 [06:09:19] RECOVERY - cp32 APT on cp32 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:09:55] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.38, 2.25, 1.56 [06:12:51] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:da00:1800:328::1/cpweb [06:25:28] PROBLEM - cp33 APT on cp33 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:26:25] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 14.98, 6.83, 3.09 [06:26:47] RECOVERY - swiftobject112 NTP time on swiftobject112 is OK: NTP OK: Offset -0.06634345651 secs [06:27:28] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:29:29] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:30:23] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.11, 3.60, 2.60 [06:31:27] RECOVERY - cp23 APT on cp23 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [06:31:28] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.54, 5.37, 2.75 [06:32:22] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.72, 2.65, 2.37 [06:32:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.85, 5.01, 3.38 [06:33:28] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.77, 3.97, 2.55 [06:35:28] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.47, 2.75, 2.27 [06:36:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.19, 3.45, 3.13 [06:38:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.06, 3.20, 3.06 [06:38:51] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:42:51] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 109.228.51.216/cpweb [06:44:10] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.51, 2.69, 1.48 [06:44:10] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:46:09] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 3.10, 3.03, 1.77 [06:46:10] RECOVERY - cp22 APT on cp22 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [06:48:51] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:52:51] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 217.174.247.33/cpweb [06:54:51] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:14:27] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.22, 1.53, 1.90 [07:14:33] PROBLEM - swiftobject112 NTP time on swiftobject112 is WARNING: NTP WARNING: Offset -0.1680939496 secs [07:18:42] RECOVERY - swiftobject112 NTP time on swiftobject112 is OK: NTP OK: Offset 0.07203936577 secs [07:20:27] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.01, 1.28, 1.67 [07:24:35] PROBLEM - cloud14 Puppet on cloud14 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [07:52:35] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:54:38] PROBLEM - cp33 APT on cp33 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [07:56:37] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [08:55:21] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [08:55:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:58:32] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.90, 1.57, 1.27 [09:02:27] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.41, 1.50, 1.30 [09:05:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.80, 3.05, 2.50 [09:07:28] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.69, 3.19, 2.63 [09:29:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.90, 3.25, 2.56 [09:29:31] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 14.62, 5.88, 2.62 [09:29:31] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [09:31:20] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.01, 3.05, 2.56 [09:31:29] RECOVERY - cp23 APT on cp23 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [09:35:28] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.55, 2.99, 2.48 [09:37:19] PROBLEM - cp32 APT on cp32 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [09:39:19] RECOVERY - cp32 APT on cp32 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [09:40:55] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 14.24, 20.67, 9.77 [09:42:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.25, 4.73, 3.28 [09:44:55] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [09:46:10] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 6.41, 3.46, 1.82 [09:48:09] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 2.96, 3.57, 2.09 [09:49:04] RECOVERY - cp22 APT on cp22 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [09:50:09] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.07, 2.62, 1.92 [09:54:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.47, 3.99, 3.96 [09:56:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.45, 1.37, 3.89 [09:56:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.99, 4.62, 4.20 [09:58:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.04, 3.63, 3.89 [09:59:59] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.11, 0.79, 3.11 [10:00:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.48, 3.85, 3.93 [10:00:36] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [10:00:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:02:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.55, 3.81 [10:08:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.44, 3.61 [10:09:13] PROBLEM - lore.checkit.ws - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - lore.checkit.ws All nameservers failed to answer the query. [10:10:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.37, 3.17, 3.50 [10:12:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.04, 2.43, 3.19 [10:15:29] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [10:16:09] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 6.53, 3.69, 2.05 [10:19:37] RECOVERY - cp22 APT on cp22 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [10:20:09] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.24, 3.07, 2.25 [10:24:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.54, 3.35 [10:26:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.71, 2.81, 3.10 [10:39:13] PROBLEM - lore.checkit.ws - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lore.checkit.ws could not be found [10:44:01] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [10:44:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.53, 5.49, 4.02 [10:46:01] RECOVERY - cp22 APT on cp22 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [10:47:13] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:50:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.95, 3.82 [10:56:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.78, 4.67, 4.09 [10:58:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.77, 3.81 [10:59:31] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [11:00:07] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 14.72, 5.86, 2.70 [11:02:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.33, 3.69, 3.71 [11:03:36] RECOVERY - cp23 APT on cp23 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [11:07:59] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.57, 3.24, 3.13 [11:12:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.95, 3.98 [11:16:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 3.60, 3.77 [11:20:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.69, 3.76 [11:22:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.51, 4.76, 4.20 [11:44:24] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [11:45:12] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [11:48:09] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 6.92, 5.10, 2.69 [11:48:33] RECOVERY - cp22 APT on cp22 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [11:50:09] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.52, 3.66, 2.45 [11:52:09] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.01, 2.75, 2.26 [11:56:00] PROBLEM - cp33 APT on cp33 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [11:57:59] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:11:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.72, 2.79, 2.10 [12:13:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.10, 2.47, 2.07 [12:51:01] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:59:45] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [12:59:58] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 14.15, 6.56, 3.15 [13:03:51] RECOVERY - cp23 APT on cp23 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [13:09:51] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.05, 3.54, 4.00 [13:13:47] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.06, 2.08, 3.28 [13:20:53] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp23.miraheze.org - NS RECORDS OK [13:23:42] PROBLEM - miraheze.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.com' expires in 15 day(s) (Wed 12 Oct 2022 13:21:03 GMT +0000). [13:31:28] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.38, 3.86, 2.82 [13:32:04] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [13:35:50] PROBLEM - miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.wiki' expires in 15 day(s) (Wed 12 Oct 2022 13:11:41 GMT +0000). [13:36:10] RECOVERY - cp23 APT on cp23 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [13:41:28] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.79, 2.93, 3.45 [13:43:28] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.96, 2.21, 3.12 [13:45:05] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [13:49:13] RECOVERY - cp22 APT on cp22 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [13:52:09] PROBLEM - wikiru.wildterra2.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikiru.wildterra2.com All nameservers failed to answer the query. [13:54:11] PROBLEM - runzeppelin.ru - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query runzeppelin.ru. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:55:08] [02miraheze/ImportDump] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 [+0/-0/±2] 13https://github.com/miraheze/ImportDump/commit/ddc0c486c787 [13:55:10] [02miraheze/ImportDump] 07dependabot[bot] 03ddc0c48 - Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 [13:55:11] [02ImportDump] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 - 13https://github.com/miraheze/ImportDump [13:55:12] [url] Page not found · GitHub · GitHub | github.com [13:55:13] [02ImportDump] 07dependabot[bot] opened pull request 03#26: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/ImportDump/pull/26 [13:55:13] [url] Page not found · GitHub · GitHub | github.com [13:55:14] [02ImportDump] 07dependabot[bot] labeled pull request 03#26: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/ImportDump/pull/26 [13:55:15] ... [13:55:16] [02ImportDump] 07dependabot[bot] labeled pull request 03#26: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/ImportDump/pull/26 [13:55:16] [url] Page not found · GitHub · GitHub | github.com [13:55:31] PROBLEM - cp33 APT on cp33 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [13:58:17] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.34, 4.78, 2.57 [13:59:41] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:00:17] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.34, 3.34, 2.31 [14:03:00] miraheze/ImportDump - dependabot[bot] the build has errored. [14:05:16] PROBLEM - pwiki.drydraytonvillagehall.org.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pwiki.drydraytonvillagehall.org.uk' expires in 15 day(s) (Wed 12 Oct 2022 13:33:40 GMT +0000). [14:09:03] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8d19247c0921...d50cfa62daa8 [14:09:05] [02miraheze/ssl] 07MirahezeSSLBot 03d50cfa6 - Bot: Update SSL cert for pwiki.drydraytonvillagehall.org.uk [14:09:15] [url] Comparing 8d19247c0921...d50cfa62daa8 · miraheze/ssl · GitHub | github.com [14:21:58] RECOVERY - wikiru.wildterra2.com - reverse DNS on sslhost is OK: SSL OK - wikiru.wildterra2.com reverse DNS resolves to cp22.miraheze.org - CNAME OK [14:43:49] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [14:45:50] RECOVERY - cp22 APT on cp22 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [14:46:09] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.85, 4.50, 2.39 [14:48:09] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.44, 3.22, 2.17 [14:53:31] RECOVERY - runzeppelin.ru - reverse DNS on sslhost is OK: SSL OK - runzeppelin.ru reverse DNS resolves to cp22.miraheze.org - NS RECORDS OK [14:55:16] PROBLEM - cp33 APT on cp33 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [14:57:16] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:59:28] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.35, 3.55, 1.92 [14:59:29] PROBLEM - cp23 APT on cp23 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [15:03:16] RECOVERY - pwiki.drydraytonvillagehall.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.drydraytonvillagehall.org.uk' will expire on Sun 25 Dec 2022 13:08:57 GMT +0000. [15:03:37] RECOVERY - cp23 APT on cp23 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [15:09:28] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.56, 3.15, 3.40 [15:25:37] PROBLEM - cp33 APT on cp33 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [15:27:39] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:28:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.28, 3.58, 2.27 [15:30:36] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.60, 3.03, 2.21 [15:56:00] PROBLEM - cp33 APT on cp33 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [15:56:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 12.57, 5.46, 2.86 [15:58:01] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:58:18] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.49, 3.99, 2.63 [15:58:32] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.23, 1.68, 1.30 [16:00:17] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.57, 2.81, 2.36 [16:00:29] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.25, 1.49, 1.27 [16:14:40] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [16:16:40] RECOVERY - cp22 APT on cp22 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [16:16:42] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03swift-hostgroup [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/2efc344ea4a9 [16:16:44] [02miraheze/puppet] 07Universal-Omega 032efc344 - Add swift HostGroup [16:16:45] [02puppet] 07Universal-Omega created branch 03swift-hostgroup - 13https://github.com/miraheze/puppet [16:16:46] [url] Page not found · GitHub · GitHub | github.com [16:16:47] [02puppet] 07Universal-Omega opened pull request 03#2874: Add swift HostGroup - 13https://github.com/miraheze/puppet/pull/2874 [16:16:47] ... [16:18:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.08, 2.81, 3.90 [16:18:53] miraheze/ImportDump - dependabot[bot] the build passed. [16:19:14] [02ImportDump] 07Universal-Omega closed pull request 03#26: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/ImportDump/pull/26 [16:19:14] [url] Page not found · GitHub · GitHub | github.com [16:19:15] [02miraheze/ImportDump] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 [16:19:17] [02ImportDump] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 - 13https://github.com/miraheze/ImportDump [16:19:17] ... [16:19:18] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ImportDump/compare/48b409a8192e...7601459669d7 [16:19:19] [url] Comparing 48b409a8192e...7601459669d7 · miraheze/ImportDump · GitHub | github.com [16:19:20] [02miraheze/ImportDump] 07dependabot[bot] 037601459 - Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 (#26) [16:20:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.50, 3.48, 4.03 [16:20:38] [02miraheze/MatomoAnalytics] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 [+0/-0/±2] 13https://github.com/miraheze/MatomoAnalytics/commit/5520c341cc07 [16:20:40] [02miraheze/MatomoAnalytics] 07dependabot[bot] 035520c34 - Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 [16:20:41] [02MatomoAnalytics] 07dependabot[bot] opened pull request 03#84: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/84 [16:20:42] [url] Page not found · GitHub · GitHub | github.com [16:20:43] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#84: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/84 [16:20:44] [url] Page not found · GitHub · GitHub | github.com [16:20:44] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#84: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/84 [16:20:45] [url] Page not found · GitHub · GitHub | github.com [16:20:46] [02MatomoAnalytics] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 - 13https://github.com/miraheze/MatomoAnalytics [16:20:46] ... [16:24:58] miraheze/MatomoAnalytics - dependabot[bot] the build passed. [16:26:00] [02puppet] 07Universal-Omega closed pull request 03#2874: Add swift HostGroup - 13https://github.com/miraheze/puppet/pull/2874 [16:26:01] [url] Page not found · GitHub · GitHub | github.com [16:26:03] [02miraheze/puppet] 07Universal-Omega deleted branch 03swift-hostgroup [16:26:04] [02puppet] 07Universal-Omega deleted branch 03swift-hostgroup - 13https://github.com/miraheze/puppet [16:26:05] ... [16:26:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.96, 3.13, 3.77 [16:28:21] [02MatomoAnalytics] 07Universal-Omega closed pull request 03#84: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/84 [16:28:22] [url] Page not found · GitHub · GitHub | github.com [16:28:23] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MatomoAnalytics/compare/92c43a8f9ee7...45e8116555d7 [16:28:23] miraheze/ImportDump - Universal-Omega the build passed. [16:28:24] [url] Comparing 92c43a8f9ee7...45e8116555d7 · miraheze/MatomoAnalytics · GitHub | github.com [16:28:24] [02miraheze/MatomoAnalytics] 07dependabot[bot] 0345e8116 - Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 (#84) [16:28:25] [02miraheze/MatomoAnalytics] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 [16:28:27] [02MatomoAnalytics] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 - 13https://github.com/miraheze/MatomoAnalytics [16:28:28] ... [16:30:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.69, 4.12, 3.94 [16:31:44] miraheze/MatomoAnalytics - Universal-Omega the build passed. [16:32:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.81, 3.81, 3.84 [16:35:49] [02CreateWiki] 07dependabot[bot] opened pull request 03#338: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/CreateWiki/pull/338 [16:35:50] [url] Page not found · GitHub · GitHub | github.com [16:35:51] [02miraheze/CreateWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/commit/5335c12f99fd [16:35:52] [02miraheze/CreateWiki] 07dependabot[bot] 035335c12 - Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 [16:35:54] [02CreateWiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 - 13https://github.com/miraheze/CreateWiki [16:35:54] [url] Page not found · GitHub · GitHub | github.com [16:35:55] [02CreateWiki] 07dependabot[bot] labeled pull request 03#338: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/CreateWiki/pull/338 [16:35:56] [url] Page not found · GitHub · GitHub | github.com [16:35:57] [02CreateWiki] 07dependabot[bot] labeled pull request 03#338: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/CreateWiki/pull/338 [16:35:57] ... [16:36:18] [02ManageWiki] 07dependabot[bot] opened pull request 03#367: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/ManageWiki/pull/367 [16:36:19] [url] Page not found · GitHub · GitHub | github.com [16:36:19] [02miraheze/ManageWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/commit/550763803def [16:36:20] ... [16:36:21] [02miraheze/ManageWiki] 07dependabot[bot] 035507638 - Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 [16:36:23] [02ManageWiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 - 13https://github.com/miraheze/ManageWiki [16:36:23] [url] Page not found · GitHub · GitHub | github.com [16:36:24] [02ManageWiki] 07dependabot[bot] labeled pull request 03#367: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/ManageWiki/pull/367 [16:36:26] [02ManageWiki] 07dependabot[bot] labeled pull request 03#367: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/ManageWiki/pull/367 [16:36:41] [02miraheze/IncidentReporting] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 [+0/-0/±2] 13https://github.com/miraheze/IncidentReporting/commit/b0c5c413eff5 [16:36:43] [02miraheze/IncidentReporting] 07dependabot[bot] 03b0c5c41 - Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 [16:36:44] [02IncidentReporting] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 - 13https://github.com/miraheze/IncidentReporting [16:36:46] [02IncidentReporting] 07dependabot[bot] opened pull request 03#49: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/IncidentReporting/pull/49 [16:36:47] [02IncidentReporting] 07dependabot[bot] labeled pull request 03#49: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/IncidentReporting/pull/49 [16:36:49] [02IncidentReporting] 07dependabot[bot] labeled pull request 03#49: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/IncidentReporting/pull/49 [16:40:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.83, 4.12, 3.92 [16:42:06] miraheze/ManageWiki - dependabot[bot] the build passed. [16:42:20] miraheze/IncidentReporting - dependabot[bot] the build passed. [16:42:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.12, 3.69, 3.78 [16:42:34] [02ManageWiki] 07Universal-Omega closed pull request 03#367: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/ManageWiki/pull/367 [16:42:35] [url] Page not found · GitHub · GitHub | github.com [16:42:36] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/compare/1e7c2afdc42b...7bdf1c41d405 [16:42:36] [url] Comparing 1e7c2afdc42b...7bdf1c41d405 · miraheze/ManageWiki · GitHub | github.com [16:42:37] [02miraheze/ManageWiki] 07dependabot[bot] 037bdf1c4 - Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 (#367) [16:42:39] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 [16:42:40] [02ManageWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 - 13https://github.com/miraheze/ManageWiki [16:42:41] ... [16:43:05] [02IncidentReporting] 07Universal-Omega closed pull request 03#49: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/IncidentReporting/pull/49 [16:43:05] [url] Page not found · GitHub · GitHub | github.com [16:43:06] [02miraheze/IncidentReporting] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 [16:43:08] [02IncidentReporting] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 - 13https://github.com/miraheze/IncidentReporting [16:43:08] [url] Page not found · GitHub · GitHub | github.com [16:43:09] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/IncidentReporting/compare/9708172bcf88...7d534779953d [16:43:10] [url] Comparing 9708172bcf88...7d534779953d · miraheze/IncidentReporting · GitHub | github.com [16:43:11] [02miraheze/IncidentReporting] 07dependabot[bot] 037d53477 - Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 (#49) [16:46:00] miraheze/CreateWiki - dependabot[bot] the build passed. [16:46:17] miraheze/ManageWiki - Universal-Omega the build passed. [16:46:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.47, 4.67, 4.09 [16:47:46] miraheze/IncidentReporting - Universal-Omega the build passed. [16:48:11] [02CreateWiki] 07Universal-Omega closed pull request 03#338: Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 - 13https://github.com/miraheze/CreateWiki/pull/338 [16:48:12] [url] Page not found · GitHub · GitHub | github.com [16:48:13] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/7eb6a8200554...eeaa93db5e91 [16:48:14] [url] Comparing 7eb6a8200554...eeaa93db5e91 · miraheze/CreateWiki · GitHub | github.com [16:48:14] [02miraheze/CreateWiki] 07dependabot[bot] 03eeaa93d - Bump stylelint-config-idiomatic-order from 8.1.0 to 9.0.0 (#338) [16:48:16] [02CreateWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 - 13https://github.com/miraheze/CreateWiki [16:48:17] [url] Page not found · GitHub · GitHub | github.com [16:48:17] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-idiomatic-order-9.0.0 [16:51:23] PROBLEM - test131 Disk Space on test131 is CRITICAL: DISK CRITICAL - free space: / 1136 MB (5% inode=66%); [16:52:55] [02miraheze/.github] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/.github/compare/554d4a24b28f...207275415de8 [16:52:55] [url] Comparing 554d4a24b28f...207275415de8 · miraheze/.github · GitHub | github.com [16:52:56] [02miraheze/.github] 07Universal-Omega 032072754 - CI: remove phpunit-unit and add REL1_39 tests [16:53:19] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/d549cff7a27b...3bf0cd3ef3c4 [16:53:20] [url] Comparing d549cff7a27b...3bf0cd3ef3c4 · miraheze/WikiDiscover · GitHub | github.com [16:53:21] [02miraheze/WikiDiscover] 07Universal-Omega 033bf0cd3 - Update workflow [16:53:22] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/65c2b827a13f...233be6e1e147 [16:53:23] [url] Comparing 65c2b827a13f...233be6e1e147 · miraheze/DataDump · GitHub | github.com [16:53:24] [02miraheze/DataDump] 07Universal-Omega 03233be6e - Update workflow [16:53:25] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/eeaa93db5e91...ee238f2f121e [16:53:26] [url] Comparing eeaa93db5e91...ee238f2f121e · miraheze/CreateWiki · GitHub | github.com [16:53:27] [02miraheze/CreateWiki] 07Universal-Omega 03ee238f2 - Update workflow [16:53:28] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/7bdf1c41d405...8097a86d5f10 [16:53:29] [url] Comparing 7bdf1c41d405...8097a86d5f10 · miraheze/ManageWiki · GitHub | github.com [16:53:30] [02miraheze/ManageWiki] 07Universal-Omega 038097a86 - Update workflow [16:53:31] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/45e8116555d7...8ede99a59da1 [16:53:32] [url] Comparing 45e8116555d7...8ede99a59da1 · miraheze/MatomoAnalytics · GitHub | github.com [16:53:33] [02miraheze/MatomoAnalytics] 07Universal-Omega 038ede99a - Update workflow [16:53:34] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/604bbd0b4c92...a31d04ff0706 [16:53:35] [url] Comparing 604bbd0b4c92...a31d04ff0706 · miraheze/RemovePII · GitHub | github.com [16:53:36] [02miraheze/RemovePII] 07Universal-Omega 03a31d04f - Update workflow [16:53:37] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/2ce9a5f10a57...3417434a2796 [16:53:38] [url] Comparing 2ce9a5f10a57...3417434a2796 · miraheze/GlobalNewFiles · GitHub | github.com [16:53:39] [02miraheze/GlobalNewFiles] 07Universal-Omega 033417434 - Update workflow [16:53:40] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/39331fcac2c6...68f6d93c85e9 [16:53:41] [url] Comparing 39331fcac2c6...68f6d93c85e9 · miraheze/MirahezeMagic · GitHub | github.com [16:53:42] [02miraheze/MirahezeMagic] 07Universal-Omega 0368f6d93 - Update workflow [16:53:43] [02miraheze/YouTube] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/YouTube/compare/3d370206d250...dcddc740e9ff [16:53:44] [url] Comparing 3d370206d250...dcddc740e9ff · miraheze/YouTube · GitHub | github.com [16:53:45] [02miraheze/YouTube] 07Universal-Omega 03dcddc74 - Update workflow [16:53:46] [02miraheze/SpriteSheet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/SpriteSheet/compare/d61b43323ad9...8d2519cd2512 [16:53:47] [url] Comparing d61b43323ad9...8d2519cd2512 · miraheze/SpriteSheet · GitHub | github.com [16:53:48] [02miraheze/SpriteSheet] 07Universal-Omega 038d2519c - Update workflow [16:53:49] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/7d534779953d...fde4784ea269 [16:53:50] [url] Comparing 7d534779953d...fde4784ea269 · miraheze/IncidentReporting · GitHub | github.com [16:53:51] [02miraheze/IncidentReporting] 07Universal-Omega 03fde4784 - Update workflow [16:53:52] [02miraheze/CustomHeader] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CustomHeader/compare/548f62d8ef5c...41fa2eaaaedd [16:53:53] [url] Comparing 548f62d8ef5c...41fa2eaaaedd · miraheze/CustomHeader · GitHub | github.com [16:53:54] [02miraheze/CustomHeader] 07Universal-Omega 0341fa2ea - Update workflow [16:53:55] [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/PDFEmbed/compare/39d20ebb83cb...6c8fcd47e8c1 [16:53:56] [url] Comparing 39d20ebb83cb...6c8fcd47e8c1 · miraheze/PDFEmbed · GitHub | github.com [16:53:57] [02miraheze/PDFEmbed] 07Universal-Omega 036c8fcd4 - Update workflow [16:53:58] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/7601459669d7...1ed9e4194579 [16:53:59] [url] Comparing 7601459669d7...1ed9e4194579 · miraheze/ImportDump · GitHub | github.com [16:54:00] [02miraheze/ImportDump] 07Universal-Omega 031ed9e41 - Update workflow [16:54:01] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/eeaae6294b77...e20c90c97a0b [16:54:02] [url] Comparing eeaae6294b77...e20c90c97a0b · miraheze/RottenLinks · GitHub | github.com [16:54:03] [02miraheze/RottenLinks] 07Universal-Omega 03e20c90c - Update workflow [17:00:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.30, 3.43, 3.91 [17:02:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.43, 6.13, 4.91 [17:18:39] miraheze/CreateWiki - Universal-Omega the build passed. [17:19:19] miraheze/ManageWiki - Universal-Omega the build passed. [17:21:30] miraheze/WikiDiscover - Universal-Omega the build passed. [17:21:39] miraheze/GlobalNewFiles - Universal-Omega the build passed. [17:22:01] miraheze/DataDump - Universal-Omega the build passed. [17:22:09] miraheze/PDFEmbed - Universal-Omega the build passed. [17:22:16] miraheze/RemovePII - Universal-Omega the build passed. [17:22:59] miraheze/YouTube - Universal-Omega the build passed. [17:23:21] miraheze/RottenLinks - Universal-Omega the build passed. [17:23:31] miraheze/MatomoAnalytics - Universal-Omega the build passed. [17:23:34] miraheze/CustomHeader - Universal-Omega the build passed. [17:23:42] miraheze/SpriteSheet - Universal-Omega the build passed. [17:23:59] miraheze/IncidentReporting - Universal-Omega the build passed. [17:27:47] miraheze/ImportDump - Universal-Omega the build passed. [17:29:40] miraheze/MirahezeMagic - Universal-Omega the build passed. [17:30:32] [02miraheze/.github] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/.github/compare/207275415de8...4940c3c8fea1 [17:30:33] [url] Comparing 207275415de8...4940c3c8fea1 · miraheze/.github · GitHub | github.com [17:30:34] [02miraheze/.github] 07Universal-Omega 034940c3c - CI: mark REL1_39 selenium and phpunit tests non-experimental [17:30:54] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/233be6e1e147...4b93a6dbbf78 [17:30:55] [url] Comparing 233be6e1e147...4b93a6dbbf78 · miraheze/DataDump · GitHub | github.com [17:30:56] [02miraheze/DataDump] 07Universal-Omega 034b93a6d - Update workflow [17:30:57] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/3bf0cd3ef3c4...03bdc8bc1b6e [17:30:58] [url] Comparing 3bf0cd3ef3c4...03bdc8bc1b6e · miraheze/WikiDiscover · GitHub | github.com [17:30:59] [02miraheze/WikiDiscover] 07Universal-Omega 0303bdc8b - Update workflow [17:31:00] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/8ede99a59da1...daf48250699c [17:31:01] [url] Comparing 8ede99a59da1...daf48250699c · miraheze/MatomoAnalytics · GitHub | github.com [17:31:02] [02miraheze/MatomoAnalytics] 07Universal-Omega 03daf4825 - Update workflow [17:31:03] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/3417434a2796...263bbe414ffc [17:31:04] [url] Comparing 3417434a2796...263bbe414ffc · miraheze/GlobalNewFiles · GitHub | github.com [17:31:05] [02miraheze/GlobalNewFiles] 07Universal-Omega 03263bbe4 - Update workflow [17:31:06] [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/PDFEmbed/compare/6c8fcd47e8c1...86efda5c0c76 [17:31:07] [url] Comparing 6c8fcd47e8c1...86efda5c0c76 · miraheze/PDFEmbed · GitHub | github.com [17:31:08] [02miraheze/PDFEmbed] 07Universal-Omega 0386efda5 - Update workflow [17:31:09] [02miraheze/YouTube] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/YouTube/compare/dcddc740e9ff...14c6f2e3bea9 [17:31:10] [url] Comparing dcddc740e9ff...14c6f2e3bea9 · miraheze/YouTube · GitHub | github.com [17:31:11] [02miraheze/YouTube] 07Universal-Omega 0314c6f2e - Update workflow [17:31:12] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/fde4784ea269...1442a7d25b11 [17:31:13] [url] Comparing fde4784ea269...1442a7d25b11 · miraheze/IncidentReporting · GitHub | github.com [17:31:14] [02miraheze/IncidentReporting] 07Universal-Omega 031442a7d - Update workflow [17:31:15] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/e20c90c97a0b...5cb37dc1f128 [17:31:16] [url] Comparing e20c90c97a0b...5cb37dc1f128 · miraheze/RottenLinks · GitHub | github.com [17:31:17] [02miraheze/RottenLinks] 07Universal-Omega 035cb37dc - Update workflow [17:31:18] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/1ed9e4194579...4f8086bae7f9 [17:31:19] [url] Comparing 1ed9e4194579...4f8086bae7f9 · miraheze/ImportDump · GitHub | github.com [17:31:20] [02miraheze/ImportDump] 07Universal-Omega 034f8086b - Update workflow [17:31:21] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/a31d04ff0706...9c997e9061ad [17:31:22] [url] Comparing a31d04ff0706...9c997e9061ad · miraheze/RemovePII · GitHub | github.com [17:31:23] [02miraheze/RemovePII] 07Universal-Omega 039c997e9 - Update workflow [17:31:24] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/8097a86d5f10...a36ef83793ac [17:31:25] [url] Comparing 8097a86d5f10...a36ef83793ac · miraheze/ManageWiki · GitHub | github.com [17:31:26] [02miraheze/ManageWiki] 07Universal-Omega 03a36ef83 - Update workflow [17:31:27] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/68f6d93c85e9...48f0f6f84e54 [17:31:28] [url] Comparing 68f6d93c85e9...48f0f6f84e54 · miraheze/MirahezeMagic · GitHub | github.com [17:31:29] [02miraheze/MirahezeMagic] 07Universal-Omega 0348f0f6f - Update workflow [17:31:30] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/ee238f2f121e...838c1f2d2a70 [17:31:31] [url] Comparing ee238f2f121e...838c1f2d2a70 · miraheze/CreateWiki · GitHub | github.com [17:31:32] [02miraheze/CreateWiki] 07Universal-Omega 03838c1f2 - Update workflow [17:31:33] [02miraheze/SpriteSheet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/SpriteSheet/compare/8d2519cd2512...dea3b3cc6702 [17:31:34] [url] Comparing 8d2519cd2512...dea3b3cc6702 · miraheze/SpriteSheet · GitHub | github.com [17:31:35] [02miraheze/SpriteSheet] 07Universal-Omega 03dea3b3c - Update workflow [17:31:36] [02miraheze/CustomHeader] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CustomHeader/compare/41fa2eaaaedd...d080867a22ca [17:31:37] [url] Comparing 41fa2eaaaedd...d080867a22ca · miraheze/CustomHeader · GitHub | github.com [17:31:38] [02miraheze/CustomHeader] 07Universal-Omega 03d080867 - Update workflow [17:35:22] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/48bfbc72524e [17:35:23] [02miraheze/CreateWiki] 07Universal-Omega 0348bfbc7 - Don't use Maintenance::$mDescription directly [17:35:25] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/CreateWiki [17:35:25] [url] Page not found · GitHub · GitHub | github.com [17:35:26] [02CreateWiki] 07Universal-Omega opened pull request 03#339: Don't use Maintenance::$mDescription directly - 13https://github.com/miraheze/CreateWiki/pull/339 [17:35:27] [url] Page not found · GitHub · GitHub | github.com [17:36:12] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/48bfbc72524e...373c7856fe86 [17:36:12] [url] Comparing 48bfbc72524e...373c7856fe86 · miraheze/CreateWiki · GitHub | github.com [17:36:13] [02miraheze/CreateWiki] 07Universal-Omega 03373c785 - Update checkLastWikiActivity.php [17:36:15] [02CreateWiki] 07Universal-Omega synchronize pull request 03#339: Don't use Maintenance::$mDescription directly - 13https://github.com/miraheze/CreateWiki/pull/339 [17:36:15] [url] Page not found · GitHub · GitHub | github.com [17:37:07] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/373c7856fe86...095a9b39a2fa [17:37:07] [url] Comparing 373c7856fe86...095a9b39a2fa · miraheze/CreateWiki · GitHub | github.com [17:37:08] [02miraheze/CreateWiki] 07Universal-Omega 03095a9b3 - Update deleteWikis.php [17:37:10] [02CreateWiki] 07Universal-Omega synchronize pull request 03#339: Don't use Maintenance::$mDescription directly - 13https://github.com/miraheze/CreateWiki/pull/339 [17:37:10] [url] Page not found · GitHub · GitHub | github.com [17:37:28] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/095a9b39a2fa...226eb3530871 [17:37:29] [url] Comparing 095a9b39a2fa...226eb3530871 · miraheze/CreateWiki · GitHub | github.com [17:37:30] [02miraheze/CreateWiki] 07Universal-Omega 03226eb35 - Update generateMissingCache.php [17:37:31] [02CreateWiki] 07Universal-Omega synchronize pull request 03#339: Don't use Maintenance::$mDescription directly - 13https://github.com/miraheze/CreateWiki/pull/339 [17:37:32] [url] Page not found · GitHub · GitHub | github.com [17:38:34] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/226eb3530871...84ed40eca0b3 [17:38:35] [url] Comparing 226eb3530871...84ed40eca0b3 · miraheze/CreateWiki · GitHub | github.com [17:38:35] [02miraheze/CreateWiki] 07Universal-Omega 0384ed40e - Update manageInactiveWikis.php [17:38:37] [02CreateWiki] 07Universal-Omega synchronize pull request 03#339: Don't use Maintenance::$mDescription directly - 13https://github.com/miraheze/CreateWiki/pull/339 [17:38:37] [url] Page not found · GitHub · GitHub | github.com [17:39:06] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/84ed40eca0b3...042d8dc9de5f [17:39:07] [url] Comparing 84ed40eca0b3...042d8dc9de5f · miraheze/CreateWiki · GitHub | github.com [17:39:07] [02miraheze/CreateWiki] 07Universal-Omega 03042d8dc - Update populateMainPage.php [17:39:09] [02CreateWiki] 07Universal-Omega synchronize pull request 03#339: Don't use Maintenance::$mDescription directly - 13https://github.com/miraheze/CreateWiki/pull/339 [17:39:10] [url] Page not found · GitHub · GitHub | github.com [17:39:27] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/042d8dc9de5f...058f3af5d475 [17:39:28] [url] Comparing 042d8dc9de5f...058f3af5d475 · miraheze/CreateWiki · GitHub | github.com [17:39:28] [02miraheze/CreateWiki] 07Universal-Omega 03058f3af - Update populateWikiCreation.php [17:39:30] [02CreateWiki] 07Universal-Omega synchronize pull request 03#339: Don't use Maintenance::$mDescription directly - 13https://github.com/miraheze/CreateWiki/pull/339 [17:39:30] [url] Page not found · GitHub · GitHub | github.com [17:39:45] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/058f3af5d475...1a5123770a8f [17:39:46] [url] Comparing 058f3af5d475...1a5123770a8f · miraheze/CreateWiki · GitHub | github.com [17:39:47] [02miraheze/CreateWiki] 07Universal-Omega 031a51237 - Update rebuildExtensionListCache.php [17:39:48] [02CreateWiki] 07Universal-Omega synchronize pull request 03#339: Don't use Maintenance::$mDescription directly - 13https://github.com/miraheze/CreateWiki/pull/339 [17:39:48] [url] Page not found · GitHub · GitHub | github.com [17:40:30] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/1a5123770a8f...9bbb7e4e3ec0 [17:40:31] [url] Comparing 1a5123770a8f...9bbb7e4e3ec0 · miraheze/CreateWiki · GitHub | github.com [17:40:32] [02miraheze/CreateWiki] 07Universal-Omega 039bbb7e4 - Update renameWiki.php [17:40:33] [02CreateWiki] 07Universal-Omega synchronize pull request 03#339: Don't use Maintenance::$mDescription directly - 13https://github.com/miraheze/CreateWiki/pull/339 [17:40:34] [url] Page not found · GitHub · GitHub | github.com [17:42:12] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/commit/eec16525d276 [17:42:14] [02miraheze/MatomoAnalytics] 07Universal-Omega 03eec1652 - Don't use Maintenance::$mDescription directly [17:42:15] [02MatomoAnalytics] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/MatomoAnalytics [17:42:16] [url] Page not found · GitHub · GitHub | github.com [17:42:17] [02MatomoAnalytics] 07Universal-Omega opened pull request 03#85: Don't use Maintenance::$mDescription directly - 13https://github.com/miraheze/MatomoAnalytics/pull/85 [17:42:17] ... [17:42:43] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/eec16525d276...52689bbee8e1 [17:42:44] [url] Comparing eec16525d276...52689bbee8e1 · miraheze/MatomoAnalytics · GitHub | github.com [17:42:45] [02miraheze/MatomoAnalytics] 07Universal-Omega 0352689bb - Update addMissingMatomos.php [17:42:46] [02MatomoAnalytics] 07Universal-Omega synchronize pull request 03#85: Don't use Maintenance::$mDescription directly - 13https://github.com/miraheze/MatomoAnalytics/pull/85 [17:42:47] [url] Page not found · GitHub · GitHub | github.com [17:44:27] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/52689bbee8e1...9eab1e8b1ecb [17:44:28] [url] Comparing 52689bbee8e1...9eab1e8b1ecb · miraheze/MatomoAnalytics · GitHub | github.com [17:44:29] [02miraheze/MatomoAnalytics] 07Universal-Omega 039eab1e8 - Update CHANGELOG.md [17:44:30] [02MatomoAnalytics] 07Universal-Omega synchronize pull request 03#85: Don't use Maintenance::$mDescription directly - 13https://github.com/miraheze/MatomoAnalytics/pull/85 [17:44:31] [url] Page not found · GitHub · GitHub | github.com [17:44:43] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/9eab1e8b1ecb...6779c77cf97b [17:44:43] [url] Comparing 9eab1e8b1ecb...6779c77cf97b · miraheze/MatomoAnalytics · GitHub | github.com [17:44:44] [02miraheze/MatomoAnalytics] 07Universal-Omega 036779c77 - Update extension.json [17:44:46] [02MatomoAnalytics] 07Universal-Omega synchronize pull request 03#85: Don't use Maintenance::$mDescription directly - 13https://github.com/miraheze/MatomoAnalytics/pull/85 [17:44:47] [url] Page not found · GitHub · GitHub | github.com [17:45:40] miraheze/DataDump - Universal-Omega the build passed. [17:52:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03add-ceph [+18/-0/±1] 13https://github.com/miraheze/puppet/commit/6ee2f60f25cc [17:52:23] [02miraheze/puppet] 07paladox 036ee2f60 - Add ceph module [17:52:24] [02puppet] 07paladox created branch 03add-ceph - 13https://github.com/miraheze/puppet [17:52:25] [url] Page not found · GitHub · GitHub | github.com [17:52:27] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.19, 1.82, 1.42 [17:52:32] [02puppet] 07paladox opened pull request 03#2875: Add ceph module - 13https://github.com/miraheze/puppet/pull/2875 [17:52:33] [url] Page not found · GitHub · GitHub | github.com [17:55:50] PROBLEM - cp33 APT on cp33 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [17:56:17] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.29, 3.61, 2.12 [17:56:27] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 0.97, 0.71, 0.29 [17:56:32] !log apt upgrade and reboot test131 and mwtask141 (includes kernel upgrade) [17:56:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:57:50] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [17:57:53] miraheze/WikiDiscover - Universal-Omega the build passed. [17:58:17] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.47, 2.68, 1.96 [17:58:27] miraheze/GlobalNewFiles - Universal-Omega the build passed. [17:59:29] miraheze/YouTube - Universal-Omega the build passed. [17:59:39] miraheze/IncidentReporting - Universal-Omega the build passed. [17:59:59] miraheze/PDFEmbed - Universal-Omega the build passed. [18:00:26] miraheze/RottenLinks - Universal-Omega the build passed. [18:00:54] miraheze/RemovePII - Universal-Omega the build passed. [18:00:58] miraheze/SpriteSheet - Universal-Omega the build passed. [18:01:05] miraheze/ManageWiki - Universal-Omega the build passed. [18:01:31] miraheze/CustomHeader - Universal-Omega the build passed. [18:02:31] !log apt upgrade and reboot bast101 (includes kernel upgrade) [18:02:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:03:20] miraheze/ImportDump - Universal-Omega the build passed. [18:05:48] PROBLEM - bast101 Puppet on bast101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[puppet-agent] [18:06:49] miraheze/MirahezeMagic - Universal-Omega the build passed. [18:07:43] RECOVERY - bast101 Puppet on bast101 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [18:07:43] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[git_pull_JobRunner],Exec[git_pull_mathoid],Exec[git_pull_3d2png],Exec[git_pull_femiwiki-deploy] [18:08:07] PROBLEM - db101 APT on db101 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [18:08:16] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1075 bytes in 0.101 second response time [18:08:22] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1075 bytes in 0.119 second response time [18:08:30] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1075 bytes in 0.078 second response time [18:08:40] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1075 bytes in 0.152 second response time [18:09:01] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 972 bytes in 0.942 second response time [18:09:05] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1075 bytes in 0.076 second response time [18:09:06] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1075 bytes in 0.094 second response time [18:09:12] PROBLEM - db101 MariaDB on db101 is CRITICAL: Can't connect to MySQL server on 'db101.miraheze.org' (115) [18:09:28] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1075 bytes in 0.177 second response time [18:11:26] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 6.04, 3.90, 2.36 [18:13:14] miraheze/MatomoAnalytics - Universal-Omega the build passed. [18:13:26] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.24, 3.74, 2.51 [18:14:08] RECOVERY - db101 APT on db101 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [18:14:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 0.98, 1.31, 3.71 [18:15:26] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 5.09, 4.30, 2.87 [18:15:42] miraheze/CreateWiki - Universal-Omega the build passed. [18:15:50] PROBLEM - cp22 APT on cp22 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [18:15:57] miraheze/MatomoAnalytics - Universal-Omega the build passed. [18:16:09] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.81, 2.87, 1.46 [18:16:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 12.60, 5.25, 4.83 [18:17:26] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.50, 3.35, 2.70 [18:17:50] RECOVERY - cp22 APT on cp22 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [18:18:57] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.52, 4.70, 2.57 [18:19:02] PROBLEM - gluster102 PowerDNS Recursor on gluster102 is CRITICAL: CRITICAL - Plugin timed out while executing system call [18:19:17] PROBLEM - gluster102 APT on gluster102 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [18:19:31] PROBLEM - mw122 HTTPS on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:09] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.81, 2.67, 1.81 [18:20:26] PROBLEM - mem101 Current Load on mem101 is CRITICAL: CRITICAL - load average: 2.88, 1.77, 0.86 [18:20:35] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:57] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.62, 4.56, 2.81 [18:20:58] RECOVERY - gluster102 PowerDNS Recursor on gluster102 is OK: DNS OK: 0.634 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [18:21:12] RECOVERY - gluster102 APT on gluster102 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:21:28] RECOVERY - mw122 HTTPS on mw122 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.016 second response time [18:22:25] PROBLEM - mem101 Current Load on mem101 is WARNING: WARNING - load average: 1.94, 1.81, 0.99 [18:22:33] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.009 second response time [18:22:50] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw121 [18:23:26] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 2.63, 3.79, 3.15 [18:23:31] PROBLEM - mem101 Puppet on mem101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[puppet-agent] [18:24:49] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [18:25:26] miraheze/CreateWiki - Universal-Omega the build passed. [18:26:25] RECOVERY - mem101 Current Load on mem101 is OK: OK - load average: 0.55, 1.41, 1.03 [18:27:26] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.34, 3.72, 3.24 [18:30:21] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 30904 bytes in 1.683 second response time [18:30:26] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 30904 bytes in 3.307 second response time [18:30:30] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 30903 bytes in 0.398 second response time [18:30:40] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 30911 bytes in 0.287 second response time [18:31:01] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 35100 bytes in 0.527 second response time [18:31:06] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 30903 bytes in 0.243 second response time [18:31:12] RECOVERY - db101 MariaDB on db101 is OK: Uptime: 85 Threads: 11 Questions: 59147 Slow queries: 3761 Opens: 4566 Open tables: 4560 Queries per second avg: 695.847 [18:31:20] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 30903 bytes in 0.176 second response time [18:31:26] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.87, 3.18, 3.16 [18:31:32] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 30903 bytes in 0.642 second response time [18:32:35] !log apt upgrade on cloud10 and all cloud10 VMs (includes kernel upgrades) [18:35:36] PROBLEM - gluster101 Disk Space on gluster101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:36:28] PROBLEM - bast101 SSH on bast101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:33] PROBLEM - ping6 on bast101 is CRITICAL: PING CRITICAL - Packet loss = 100% [18:36:49] PROBLEM - ping6 on db101 is CRITICAL: PING CRITICAL - Packet loss = 100% [18:36:56] PROBLEM - gluster102 SSH on gluster102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:57] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [18:36:58] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1073 bytes in 3.177 second response time [18:36:58] PROBLEM - ping6 on gluster101 is CRITICAL: PING CRITICAL - Packet loss = 100% [18:37:02] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:02] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:06] PROBLEM - mem101 SSH on mem101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:06] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:06] PROBLEM - bast101 Current Load on bast101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:37:17] PROBLEM - db101 SSH on db101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:31] PROBLEM - mon141 HTTPS on mon141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:32] PROBLEM - Host bast101 is DOWN: PING CRITICAL - Packet loss = 100% [18:37:35] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:35] PROBLEM - ping6 on mem101 is CRITICAL: PING CRITICAL - Packet loss = 100% [18:37:43] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.024 second response time [18:37:46] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:51] PROBLEM - Host gluster102 is DOWN: PING CRITICAL - Packet loss = 100% [18:37:57] PROBLEM - ping6 on db112 is CRITICAL: PING CRITICAL - Packet loss = 100% [18:38:05] PROBLEM - mem101 memcached on mem101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:38:12] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4167 bytes in 2.066 second response time [18:38:13] PROBLEM - mem101 NTP time on mem101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:38:17] PROBLEM - matomo131 HTTPS on matomo131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.017 second response time [18:38:20] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:38:30] PROBLEM - Host gluster101 is DOWN: PING CRITICAL - Packet loss = 100% [18:38:35] PROBLEM - mon141 grafana.miraheze.org HTTPS on mon141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:38:38] PROBLEM - db112 APT on db112 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:38:43] !log shutdown all VMs on cloud10 [18:38:44] PROBLEM - Host mem101 is DOWN: PING CRITICAL - Packet loss = 100% [18:38:52] !log reboot cloud10 [18:38:52] PROBLEM - db112 SSH on db112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:39:00] PROBLEM - Host db112 is DOWN: PING CRITICAL - Packet loss = 100% [18:39:04] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 50% [18:39:13] PROBLEM - Host db101 is DOWN: PING CRITICAL - Packet loss = 100% [18:39:15] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [18:39:25] PROBLEM - ping6 on cloud10 is CRITICAL: PING CRITICAL - Packet loss = 100% [18:39:42] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [18:39:47] PROBLEM - cloud10 conntrack_table_size on cloud10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:39:48] PROBLEM - cloud10 APT on cloud10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:39:49] PROBLEM - cloud10 SSH on cloud10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:39:50] PROBLEM - cloud10 NTP time on cloud10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:40:19] PROBLEM - cloud10 Current Load on cloud10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:40:19] PROBLEM - cloud10 IPMI Sensors on cloud10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:40:22] PROBLEM - cloud10 Disk Space on cloud10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:40:41] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 3 backends are down. mw121 mw131 mw132 [18:40:41] PROBLEM - Host cloud10 is DOWN: PING CRITICAL - Packet loss = 100% [18:42:42] RECOVERY - Host cloud10 is UP: PING OK - Packet loss = 0%, RTA = 1.55 ms [18:43:06] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 1 backends are down. mw132 [18:43:06] !log start all VMs on cloud10 [18:43:09] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 44% [18:43:11] RECOVERY - Host gluster101 is UP: PING OK - Packet loss = 61%, RTA = 0.86 ms [18:43:18] RECOVERY - Host mem101 is UP: PING OK - Packet loss = 44%, RTA = 1.43 ms [18:43:21] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config],Exec[git_pull_landing],Exec[git_pull_ErrorPages],Exec[git_pull_JobRunner] [18:43:25] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.94, 0.50, 0.17 [18:43:25] PROBLEM - mem101 NTP time on mem101 is WARNING: NTP WARNING: Offset 0.2462976873 secs [18:43:28] RECOVERY - Host db112 is UP: PING OK - Packet loss = 0%, RTA = 0.85 ms [18:43:31] RECOVERY - ping6 on cloud10 is OK: PING OK - Packet loss = 0%, RTA = 1.91 ms [18:43:38] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 35101 bytes in 2.070 second response time [18:43:42] PROBLEM - gluster101 Disk Space on gluster101 is WARNING: DISK WARNING - free space: / 102688 MB (10% inode=81%); [18:43:42] RECOVERY - Host db101 is UP: PING OK - Packet loss = 0%, RTA = 0.78 ms [18:43:43] RECOVERY - ping6 on mem101 is OK: PING OK - Packet loss = 0%, RTA = 1.69 ms [18:43:46] RECOVERY - mon141 HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.079 second response time [18:43:48] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 30904 bytes in 1.058 second response time [18:43:48] RECOVERY - cloud10 SSH on cloud10 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:43:54] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 30903 bytes in 0.853 second response time [18:44:02] RECOVERY - matomo131 HTTPS on matomo131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.342 second response time [18:44:03] RECOVERY - ping6 on db112 is OK: PING OK - Packet loss = 0%, RTA = 0.70 ms [18:44:07] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 18991 bytes in 0.127 second response time [18:44:09] RECOVERY - cloud10 Current Load on cloud10 is OK: OK - load average: 9.58, 3.12, 1.11 [18:44:13] RECOVERY - mem101 memcached on mem101 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:105 port 11211 [18:44:14] !log test [18:44:14] RECOVERY - cloud10 APT on cloud10 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:44:14] RECOVERY - cloud10 conntrack_table_size on cloud10 is OK: OK: nf_conntrack is 0 % full RECOVERY - cloud10 NTP time on cloud10 is OK: NTP OK: Offset -0.001562178135 secs [18:44:18] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 30903 bytes in 0.248 second response time [18:44:18] RECOVERY - mem101 Puppet on mem101 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:44:19] PROBLEM - db101 NTP time on db101 is CRITICAL: NTP CRITICAL: Offset 0.7618474364 secs [18:44:19] RECOVERY - cloud10 IPMI Sensors on cloud10 is OK: IPMI Status: OK [18:44:24] RECOVERY - cloud10 Disk Space on cloud10 is OK: DISK OK - free space: / 81353 MB (89% inode=98%); [18:44:24] RECOVERY - Host bast101 is UP: PING OK - Packet loss = 0%, RTA = 0.70 ms [18:44:36] RECOVERY - Host gluster102 is UP: PING OK - Packet loss = 0%, RTA = 1.16 ms [18:44:41] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [18:44:42] RECOVERY - mon141 grafana.miraheze.org HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 200 OK - 36298 bytes in 0.056 second response time [18:44:44] RECOVERY - bast101 SSH on bast101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:44:44] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17684 bytes in 0.121 second response time [18:44:55] !log test [18:44:56] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 30911 bytes in 0.346 second response time [18:44:58] RECOVERY - bast101 Current Load on bast101 is OK: OK - load average: 0.29, 0.23, 0.09 [18:45:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:45:01] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 1% [18:45:01] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [18:45:03] RECOVERY - db112 SSH on db112 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:45:04] RECOVERY - ping6 on bast101 is OK: PING OK - Packet loss = 0%, RTA = 2.13 ms [18:45:06] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 1% [18:45:06] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [18:45:10] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 30903 bytes in 0.339 second response time [18:45:10] RECOVERY - ping6 on gluster101 is OK: PING OK - Packet loss = 0%, RTA = 0.82 ms [18:45:14] RECOVERY - mem101 SSH on mem101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:45:14] RECOVERY - ping6 on db101 is OK: PING OK - Packet loss = 0%, RTA = 0.75 ms [18:45:14] RECOVERY - gluster102 SSH on gluster102 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:45:16] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 30903 bytes in 0.348 second response time [18:45:17] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 30903 bytes in 0.298 second response time [18:45:24] RECOVERY - db112 APT on db112 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:45:26] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.77, 2.93, 2.54 [18:45:26] RECOVERY - mem101 NTP time on mem101 is OK: NTP OK: Offset -0.001481771469 secs [18:45:31] !log apt upgrade on cloud10 and all cloud10 VMs (includes kernel upgrades) [18:45:39] RECOVERY - db101 SSH on db101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:45:41] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [18:45:51] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_JobRunner] [18:46:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:46:09] !log shutdown all VMs on cloud10 [18:46:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:46:15] !log reboot cloud10 [18:46:19] RECOVERY - db101 NTP time on db101 is OK: NTP OK: Offset 0.0008142888546 secs [18:46:23] !log start all VMs on cloud10 [18:46:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:46:33] !log SET GLOBAL read_only = 0; on db101 and db112 [18:46:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:46:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:47:26] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.32, 2.30, 2.36 [18:51:13] [02mw-config] 07AgentIsai opened pull request 03#4909: Update sitenotice to inform users of ongoing maintenance - 13https://github.com/miraheze/mw-config/pull/4909 [18:51:13] [url] Page not found · GitHub · GitHub | github.com [18:52:09] [02mw-config] 07Universal-Omega closed pull request 03#4909: Update sitenotice to inform users of ongoing maintenance - 13https://github.com/miraheze/mw-config/pull/4909 [18:52:10] ... [18:52:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2001e1434963...c48929862a05 [18:52:12] [url] Comparing 2001e1434963...c48929862a05 · miraheze/mw-config · GitHub | github.com [18:52:12] [02miraheze/mw-config] 07AgentIsai 03c489298 - Update sitenotice to inform users of ongoing maintenance (#4909) [18:52:26] miraheze/mw-config - AgentIsai the build passed. [18:53:10] miraheze/mw-config - Universal-Omega the build passed. [18:53:28] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 12.19, 4.84, 2.09 [18:53:38] !log apt upgrade on cloud12 and all cloud12 VMs (includes kernel upgrades) [18:54:46] PROBLEM - cp33 APT on cp33 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [18:55:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 12.43, 9.67, 4.51 [18:55:26] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 6.96, 4.40, 3.14 [18:55:38] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 3.64, 2.95, 1.42 [18:55:50] PROBLEM - db121 APT on db121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [18:56:08] PROBLEM - mail121 Current Load on mail121 is CRITICAL: CRITICAL - load average: 3.60, 2.73, 1.25 [18:56:47] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:56:50] PROBLEM - jobchron121 Current Load on jobchron121 is WARNING: WARNING - load average: 1.97, 1.51, 0.73 [18:56:54] PROBLEM - db121 MariaDB on db121 is CRITICAL: Can't connect to MySQL server on 'db121.miraheze.org' (115) [18:56:59] !log [@test131] starting deploy of {'config': True} to all [18:57:00] !log [@test131] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [18:57:00] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 950 bytes in 0.196 second response time [18:57:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:50] !log apt upgrade on cloud12 and all cloud12 VMs (includes kernel upgrades) [18:57:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:58:33] PROBLEM - mail121 APT on mail121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [18:58:56] PROBLEM - reports121 Puppet on reports121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[puppet-agent] [18:59:02] PROBLEM - mail121 IMAP on mail121 is CRITICAL: connect to address 2a10:6740::6:307 and port 143: Connection refused [18:59:45] PROBLEM - gluster121 Puppet on gluster121 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): Package[logrotate],Package[nano],Package[postfix],Package[glusterfs-server] [18:59:50] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [19:00:35] RECOVERY - mail121 APT on mail121 is OK: APT OK: 19 packages available for upgrade (0 critical updates). [19:00:50] RECOVERY - jobchron121 Current Load on jobchron121 is OK: OK - load average: 1.43, 1.58, 0.95 [19:01:51] RECOVERY - db121 APT on db121 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [19:02:19] PROBLEM - graylog121 Puppet on graylog121 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[puppet-agent],Package[mongodb_server] [19:03:28] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 1.93, 3.50, 2.96 [19:05:28] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.88, 2.97, 2.83 [19:05:38] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [19:05:49] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [19:06:00] PROBLEM - mail121 Current Load on mail121 is WARNING: WARNING - load average: 1.48, 1.86, 1.52 [19:06:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03add-ceph [+18/-0/±2] 13https://github.com/miraheze/puppet/compare/6ee2f60f25cc...b9e90dab9746 [19:06:44] [url] Comparing 6ee2f60f25cc...b9e90dab9746 · miraheze/puppet · GitHub | github.com [19:06:45] [02miraheze/puppet] 07paladox 03b9e90da - Add ceph module [19:06:46] [02puppet] 07paladox synchronize pull request 03#2875: Add ceph module - 13https://github.com/miraheze/puppet/pull/2875 [19:06:47] [url] Page not found · GitHub · GitHub | github.com [19:08:00] RECOVERY - mail121 Current Load on mail121 is OK: OK - load average: 1.40, 1.70, 1.50 [19:08:51] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Package[locales-all],Package[puppet-agent],Package[glusterfs-client] [19:09:02] RECOVERY - mail121 IMAP on mail121 is OK: IMAP OK - 0.004 second response time on 2a10:6740::6:307 port 143 [* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ STARTTLS LOGINDISABLED] Dovecot (Debian) ready.] [19:10:21] !log [@mwtask141] starting deploy of {'config': True} to all [19:10:22] !log [@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [19:10:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:10:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:10:58] !log [universalomega@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [19:11:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:11:05] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 6s [19:11:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:11:38] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 0.63, 1.97, 1.96 [19:13:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.16, 2.47, 3.68 [19:15:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 0.55, 1.80, 3.28 [19:15:38] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 0.24, 1.01, 1.56 [19:19:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03add-ceph [+18/-0/±2] 13https://github.com/miraheze/puppet/compare/b9e90dab9746...b1213b4739f3 [19:19:48] [url] Comparing b9e90dab9746...b1213b4739f3 · miraheze/puppet · GitHub | github.com [19:19:49] [02miraheze/puppet] 07paladox 03b1213b4 - Add ceph module [19:19:50] [02puppet] 07paladox synchronize pull request 03#2875: Add ceph module - 13https://github.com/miraheze/puppet/pull/2875 [19:19:51] [url] Page not found · GitHub · GitHub | github.com [19:22:26] PROBLEM - db121 Puppet on db121 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 3 minutes ago with 5 failures. Failed resources (up to 3 shown): Package[logrotate],Package[nano],Package[postfix],Package[mariadb-server-10.5] [19:23:29] !log apt upgrade and reboot ns1 (includes kernel upgrade) [19:23:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:24:40] What is the update being made at this time? [19:24:44] curiosity [19:27:48] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:28:20] RECOVERY - graylog121 Puppet on graylog121 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [19:28:56] RECOVERY - reports121 Puppet on reports121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:31:26] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 2.18, 3.01, 3.87 [19:33:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.90, 3.48, 2.77 [19:33:29] !log apt upgrade and reboot cp22 and cp23 [19:33:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:34:10] !log apt upgrade on cloud13 and all cloud13 VMs (includes kernel upgrades) [19:34:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:34:45] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [19:35:01] PROBLEM - mem131 APT on mem131 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [19:35:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.14, 3.21, 2.75 [19:35:30] PROBLEM - db131 MariaDB on db131 is CRITICAL: Can't connect to MySQL server on 'db131.miraheze.org' (115) [19:35:53] PROBLEM - mem131 Current Load on mem131 is CRITICAL: CRITICAL - load average: 3.83, 1.92, 0.96 [19:37:18] PROBLEM - es131 Current Load on es131 is WARNING: WARNING - load average: 3.81, 2.97, 1.98 [19:39:10] RECOVERY - mem131 APT on mem131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:39:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 12.53, 5.60, 3.69 [19:39:15] RECOVERY - es131 Current Load on es131 is OK: OK - load average: 2.68, 2.83, 2.04 [19:39:26] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 7.37, 4.61, 4.15 [19:39:52] PROBLEM - mem131 Current Load on mem131 is WARNING: WARNING - load average: 1.08, 1.78, 1.15 [19:40:43] hispano76: kernel I believe [19:40:52] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 26417 bytes in 0.273 second response time [19:40:56] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 26417 bytes in 0.607 second response time [19:41:05] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 26418 bytes in 1.264 second response time [19:41:16] PROBLEM - mem131 memcached on mem131 is CRITICAL: connect to address 2a10:6740::6:405 and port 11211: Connection refused [19:41:19] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:41:37] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 26417 bytes in 0.815 second response time [19:41:51] RECOVERY - mem131 Current Load on mem131 is OK: OK - load average: 0.26, 1.24, 1.02 [19:41:51] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[locales-all] [19:42:30] !log shutdown all VMs on cloud13 [19:42:40] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:43:10] !log reboot cloud13 [19:43:37] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:44:54] PROBLEM - ping6 on prometheus131 is CRITICAL: PING CRITICAL - Packet loss = 100% [19:45:03] PROBLEM - db131 MariaDB on db131 is UNKNOWN: [19:45:30] PROBLEM - prometheus131 SSH on prometheus131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:45:42] PROBLEM - matomo131 HTTPS on matomo131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:45:52] PROBLEM - ping6 on es131 is CRITICAL: PING CRITICAL - Packet loss = 100% [19:45:54] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 75% [19:46:18] PROBLEM - ping6 on db131 is CRITICAL: PING CRITICAL - Packet loss = 100% [19:46:23] PROBLEM - ping6 on test131 is CRITICAL: PING CRITICAL - Packet loss = 100% [19:46:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 0.54, 3.30, 3.65 [19:46:30] PROBLEM - mw131 SSH on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:46:31] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:46:40] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:46:41] PROBLEM - ping6 on mw131 is CRITICAL: PING CRITICAL - Packet loss = 100% [19:46:54] PROBLEM - ping6 on matomo131 is CRITICAL: PING CRITICAL - Packet loss = 100% [19:47:03] PROBLEM - test131 SSH on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:47:04] PROBLEM - db131 Current Load on db131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:47:10] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [19:47:17] PROBLEM - test131 HTTPS on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:47:21] PROBLEM - Host test131 is DOWN: PING CRITICAL - Packet loss = 100% [19:47:24] PROBLEM - ping6 on mw132 is CRITICAL: PING CRITICAL - Packet loss = 100% [19:47:30] PROBLEM - Host mw132 is DOWN: PING CRITICAL - Packet loss = 100% [19:47:37] PROBLEM - Host mw131 is DOWN: PING CRITICAL - Packet loss = 100% [19:47:49] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [19:47:51] PROBLEM - ping6 on mem131 is CRITICAL: PING CRITICAL - Packet loss = 100% [19:48:09] PROBLEM - db131 SSH on db131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:11] PROBLEM - Host mem131 is DOWN: PING CRITICAL - Packet loss = 100% [19:48:21] PROBLEM - matomo131 SSH on matomo131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:23] PROBLEM - prometheus131 Disk Space on prometheus131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:48:30] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [19:48:32] PROBLEM - prometheus131 Prometheus on prometheus131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:33] PROBLEM - db131 conntrack_table_size on db131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:48:33] PROBLEM - matomo131 conntrack_table_size on matomo131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:48:37] PROBLEM - es131 Disk Space on es131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:48:41] PROBLEM - matomo131 Redis Process on matomo131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:48:43] PROBLEM - Host matomo131 is DOWN: PING CRITICAL - Packet loss = 100% [19:48:45] PROBLEM - es131 ferm_active on es131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:48:46] PROBLEM - Host db131 is DOWN: PING CRITICAL - Packet loss = 100% [19:48:48] PROBLEM - es131 SSH on es131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:51] PROBLEM - ping6 on cloud13 is CRITICAL: PING CRITICAL - Packet loss = 100% [19:48:54] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 53% [19:49:00] PROBLEM - cloud13 SSH on cloud13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:49:00] PROBLEM - prometheus131 conntrack_table_size on prometheus131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:49:09] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 0.25, 2.02, 3.10 [19:49:11] PROBLEM - Host prometheus131 is DOWN: PING CRITICAL - Packet loss = 100% [19:49:20] PROBLEM - es131 Puppet on es131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:49:22] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [19:49:23] PROBLEM - Host es131 is DOWN: PING CRITICAL - Packet loss = 100% [19:49:32] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.27, 2.68, 3.72 [19:49:47] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [19:49:48] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [19:50:01] PROBLEM - Host cloud13 is DOWN: PING CRITICAL - Packet loss = 100% [19:50:42] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 45% [19:50:53] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 53% [19:50:59] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: CRITICAL - NGINX Error Rate is 82% [19:51:29] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 0.76, 2.07, 3.38 [19:51:49] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is WARNING: WARNING - NGINX Error Rate is 53% [19:53:04] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [19:53:08] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 23% [19:53:49] RECOVERY - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is OK: OK - NGINX Error Rate is 21% [19:54:50] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 36% [19:57:11] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: CRITICAL - NGINX Error Rate is 85% [19:57:12] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 37% [19:57:57] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 74% [19:59:15] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 58% [19:59:58] RECOVERY - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is OK: OK - NGINX Error Rate is 29% [20:01:19] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: CRITICAL - NGINX Error Rate is 84% [20:02:49] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 88 Threads: 9 Questions: 57 Slow queries: 10 Opens: 80 Open tables: 74 Queries per second avg: 0.647 [20:03:06] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 53% [20:04:06] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is WARNING: WARNING - NGINX Error Rate is 45% [20:05:08] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [20:05:38] PROBLEM - es141 Puppet on es141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Elasticsearch_template[graylog-internal] [20:07:05] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 58% [20:07:08] RECOVERY - Host cloud13 is UP: PING OK - Packet loss = 0%, RTA = 0.56 ms [20:07:08] RECOVERY - Host es131 is UP: PING WARNING - Packet loss = 80%, RTA = 378.18 ms [20:07:09] !log shutdown all VMs on cloud13 [20:07:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.42, 2.82, 2.20 [20:07:24] PROBLEM - es131 NTP time on es131 is WARNING: NTP WARNING: Offset -0.2921687663 secs [20:07:25] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 57% [20:07:29] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [20:07:33] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 30902 bytes in 0.255 second response time [20:07:41] RECOVERY - Host mw131 is UP: PING OK - Packet loss = 0%, RTA = 0.89 ms [20:07:43] !log shutdown all VMs on cloud13 [20:07:44] RECOVERY - Host test131 is UP: PING OK - Packet loss = 0%, RTA = 1.53 ms [20:07:47] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 30902 bytes in 2.485 second response time [20:08:03] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 30902 bytes in 0.217 second response time [20:08:07] RECOVERY - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is OK: OK - NGINX Error Rate is 25% [20:08:10] RECOVERY - ping6 on es131 is OK: PING OK - Packet loss = 0%, RTA = 1.42 ms [20:08:10] RECOVERY - Host mw132 is UP: PING OK - Packet loss = 0%, RTA = 2.15 ms [20:08:14] RECOVERY - es131 SSH on es131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:08:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:08:20] RECOVERY - Host mem131 is UP: PING OK - Packet loss = 0%, RTA = 0.72 ms [20:08:23] RECOVERY - Host prometheus131 is UP: PING OK - Packet loss = 0%, RTA = 0.81 ms [20:08:27] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 35094 bytes in 6.383 second response time [20:08:29] RECOVERY - cloud13 SSH on cloud13 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:08:30] RECOVERY - ping6 on cloud13 is OK: PING OK - Packet loss = 0%, RTA = 1.15 ms [20:08:30] RECOVERY - mw131 SSH on mw131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:08:36] RECOVERY - Host matomo131 is UP: PING OK - Packet loss = 0%, RTA = 2.72 ms [20:08:37] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 30902 bytes in 0.315 second response time [20:08:39] PROBLEM - test131 Check Gluster Clients on test131 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [20:08:40] RECOVERY - es131 Disk Space on es131 is OK: DISK OK - free space: / 136534 MB (68% inode=99%); [20:08:41] RECOVERY - prometheus131 conntrack_table_size on prometheus131 is OK: OK: nf_conntrack is 0 % full [20:08:43] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 30903 bytes in 2.575 second response time [20:08:47] RECOVERY - Host db131 is UP: PING OK - Packet loss = 0%, RTA = 0.81 ms [20:08:47] RECOVERY - db131 conntrack_table_size on db131 is OK: OK: nf_conntrack is 0 % full [20:08:48] RECOVERY - ping6 on mw131 is OK: PING OK - Packet loss = 0%, RTA = 1.47 ms [20:08:48] RECOVERY - matomo131 conntrack_table_size on matomo131 is OK: OK: nf_conntrack is 0 % full [20:08:49] PROBLEM - mw131 Check Gluster Clients on mw131 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [20:08:50] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 30903 bytes in 1.505 second response time [20:08:52] RECOVERY - es131 ferm_active on es131 is OK: OK ferm input default policy is set [20:08:55] RECOVERY - ping6 on db131 is OK: PING OK - Packet loss = 0%, RTA = 1.45 ms [20:08:58] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 30910 bytes in 0.261 second response time [20:09:00] PROBLEM - db131 NTP time on db131 is WARNING: NTP WARNING: Offset 0.277982235 secs [20:09:01] RECOVERY - ping6 on test131 is OK: PING OK - Packet loss = 0%, RTA = 0.86 ms [20:09:04] !log reboot cloud13 [20:09:09] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [20:09:09] RECOVERY - test131 HTTPS on test131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 549 bytes in 0.011 second response time [20:09:10] RECOVERY - mw132 HTTPS on mw132 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.007 second response time [20:09:12] RECOVERY - mem131 memcached on mem131 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:405 port 11211 [20:09:13] RECOVERY - es131 Puppet on es131 is OK: OK: Puppet is currently enabled, last run 42 minutes ago with 0 failures [20:09:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.16, 2.18, 2.03 [20:09:23] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 11% [20:09:25] RECOVERY - ping6 on matomo131 is OK: PING OK - Packet loss = 0%, RTA = 0.85 ms [20:09:25] RECOVERY - test131 SSH on test131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:09:27] RECOVERY - es131 NTP time on es131 is OK: NTP OK: Offset 0.001880079508 secs [20:09:27] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 49% [20:09:30] RECOVERY - ping6 on mw132 is OK: PING OK - Packet loss = 0%, RTA = 2.61 ms [20:09:34] PROBLEM - mw132 Check Gluster Clients on mw132 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [20:09:45] RECOVERY - ping6 on mem131 is OK: PING OK - Packet loss = 0%, RTA = 0.81 ms [20:09:45] RECOVERY - matomo131 SSH on matomo131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:09:55] RECOVERY - prometheus131 Prometheus on prometheus131 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:402 port 9090 [20:09:59] RECOVERY - prometheus131 SSH on prometheus131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:10:10] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 0.89, 0.54, 0.22 [20:10:15] RECOVERY - matomo131 HTTPS on matomo131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.214 second response time [20:10:20] RECOVERY - db131 SSH on db131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:10:20] RECOVERY - prometheus131 Disk Space on prometheus131 is OK: DISK OK - free space: / 46988 MB (35% inode=98%); [20:10:20] RECOVERY - ping6 on prometheus131 is OK: PING OK - Packet loss = 0%, RTA = 1.28 ms [20:10:25] PROBLEM - db131 MariaDB on db131 is CRITICAL: Can't connect to MySQL server on 'db131.miraheze.org' (115) [20:10:30] RECOVERY - matomo131 Redis Process on matomo131 is OK: PROCS OK: 1 process with args 'redis-server' [20:10:59] RECOVERY - db131 NTP time on db131 is OK: NTP OK: Offset -0.003264158964 secs [20:11:25] !log shutdown all VMs on cloud12 [20:11:32] RECOVERY - mw132 Check Gluster Clients on mw132 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [20:12:30] PROBLEM - gluster121 Disk Space on gluster121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:12:53] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:13:17] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 952 bytes in 3.251 second response time [20:13:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:13:40] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [20:13:58] RECOVERY - mw131 Check Gluster Clients on mw131 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [20:14:32] PROBLEM - graylog121 HTTPS on graylog121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:14:40] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 73% [20:14:54] PROBLEM - ping6 on gluster121 is CRITICAL: PING CRITICAL - Packet loss = 100% [20:14:57] PROBLEM - bast121 SSH on bast121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:02] !log reboot cloud12 [20:15:02] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:10] PROBLEM - gluster122 SSH on gluster122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:13] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [20:15:14] The bot disconnected CosmicAlpha [20:15:23] I know. [20:15:28] PROBLEM - ping6 on gluster122 is CRITICAL: PING CRITICAL - Packet loss = 100% [20:15:29] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:35] PROBLEM - es131 Puppet on es131 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 6 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[elasticsearch],Elasticsearch_template[graylog-internal] [20:15:45] PROBLEM - jobchron121 SSH on jobchron121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:45] PROBLEM - ping6 on cp32 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 106.23 ms [20:15:48] PROBLEM - mail121 webmail.miraheze.org HTTPS on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:16:03] !log apt upgrade on cloud14 and all cloud14 VMs (includes kernel upgrades) [20:16:03] PROBLEM - bast121 NTP time on bast121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:16:06] PROBLEM - bast121 Puppet on bast121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:16:20] PROBLEM - mail121 IMAP on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:16:24] PROBLEM - mw122 HTTPS on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:16:30] PROBLEM - ping6 on graylog121 is CRITICAL: PING CRITICAL - Packet loss = 100% [20:16:36] PROBLEM - ping6 on jobchron121 is CRITICAL: PING CRITICAL - Packet loss = 100% [20:16:37] PROBLEM - graylog121 SSH on graylog121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:16:37] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:17:21] PROBLEM - mw121 Disk Space on mw121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. PROBLEM - Host graylog121 is DOWN: PING CRITICAL - Packet loss = 100% PROBLEM - graylog121 APT on graylog121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:17:48] PROBLEM - mw122 PowerDNS Recursor on mw122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. PROBLEM - mw121 APT on mw121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:18:10] PROBLEM - mw121 APT on mw121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:18:11] PROBLEM - gluster122 PowerDNS Recursor on gluster122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:18:11] PROBLEM - mw122 conntrack_table_size on mw122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:19:17] PROBLEM - Host phab121 is DOWN: CRITICAL - Plugin timed out after 30 seconds [20:19:17] PROBLEM - Host ns2 is DOWN: PING CRITICAL - Packet loss = 100% [20:19:17] PROBLEM - Host phab121 is DOWN: CRITICAL - Plugin timed out after 30 seconds [20:19:17] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [20:19:28] PROBLEM - db121 MariaDB on db121 is UNKNOWN: PROBLEM - db121 MariaDB on db121 is UNKNOWN: [20:19:30] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.021 second response time [20:19:38] PROBLEM - ping6 on reports121 is CRITICAL: PING CRITICAL - Packet loss = 100% [20:19:40] PROBLEM - gluster121 SSH on gluster121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:19:47] PROBLEM - Host gluster121 is DOWN: PING CRITICAL - Packet loss = 100% [20:19:49] PROBLEM - mail121 HTTPS on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:19:51] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 56% [20:19:54] PROBLEM - ping6 on db121 is CRITICAL: PING CRITICAL - Packet loss = 100% [20:19:58] PROBLEM - Host gluster122 is DOWN: PING CRITICAL - Packet loss = 100% [20:20:08] PROBLEM - db121 SSH on db121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:20:08] PROBLEM - reports121 SSH on reports121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds PROBLEM - reports121 HTTPS on reports121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:20:27] PROBLEM - Host db121 is DOWN: PING CRITICAL - Packet loss = 100% [20:20:28] PROBLEM - Host bast121 is DOWN: PING CRITICAL - Packet loss = 100% [20:20:43] PROBLEM - mail121 Current Load on mail121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. PROBLEM - jobchron121 Current Load on jobchron121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:20:43] PROBLEM - mw121 ferm_active on mw121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:21:05] !log start all VMs on cloud12 [20:21:06] PROBLEM - jobchron121 PowerDNS Recursor on jobchron121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. PROBLEM - jobchron121 Disk Space on jobchron121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:21:09] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 22% PROBLEM - Host jobchron121 is DOWN: CRITICAL - Plugin timed out after 30 seconds PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:21:27] PROBLEM - mail121 SMTP on mail121 is CRITICAL: connect to address 2a10:6740::6:307 and port 25: Connection refusedSMTP CRITICAL - 1.517 sec. response time PROBLEM - mail121 APT on mail121 is CRITICAL: connect to address 2a10:6740::6:307 port 5666: Connection refusedconnect to host 2a10:6740::6:307 port 5666: Connection refused PROBLEM - mail121 PowerDNS Recursor on mail121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 se [20:21:28] PROBLEM - mail121 NTP time on mail121 is WARNING: NTP WARNING: Offset 0.4584085047 secs PROBLEM - wikileague.cf - reverse DNS on sslhost is UNKNOWN: [20:21:32] PROBLEM - mw122 APT on mw122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:21:33] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 30902 bytes in 0.509 second response time [20:21:46] RECOVERY - mail121 webmail.miraheze.org HTTPS on mail121 is OK: HTTP OK: HTTP/1.1 200 OK - 6281 bytes in 3.503 second response time PROBLEM - ping6 on cp32 is WARNING: PING WARNING - Packet loss = 0%, RTA = 106.19 ms [20:21:46] PROBLEM - agentisai.com - reverse DNS on sslhost is UNKNOWN: [20:22:05] !log start all VMs on cloud13 [20:22:42] RECOVERY - mail121 IMAP on mail121 is OK: IMAP OK - 0.002 second response time on 2a10:6740::6:307 port 143 [* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ STARTTLS LOGINDISABLED] Dovecot (Debian) ready.] [20:22:47] PROBLEM - mw122 Check Gluster Clients on mw122 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [20:22:52] RECOVERY - mw122 HTTPS on mw122 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.721 second response time [20:23:54] PROBLEM - mw121 Check Gluster Clients on mw121 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' RECOVERY - mw122 PowerDNS Recursor on mw122 is OK: DNS OK: 0.078 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 RECOVERY - mw121 Disk Space on mw121 is OK: DISK OK - free space: / 10022 MB (45% inode=80%); [20:23:56] RECOVERY - mw121 APT on mw121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:27:36] !log start all VMs on cloud13 [20:27:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:28:06] !log shutdown all VMs on cloud12 [20:28:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:28:15] !log reboot cloud12 [20:28:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:28:33] !log start all VMs on cloud12 [20:28:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:28:58] !log apt upgrade on cloud14 and all cloud14 VMs (includes kernel upgrades) [20:29:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:30:02] !log shutdown all VMs on cloud14 [20:30:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:30:09] !log reboot cloud14 [20:30:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:44:35] RECOVERY - Host gluster122 is UP: PING OK - Packet loss = 0%, RTA = 7.50 ms [20:44:38] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 30903 bytes in 1.485 second response time [20:44:44] !log start all VMs on cloud14 [20:44:45] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.47, 4.31, 3.33 [20:44:46] RECOVERY - reports121 HTTPS on reports121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.217 second response time [20:44:47] RECOVERY - gluster122 PowerDNS Recursor on gluster122 is OK: DNS OK: 0.647 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [20:44:48] RECOVERY - mail121 PowerDNS Recursor on mail121 is OK: DNS OK: 0.725 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [20:44:51] PROBLEM - mail121 Puppet on mail121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:44:51] RECOVERY - mw121 Disk Space on mw121 is OK: DISK OK - free space: / 9652 MB (43% inode=80%); [20:44:53] RECOVERY - Host bast121 is UP: PING OK - Packet loss = 0%, RTA = 1.73 ms [20:44:58] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:45:01] RECOVERY - bast121 NTP time on bast121 is OK: NTP OK: Offset -0.001971930265 secs [20:45:02] RECOVERY - test131 Check Gluster Clients on test131 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [20:45:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:45:09] RECOVERY - mw121 APT on mw121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:45:13] RECOVERY - mw122 HTTPS on mw122 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.224 second response time [20:45:13] RECOVERY - Host gluster121 is UP: PING OK - Packet loss = 0%, RTA = 7.24 ms [20:45:18] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is WARNING: HTTP WARNING: HTTP/1.1 401 Unauthorized - 476 bytes in 0.096 second response time [20:45:21] RECOVERY - Host jobchron121 is UP: PING OK - Packet loss = 0%, RTA = 7.31 ms [20:45:24] RECOVERY - Host graylog121 is UP: PING OK - Packet loss = 0%, RTA = 3.70 ms [20:45:24] RECOVERY - jobchron121 Disk Space on jobchron121 is OK: DISK OK - free space: / 5697 MB (64% inode=84%); [20:45:25] RECOVERY - ping6 on gluster121 is OK: PING OK - Packet loss = 0%, RTA = 2.49 ms [20:45:25] RECOVERY - mw122 conntrack_table_size on mw122 is OK: OK: nf_conntrack is 3 % full [20:45:26] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:45:27] RECOVERY - Host ns2 is UP: PING OK - Packet loss = 0%, RTA = 2.44 ms [20:45:31] !log SET GLOBAL read_only = 0; on db121, db131, and db141 [20:45:32] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 30902 bytes in 0.313 second response time [20:45:32] RECOVERY - mail121 HTTPS on mail121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.089 second response time [20:45:32] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 30903 bytes in 3.524 second response time [20:45:34] RECOVERY - graylog121 SSH on graylog121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:45:35] RECOVERY - mail121 IMAP on mail121 is OK: IMAP OK - 0.064 second response time on 2a10:6740::6:307 port 143 [* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ STARTTLS LOGINDISABLED] Dovecot (Debian) ready.] [20:45:35] RECOVERY - mw122 APT on mw122 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:45:43] RECOVERY - mail121 Current Load on mail121 is OK: OK - load average: 0.13, 0.14, 0.13 [20:45:46] PROBLEM - gluster122 Puppet on gluster122 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:45:46] RECOVERY - jobchron121 PowerDNS Recursor on jobchron121 is OK: DNS OK: 0.292 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [20:45:49] RECOVERY - mail121 SMTP on mail121 is OK: SMTP OK - 0.224 sec. response time [20:45:52] RECOVERY - reports121 SSH on reports121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:45:54] RECOVERY - jobchron121 Current Load on jobchron121 is OK: OK - load average: 0.34, 0.15, 0.10 [20:45:55] RECOVERY - gluster122 SSH on gluster122 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:45:56] RECOVERY - ping6 on reports121 is OK: PING OK - Packet loss = 0%, RTA = 8.26 ms [20:45:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:46:01] RECOVERY - mw121 ferm_active on mw121 is OK: OK ferm input default policy is set [20:46:02] PROBLEM - gluster121 Disk Space on gluster121 is WARNING: DISK WARNING - free space: / 103111 MB (10% inode=81%); [20:46:03] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 9% [20:46:04] RECOVERY - graylog121 APT on graylog121 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [20:46:06] RECOVERY - mw122 PowerDNS Recursor on mw122 is OK: DNS OK: 0.303 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [20:46:06] RECOVERY - ping6 on gluster122 is OK: PING OK - Packet loss = 0%, RTA = 6.33 ms [20:46:08] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 35093 bytes in 0.347 second response time [20:46:09] RECOVERY - ping6 on graylog121 is OK: PING OK - Packet loss = 0%, RTA = 3.32 ms [20:46:11] RECOVERY - bast121 SSH on bast121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:46:15] RECOVERY - mail121 NTP time on mail121 is OK: NTP OK: Offset 0.00269934535 secs [20:46:16] RECOVERY - Host db121 is UP: PING OK - Packet loss = 0%, RTA = 2.52 ms [20:46:17] RECOVERY - mail121 APT on mail121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:46:18] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 23 minutes ago with 0 failures [20:46:20] RECOVERY - gluster121 SSH on gluster121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:46:23] RECOVERY - Host phab121 is UP: PING OK - Packet loss = 0%, RTA = 4.35 ms [20:46:23] PROBLEM - mon141 Current Load on mon141 is CRITICAL: CRITICAL - load average: 28.42, 11.11, 4.10 [20:46:25] RECOVERY - ping6 on jobchron121 is OK: PING OK - Packet loss = 0%, RTA = 3.40 ms [20:46:28] PROBLEM - mon141 Puppet on mon141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:46:28] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 30902 bytes in 0.503 second response time [20:46:36] RECOVERY - ping6 on db121 is OK: PING OK - Packet loss = 0%, RTA = 1.95 ms [20:46:46] RECOVERY - gluster121 Puppet on gluster121 is OK: OK: Puppet is currently enabled, last run 19 minutes ago with 0 failures [20:46:46] RECOVERY - db121 SSH on db121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:46:56] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 1575 Threads: 28 Questions: 1060467 Slow queries: 110844 Opens: 9377 Open tables: 9371 Queries per second avg: 673.312 [20:47:16] RECOVERY - jobchron121 SSH on jobchron121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:47:21] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:47:21] PROBLEM - mwtask141 Check Gluster Clients on mwtask141 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [20:47:21] PROBLEM - cp23 Puppet on cp23 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:47:21] PROBLEM - mw131 Puppet on mw131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:47:22] PROBLEM - bast101 Puppet on bast101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:47:22] PROBLEM - cp22 Puppet on cp22 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:47:22] PROBLEM - gluster101 Puppet on gluster101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:47:22] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:47:22] PROBLEM - swiftproxy111 Puppet on swiftproxy111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:47:23] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:47:23] PROBLEM - swiftobject113 Puppet on swiftobject113 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:47:24] PROBLEM - matomo131 Puppet on matomo131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:47:40] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Puppet has 77 failures. Last run 16 minutes ago with 77 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/fid.koyomi-ya.net.crt],File[/etc/ssl/private/fid.koyomi-ya.net.key],File[/etc/ssl/localcerts/sona.pona.la.crt],File[/etc/ssl/private/sona.pona.la.key] [20:47:43] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:47:45] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:47:50] PROBLEM - cloud14 Puppet on cloud14 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:47:52] PROBLEM - cp33 Puppet on cp33 is CRITICAL: CRITICAL: Puppet has 374 failures. Last run 20 minutes ago with 374 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/wmworld.sktz.live.crt],File[/etc/ssl/private/wmworld.sktz.live.key],File[/etc/ssl/localcerts/wiki.umbranet.mandelbrotgames.com.crt],File[/etc/ssl/private/wiki.umbranet.mandelbrotgames.com.key] [20:47:54] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 4.17, 2.31, 0.96 [20:47:57] PROBLEM - mem131 Puppet on mem131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:48:00] PROBLEM - swiftobject112 Puppet on swiftobject112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:48:11] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:48:12] PROBLEM - swiftobject111 Puppet on swiftobject111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:48:15] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [20:48:16] PROBLEM - phab121 Puppet on phab121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:48:23] PROBLEM - cp32 Puppet on cp32 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:48:24] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [20:48:38] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [20:49:23] RECOVERY - mw142 JobRunner Service on mw142 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [20:49:39] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:49:48] RECOVERY - puppet141 Current Load on puppet141 is OK: OK - load average: 2.13, 2.25, 1.10 [20:50:01] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [20:50:18] RECOVERY - mon141 Puppet on mon141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:50:56] PROBLEM - agentisai.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for agentisai.com could not be found [20:51:11] PROBLEM - wikileague.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikileague.cf could not be found [20:51:37] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [20:52:20] PROBLEM - mon141 Current Load on mon141 is WARNING: WARNING - load average: 0.65, 3.88, 3.02 [20:52:49] !log shutdown all VMs on cloud13 [20:53:35] PROBLEM - db131 MariaDB on db131 is UNKNOWN: [20:54:48] RECOVERY - mon141 Current Load on mon141 is OK: OK - load average: 0.66, 2.60, 2.66 [20:56:07] PROBLEM - matomo131 SSH on matomo131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:56:13] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:56:22] PROBLEM - ping6 on mem131 is CRITICAL: PING CRITICAL - Packet loss = 100% [20:56:23] PROBLEM - ping6 on mw132 is CRITICAL: PING CRITICAL - Packet loss = 100% [20:56:30] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:56:36] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:56:47] PROBLEM - prometheus131 Prometheus on prometheus131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:56:53] PROBLEM - ping6 on mw131 is CRITICAL: PING CRITICAL - Packet loss = 100% [20:56:55] RECOVERY - cp33 Puppet on cp33 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [20:57:02] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:57:06] PROBLEM - es131 SSH on es131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:10] PROBLEM - prometheus131 SSH on prometheus131 is CRITICAL: connect to address 2a10:6740::6:402 and port 22: No route to host [20:57:11] PROBLEM - db131 APT on db131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:57:15] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:17] PROBLEM - db131 NTP time on db131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:57:18] PROBLEM - Host mw131 is DOWN: PING CRITICAL - Packet loss = 100% [20:57:25] PROBLEM - test131 HTTPS on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:26] PROBLEM - mw132 nutcracker process on mw132 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:57:30] PROBLEM - mw132 conntrack_table_size on mw132 is CRITICAL: connect to address 2a10:6740::6:404 port 5666: No route to hostconnect to host 2a10:6740::6:404 port 5666: No route to host [20:57:30] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: connect to address 2a10:6740::6:404 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [20:57:30] PROBLEM - Host mw132 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:404) [20:57:36] PROBLEM - matomo131 HTTPS on matomo131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:40] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 40% [20:57:40] PROBLEM - ping6 on prometheus131 is CRITICAL: PING CRITICAL - Packet loss = 100% [20:57:42] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [20:57:46] PROBLEM - mem131 SSH on mem131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:47] PROBLEM - db131 SSH on db131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:57] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [20:58:00] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:04] PROBLEM - ping6 on matomo131 is CRITICAL: PING CRITICAL - Packet loss = 100% [20:58:07] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:10] PROBLEM - ping6 on es131 is CRITICAL: PING CRITICAL - Packet loss = 100% [20:58:15] PROBLEM - Host test131 is DOWN: PING CRITICAL - Packet loss = 100% [20:58:16] PROBLEM - Host prometheus131 is DOWN: PING CRITICAL - Packet loss = 100% [20:58:17] RECOVERY - cp23 Puppet on cp23 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:58:18] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [20:58:26] PROBLEM - ping6 on db131 is CRITICAL: PING CRITICAL - Packet loss = 100% [20:58:28] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:33] PROBLEM - Host matomo131 is DOWN: PING CRITICAL - Packet loss = 100% [20:58:38] PROBLEM - es131 Current Load on es131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:58:41] PROBLEM - mem131 memcached on mem131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:47] PROBLEM - db131 Disk Space on db131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:58:59] PROBLEM - Host es131 is DOWN: PING CRITICAL - Packet loss = 100% [20:59:08] PROBLEM - ping6 on cloud13 is CRITICAL: PING CRITICAL - Packet loss = 100% [20:59:11] PROBLEM - mem131 ferm_active on mem131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:59:15] PROBLEM - mem131 PowerDNS Recursor on mem131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:59:21] PROBLEM - Host mem131 is DOWN: PING CRITICAL - Packet loss = 100% [20:59:22] RECOVERY - db112 NTP time on db112 is OK: NTP OK: Offset 0.02980333567 secs [20:59:23] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 53% [20:59:33] PROBLEM - db131 PowerDNS Recursor on db131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:59:48] PROBLEM - cloud13 ferm_active on cloud13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:59:50] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [20:59:58] PROBLEM - db131 Current Load on db131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:00:00] RECOVERY - db141 MariaDB on db141 is OK: Uptime: 94 Threads: 16 Questions: 498 Slow queries: 35 Opens: 176 Open tables: 170 Queries per second avg: 5.297 [21:00:02] PROBLEM - db131 Puppet on db131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:00:07] PROBLEM - cloud13 SSH on cloud13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:09] PROBLEM - Host db131 is DOWN: PING CRITICAL - Packet loss = 100% [21:00:22] PROBLEM - Host cloud13 is DOWN: PING CRITICAL - Packet loss = 100% [21:00:57] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 51% [21:01:23] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [21:03:04] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [21:03:05] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:03:23] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 28% [21:03:24] RECOVERY - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is OK: OK - NGINX Error Rate is 32% [21:04:06] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [21:04:07] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [21:04:24] RECOVERY - bast101 Puppet on bast101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:05:07] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 36% [21:05:46] RECOVERY - swiftobject111 Puppet on swiftobject111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:06:04] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 20% [21:07:02] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [21:07:22] RECOVERY - Host cloud13 is UP: PING OK - Packet loss = 0%, RTA = 0.60 ms [21:07:27] RECOVERY - Host db131 is UP: PING OK - Packet loss = 78%, RTA = 1.83 ms [21:07:28] RECOVERY - Host mw131 is UP: PING OK - Packet loss = 73%, RTA = 202.27 ms [21:07:33] RECOVERY - ping6 on db131 is OK: PING OK - Packet loss = 0%, RTA = 0.79 ms [21:07:39] RECOVERY - Host mw132 is UP: PING OK - Packet loss = 50%, RTA = 0.97 ms [21:07:40] RECOVERY - ping6 on mw132 is OK: PING OK - Packet loss = 0%, RTA = 0.73 ms [21:07:41] RECOVERY - ping6 on cloud13 is OK: PING OK - Packet loss = 0%, RTA = 1.22 ms [21:07:42] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 30903 bytes in 3.991 second response time [21:07:48] RECOVERY - cloud13 ferm_active on cloud13 is OK: OK ferm input default policy is set [21:07:48] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:08:02] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 0.83, 0.22, 0.08 [21:08:05] !log shutdown all VMs on cloud13 [21:08:06] RECOVERY - db131 Puppet on db131 is OK: OK: Puppet is currently enabled, last run 38 minutes ago with 0 failures [21:08:07] RECOVERY - ping6 on mw131 is OK: PING OK - Packet loss = 0%, RTA = 0.87 ms [21:08:08] RECOVERY - Host test131 is UP: PING OK - Packet loss = 0%, RTA = 1.76 ms [21:08:15] RECOVERY - Host matomo131 is UP: PING OK - Packet loss = 0%, RTA = 0.55 ms [21:08:19] RECOVERY - Host prometheus131 is UP: PING OK - Packet loss = 0%, RTA = 0.56 ms [21:08:23] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 30894 bytes in 0.306 second response time [21:08:23] RECOVERY - test131 HTTPS on test131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 549 bytes in 0.010 second response time [21:08:25] RECOVERY - db131 MariaDB on db131 is OK: Uptime: 63 Threads: 19 Questions: 2189 Slow queries: 176 Opens: 336 Open tables: 330 Queries per second avg: 34.746 [21:08:26] PROBLEM - mw132 Check Gluster Clients on mw132 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [21:08:27] RECOVERY - cloud13 SSH on cloud13 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [21:08:31] RECOVERY - db131 APT on db131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:08:35] RECOVERY - Host es131 is UP: PING OK - Packet loss = 0%, RTA = 1.07 ms [21:08:41] RECOVERY - Host mem131 is UP: PING OK - Packet loss = 0%, RTA = 0.57 ms [21:08:41] RECOVERY - mw132 HTTPS on mw132 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.011 second response time [21:08:48] RECOVERY - db131 SSH on db131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [21:08:50] RECOVERY - ping6 on prometheus131 is OK: PING OK - Packet loss = 0%, RTA = 0.58 ms [21:08:56] RECOVERY - mem131 SSH on mem131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [21:08:59] RECOVERY - ping6 on matomo131 is OK: PING OK - Packet loss = 0%, RTA = 0.73 ms [21:09:00] RECOVERY - db131 NTP time on db131 is OK: NTP OK: Offset 0.004002094269 secs [21:09:01] RECOVERY - matomo131 HTTPS on matomo131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.207 second response time [21:09:02] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 30894 bytes in 5.660 second response time [21:09:04] RECOVERY - mw132 nutcracker process on mw132 is OK: PROCS OK: 1 process with UID = 115 (nutcracker), command name 'nutcracker' [21:09:08] RECOVERY - mw132 conntrack_table_size on mw132 is OK: OK: nf_conntrack is 0 % full [21:09:15] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 35094 bytes in 3.553 second response time [21:09:16] PROBLEM - mw131 Check Gluster Clients on mw131 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [21:09:19] !log shutdown all VMs on cloud13 [21:09:21] RECOVERY - db131 PowerDNS Recursor on db131 is OK: DNS OK: 0.050 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [21:09:21] RECOVERY - db131 Disk Space on db131 is OK: DISK OK - free space: / 173981 MB (73% inode=98%); [21:09:22] RECOVERY - ping6 on es131 is OK: PING OK - Packet loss = 0%, RTA = 0.82 ms [21:09:31] PROBLEM - test131 Check Gluster Clients on test131 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [21:09:32] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 30895 bytes in 1.173 second response time [21:09:38] RECOVERY - ping6 on mem131 is OK: PING OK - Packet loss = 0%, RTA = 0.49 ms [21:09:38] RECOVERY - es131 Current Load on es131 is OK: OK - load average: 0.45, 0.34, 0.14 [21:09:40] RECOVERY - mem131 memcached on mem131 is OK: TCP OK - 0.000 second response time on 2a10:6740::6:405 port 11211 [21:09:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:09:41] PROBLEM - mw131 JobRunner Service on mw131 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [21:09:41] RECOVERY - matomo131 SSH on matomo131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [21:09:46] PROBLEM - test131 JobRunner Service on test131 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [21:09:47] !log start all VMs on cloud13 [21:09:51] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 30894 bytes in 0.299 second response time [21:09:58] RECOVERY - mem131 ferm_active on mem131 is OK: OK ferm input default policy is set [21:09:58] RECOVERY - mem131 PowerDNS Recursor on mem131 is OK: DNS OK: 0.091 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [21:10:11] RECOVERY - prometheus131 SSH on prometheus131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [21:10:16] RECOVERY - prometheus131 Prometheus on prometheus131 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:402 port 9090 [21:10:21] RECOVERY - es131 SSH on es131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [21:10:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:10:35] RECOVERY - bast121 Puppet on bast121 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [21:10:41] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [21:10:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:10:44] !log reboot db141 [21:10:54] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [21:11:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:11:10] RECOVERY - matomo131 Puppet on matomo131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:11:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.65, 7.97, 4.62 [21:11:15] RECOVERY - mw131 Check Gluster Clients on mw131 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [21:11:18] RECOVERY - gluster122 Puppet on gluster122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:11:19] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:11:24] RECOVERY - mem131 Puppet on mem131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:11:36] !log SET GLOBAL read_only = 0; on db131 and db141 [21:11:37] RECOVERY - mw131 JobRunner Service on mw131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [21:11:41] RECOVERY - test131 JobRunner Service on test131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [21:11:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:11:46] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 30894 bytes in 0.308 second response time [21:12:02] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 12.46, 7.39, 3.65 [21:12:12] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [21:12:15] RECOVERY - mw132 Check Gluster Clients on mw132 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [21:12:17] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [21:12:27] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 4.25, 3.12, 1.90 [21:12:33] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [21:12:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.56, 2.80, 1.96 [21:12:43] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:12:50] RECOVERY - swiftproxy111 Puppet on swiftproxy111 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [21:12:50] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [21:12:56] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:13:02] RECOVERY - swiftobject113 Puppet on swiftobject113 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [21:13:06] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:13:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.45, 8.39, 5.21 [21:13:19] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [21:13:34] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:14:02] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 8.93, 7.64, 4.20 [21:14:21] RECOVERY - puppet141 Current Load on puppet141 is OK: OK - load average: 1.82, 2.59, 1.85 [21:14:45] RECOVERY - cp22 Puppet on cp22 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:15:18] RECOVERY - swiftobject112 Puppet on swiftobject112 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [21:15:59] !log run puppet on all servers [21:16:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:16:08] RECOVERY - gluster101 Puppet on gluster101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:16:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.53, 3.02, 2.27 [21:17:05] !log mount /mnt/mediawiki-static on mwtask141 and test131 [21:17:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:17:16] RECOVERY - mwtask141 Check Gluster Clients on mwtask141 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [21:17:20] RECOVERY - test131 Check Gluster Clients on test131 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [21:20:06] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 4.93, 3.75, 2.53 [21:24:58] PROBLEM - cp32 APT on cp32 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [21:26:56] RECOVERY - cp32 APT on cp32 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:30:06] PROBLEM - puppet141 Current Load on puppet141 is WARNING: WARNING - load average: 1.46, 3.45, 3.25 [21:32:06] RECOVERY - puppet141 Current Load on puppet141 is OK: OK - load average: 0.99, 2.65, 2.98 [21:57:02] PROBLEM - cp33 APT on cp33 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [21:58:59] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:11:53] RECOVERY - cloud14 IPMI Sensors on cloud14 is OK: IPMI Status: OK [22:14:39] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [22:14:40] [url] Page not found · GitHub · GitHub | github.com [22:14:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/66c57bc60e9c [22:14:42] [02miraheze/puppet] 07paladox 0366c57bc - elasticsearch: set cluster.initial_master_nodes [22:14:44] [02puppet] 07paladox opened pull request 03#2876: elasticsearch: set cluster.initial_master_nodes - 13https://github.com/miraheze/puppet/pull/2876 [22:14:45] [url] Page not found · GitHub · GitHub | github.com [22:17:39] [02puppet] 07paladox synchronize pull request 03#2876: elasticsearch: set cluster.initial_master_nodes - 13https://github.com/miraheze/puppet/pull/2876 [22:17:40] [url] Page not found · GitHub · GitHub | github.com [22:17:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/66c57bc60e9c...343562dbc9d6 [22:17:41] [url] Comparing 66c57bc60e9c...343562dbc9d6 · miraheze/puppet · GitHub | github.com [22:17:42] [02miraheze/puppet] 07paladox 03343562d - Update elasticsearch.pp [22:18:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/343562dbc9d6...84903269f3f0 [22:18:09] [url] Comparing 343562dbc9d6...84903269f3f0 · miraheze/puppet · GitHub | github.com [22:18:10] [02miraheze/puppet] 07paladox 038490326 - Update es131.yaml [22:18:11] [02puppet] 07paladox synchronize pull request 03#2876: elasticsearch: set cluster.initial_master_nodes - 13https://github.com/miraheze/puppet/pull/2876 [22:18:12] ... [22:18:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/84903269f3f0...a88f77f8a538 [22:18:20] [url] Comparing 84903269f3f0...a88f77f8a538 · miraheze/puppet · GitHub | github.com [22:18:21] [02miraheze/puppet] 07paladox 03a88f77f - Update es141.yaml [22:18:22] [02puppet] 07paladox synchronize pull request 03#2876: elasticsearch: set cluster.initial_master_nodes - 13https://github.com/miraheze/puppet/pull/2876 [22:18:23] [url] Page not found · GitHub · GitHub | github.com [22:25:59] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.60, 3.14, 2.13 [22:27:58] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.81, 2.58, 2.04 [22:48:22] PROBLEM - es131 Puppet on es131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 minutes ago with 1 failures [22:58:39] [02puppet] 07paladox synchronize pull request 03#2876: elasticsearch: set cluster.initial_master_nodes - 13https://github.com/miraheze/puppet/pull/2876 [22:58:39] [url] Page not found · GitHub · GitHub | github.com [22:58:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a88f77f8a538...491c3e3e6e16 [22:58:41] [url] Comparing a88f77f8a538...491c3e3e6e16 · miraheze/puppet · GitHub | github.com [22:58:42] [02miraheze/puppet] 07paladox 03491c3e3 - Update es131.yaml [22:59:20] RECOVERY - graylog121 HTTPS on graylog121 is OK: HTTP OK: HTTP/1.1 200 OK - 1367 bytes in 0.197 second response time [22:59:40] [02puppet] 07paladox synchronize pull request 03#2876: elasticsearch: set cluster.initial_master_nodes - 13https://github.com/miraheze/puppet/pull/2876 [22:59:41] [url] Page not found · GitHub · GitHub | github.com [22:59:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/491c3e3e6e16...eb8c1869c745 [22:59:42] [url] Comparing 491c3e3e6e16...eb8c1869c745 · miraheze/puppet · GitHub | github.com [22:59:43] [02miraheze/puppet] 07paladox 03eb8c186 - Update es141.yaml [23:00:23] PROBLEM - es131 Puppet on es131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Elasticsearch_template[graylog-internal] [23:01:13] RECOVERY - es141 Puppet on es141 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:01:13] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 19.82, 8.94, 3.83 [23:01:53] PROBLEM - graylog121 APT on graylog121 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [23:02:22] RECOVERY - es131 Puppet on es131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:11:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.29, 9.75, 7.78 [23:12:18] RECOVERY - graylog121 APT on graylog121 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [23:13:30] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.61, 8.45, 7.54 [23:18:56] PROBLEM - db112 NTP time on db112 is WARNING: NTP WARNING: Offset 0.4516237378 secs [23:26:44] PROBLEM - mw131 NTP time on mw131 is WARNING: NTP WARNING: Offset 0.4508121312 secs [23:28:21] PROBLEM - mw132 NTP time on mw132 is WARNING: NTP WARNING: Offset 0.4502031505 secs [23:28:45] PROBLEM - cp33 APT on cp33 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [23:29:42] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.37, 3.92, 2.58 [23:30:43] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:31:42] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.05, 3.07, 2.43 [23:32:30] PROBLEM - mw132 NTP time on mw132 is CRITICAL: NTP CRITICAL: Offset 0.8652973771 secs [23:32:44] PROBLEM - mw131 NTP time on mw131 is CRITICAL: NTP CRITICAL: Offset 0.8656224608 secs [23:33:17] PROBLEM - db112 NTP time on db112 is CRITICAL: NTP CRITICAL: Offset 0.8652296364 secs [23:48:35] [02puppet] 07paladox closed pull request 03#2876: elasticsearch: set cluster.initial_master_nodes - 13https://github.com/miraheze/puppet/pull/2876 [23:48:35] [url] Page not found · GitHub · GitHub | github.com [23:48:36] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [23:48:38] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [23:48:38] [url] Page not found · GitHub · GitHub | github.com [23:48:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/be40d10f6313...3b9f752354ba [23:48:40] [url] Comparing be40d10f6313...3b9f752354ba · miraheze/puppet · GitHub | github.com [23:48:41] [02miraheze/puppet] 07paladox 033b9f752 - elasticsearch: set cluster.initial_master_nodes (#2876) [23:52:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.99, 3.02, 2.25 [23:54:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.30, 2.31, 2.08