[00:01:18] !log [universalomega@test131] starting deploy of {'folders': 'w/extensions/SocialProfile'} to all [00:01:19] !log [universalomega@test131] finished deploy of {'folders': 'w/extensions/SocialProfile'} to all - SUCCESS in 0s [00:01:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:01:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:02:54] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.66, 4.87, 3.53 [00:03:02] !log [universalomega@test131] starting deploy of {'folders': 'w/extensions/SocialProfile'} to all [00:03:03] !log [universalomega@test131] finished deploy of {'folders': 'w/extensions/SocialProfile'} to all - SUCCESS in 0s [00:03:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:03:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:04:34] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.75, 3.18, 1.96 [00:04:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.58, 3.91, 3.33 [00:06:33] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.60, 3.58, 2.28 [00:06:44] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.05, 3.27, 3.16 [00:08:32] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 2.08, 3.04, 2.24 [00:10:45] !log [universalomega@test131] starting deploy of {'folders': 'w/extensions/SocialProfile'} to all [00:10:46] !log [universalomega@test131] finished deploy of {'folders': 'w/extensions/SocialProfile'} to all - SUCCESS in 0s [00:10:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:10:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:15:10] !log [universalomega@test131] starting deploy of {'folders': 'w/extensions/SocialProfile'} to all [00:15:11] !log [universalomega@test131] finished deploy of {'folders': 'w/extensions/SocialProfile'} to all - SUCCESS in 0s [00:15:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:15:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:24:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.92, 3.51, 3.00 [00:26:35] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.86, 4.79, 3.53 [00:32:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.25, 11.36, 9.83 [00:32:23] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.60, 3.26, 3.30 [00:33:36] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.48, 9.99, 8.69 [00:34:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.59, 10.73, 9.79 [00:35:32] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.05, 3.15, 2.78 [00:35:36] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.84, 8.99, 8.51 [00:37:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.28, 3.12, 2.81 [00:37:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.12, 9.68, 9.61 [00:41:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.53, 10.54, 10.00 [00:43:41] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.88, 10.07, 9.90 [00:46:55] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1205218732 secs [00:57:30] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [00:58:33] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.31, 4.43, 3.14 [01:00:28] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.91, 3.95, 3.13 [01:01:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.10, 3.60, 2.64 [01:01:30] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw142 [01:02:23] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.50, 3.12, 2.92 [01:07:36] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.33, 9.30, 8.48 [01:08:55] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.07818710804 secs [01:09:36] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.50, 9.27, 8.57 [01:26:06] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.15, 5.26, 4.13 [01:27:56] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 2.90, 4.97, 3.75 [01:28:04] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 5.64, 5.25, 4.26 [01:29:53] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.96, 4.00, 3.54 [01:30:02] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.37, 4.90, 4.28 [01:31:49] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.78, 3.14, 3.27 [01:36:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.11, 10.71, 9.94 [01:39:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.32, 10.11, 9.88 [01:43:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.08, 3.02, 3.81 [01:45:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.13, 3.35, 3.82 [01:47:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.41, 3.79 [01:49:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.75, 4.24, 4.02 [01:54:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.62, 3.60, 3.83 [01:56:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.27, 10.81, 9.51 [01:57:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.43, 9.84, 9.31 [02:01:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.71, 4.85, 4.10 [02:06:32] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.12, 3.90, 3.17 [02:08:32] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.90, 3.97, 3.29 [02:10:01] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.88, 3.86, 3.37 [02:11:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.55, 3.69, 3.92 [02:11:57] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.57, 3.29, 3.21 [02:12:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.63, 3.05, 3.12 [02:13:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.57, 3.96, 3.98 [02:17:38] !log [universalomega@test131] starting deploy of {'folders': 'w/extensions/SocialProfile'} to all [02:17:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.78, 9.81, 9.15 [02:17:39] !log [universalomega@test131] finished deploy of {'folders': 'w/extensions/SocialProfile'} to all - SUCCESS in 0s [02:17:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:17:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:19:00] !log [universalomega@test131] starting deploy of {'folders': 'w/extensions/SocialProfile'} to all [02:19:01] !log [universalomega@test131] finished deploy of {'folders': 'w/extensions/SocialProfile'} to all - SUCCESS in 0s [02:19:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:19:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:19:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.35, 10.43, 9.43 [02:19:49] !log [universalomega@test131] starting deploy of {'folders': 'w/extensions/SocialProfile'} to all [02:19:50] !log [universalomega@test131] finished deploy of {'folders': 'w/extensions/SocialProfile'} to all - SUCCESS in 0s [02:19:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:19:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:21:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.29, 10.08, 9.41 [02:23:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.52, 10.00, 9.47 [02:25:36] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 10.40, 5.46, 4.05 [02:29:28] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.04, 3.35, 3.54 [02:31:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.06, 10.09, 9.61 [02:33:21] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.33, 3.61, 3.58 [02:35:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.68, 2.93, 3.34 [02:35:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.62, 9.97, 9.67 [02:37:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.55, 3.94 [02:39:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.17, 4.17, 3.40 [02:40:05] !log [void@mw121] disable approvedrevs, flaggedrevs, wikibaseclient, and wikibaserepository on politicswiki T9826 [02:40:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:41:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.94, 3.60, 3.30 [02:43:30] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.11, 2.74, 3.02 [02:45:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.62, 3.77 [02:47:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.53, 3.09, 3.55 [02:51:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.54, 2.89, 3.36 [02:59:40] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.12, 2.70, 1.66 [03:01:40] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.66, 2.22, 1.61 [03:02:40] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 74.208.203.152/cpweb, 2607:f1c0:1800:26f::1/cpweb [03:03:21] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 74.208.203.152/cpweb, 2607:f1c0:1800:26f::1/cpweb [03:03:35] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 34.20, 17.45, 8.51 [03:03:50] PROBLEM - cp33 PowerDNS Recursor on cp33 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:04:40] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:05:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:05:29] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.89, 3.72, 3.03 [03:05:48] RECOVERY - cp33 PowerDNS Recursor on cp33 is OK: DNS OK: 0.138 seconds response time. miraheze.org returns 108.175.15.182,2607:f1c0:1800:26f::1,2607:f1c0:1800:8100::1,74.208.203.152 [03:09:29] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.26, 2.95, 2.93 [03:13:22] !log [void@mw121] = new ManageWikiExtensions('politicswiki'); ->remove(->list()); ->add(); ->commit(); # Resets wiki extensions to default [03:13:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:14:03] !log [void@mw121] $mwe = new ManageWikiExtensions(politicswiki); $mwe->remove($mwe->list()); $mwe->add($wgManageWikiExtensionsDefault); $mwe->commit(); # Resets wiki extensions to default [03:14:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:16:59] !log [void@mw121] sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateGroupPermissionsWithDefaults.php --wiki=politicswiki --overwrite [03:17:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:22:57] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1044858992 secs [03:23:50] [02miraheze/ErrorPages] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/styfle/cancel-workflow-action-0.11.0 [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/commit/79f657160ab3 [03:23:51] [02miraheze/ErrorPages] 07dependabot[bot] 0379f6571 - Bump styfle/cancel-workflow-action from 0.10.1 to 0.11.0 [03:23:54] [02ErrorPages] 07dependabot[bot] opened pull request 03#12: Bump styfle/cancel-workflow-action from 0.10.1 to 0.11.0 - 13https://github.com/miraheze/ErrorPages/pull/12 [03:23:55] [url] Page not found · GitHub · GitHub | github.com [03:23:57] [02ErrorPages] 07dependabot[bot] created branch 03dependabot/github_actions/styfle/cancel-workflow-action-0.11.0 - 13https://github.com/miraheze/ErrorPages [03:23:58] [url] Page not found · GitHub · GitHub | github.com [03:23:59] [02ErrorPages] 07dependabot[bot] labeled pull request 03#12: Bump styfle/cancel-workflow-action from 0.10.1 to 0.11.0 - 13https://github.com/miraheze/ErrorPages/pull/12 [03:24:00] [url] Page not found · GitHub · GitHub | github.com [03:24:35] miraheze/ErrorPages - dependabot[bot] the build passed. [03:29:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 0.90, 2.40, 3.74 [03:29:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 10.71, 5.15, 3.32 [03:31:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.10, 3.58, 4.01 [03:31:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.58, 3.88, 3.07 [03:35:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.58, 3.60, 3.96 [03:37:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.30, 10.00, 8.27 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.04, 10.20, 8.34 [03:38:28] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.09, 3.84, 3.22 [03:39:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.74, 3.98, 3.34 [03:39:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.41, 10.06, 8.51 [03:40:28] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.78, 3.21, 3.07 [03:41:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.88, 3.59, 3.26 [03:41:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.92, 9.27, 8.41 [03:41:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.33, 10.94, 9.16 [03:43:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.95, 3.15, 3.15 [03:43:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.46, 9.53, 8.85 [03:45:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.95, 2.18, 3.15 [03:52:58] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.0794647634 secs [03:55:59] [02ErrorPages] 07Universal-Omega closed pull request 03#12: Bump styfle/cancel-workflow-action from 0.10.1 to 0.11.0 - 13https://github.com/miraheze/ErrorPages/pull/12 [03:56:00] [url] Page not found · GitHub · GitHub | github.com [03:56:01] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/bce46dfb0186...ba49e8747a87 [03:56:02] [url] Comparing bce46dfb0186...ba49e8747a87 · miraheze/ErrorPages · GitHub | github.com [03:56:04] [02miraheze/ErrorPages] 07dependabot[bot] 03ba49e87 - Bump styfle/cancel-workflow-action from 0.10.1 to 0.11.0 (#12) [03:56:06] [02ErrorPages] 07Universal-Omega deleted branch 03dependabot/github_actions/styfle/cancel-workflow-action-0.11.0 - 13https://github.com/miraheze/ErrorPages [03:56:07] ... [03:56:09] [02miraheze/ErrorPages] 07Universal-Omega deleted branch 03dependabot/github_actions/styfle/cancel-workflow-action-0.11.0 [03:56:48] miraheze/ErrorPages - Universal-Omega the build passed. [04:00:31] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.37, 4.21, 2.56 [04:04:27] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.26, 3.64, 2.81 [04:06:26] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.13, 2.70, 2.56 [04:12:35] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ErrorPages Sync] [04:13:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.62, 3.72, 2.97 [04:15:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.54, 3.69, 3.03 [04:16:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.55, 3.32, 2.17 [04:18:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.77, 3.18, 2.27 [04:21:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.09, 3.36, 3.18 [04:26:10] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.60, 3.76, 2.99 [04:30:02] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.22, 2.71, 2.77 [04:40:32] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [04:57:21] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.45, 3.45, 2.58 [04:59:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.63, 3.21, 2.61 [05:01:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.60, 3.11, 2.56 [05:03:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.34, 2.41, 2.38 [05:06:26] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 10.14, 4.70, 3.16 [05:12:25] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.61, 3.66, 3.32 [05:14:25] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.18, 2.94, 3.10 [05:57:21] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.78, 3.65, 2.57 [05:58:13] [02mw-config] 07AgentIsai opened pull request 03#4927: Add new rejection canned responses - 13https://github.com/miraheze/mw-config/pull/4927 [05:58:14] [url] Page not found · GitHub · GitHub | github.com [05:59:18] miraheze/mw-config - AgentIsai the build passed. [05:59:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.70, 2.87, 2.41 [06:12:34] [02mw-config] 07Universal-Omega reviewed pull request 03#4927 commit - 13https://github.com/miraheze/mw-config/pull/4927#discussion_r994187171 [06:12:35] [url] Add new rejection canned responses by AgentIsai · Pull Request #4927 · miraheze/mw-config · GitHub | github.com [06:16:24] [02mw-config] 07Universal-Omega reviewed pull request 03#4927 commit - 13https://github.com/miraheze/mw-config/pull/4927#discussion_r994198426 [06:16:26] [url] Add new rejection canned responses by AgentIsai · Pull Request #4927 · miraheze/mw-config · GitHub | github.com [06:16:26] [02mw-config] 07Universal-Omega reviewed pull request 03#4927 commit - 13https://github.com/miraheze/mw-config/pull/4927#discussion_r994189920 [06:16:27] [url] Add new rejection canned responses by AgentIsai · Pull Request #4927 · miraheze/mw-config · GitHub | github.com [06:19:17] [02mw-config] 07Universal-Omega reviewed pull request 03#4927 commit - 13https://github.com/miraheze/mw-config/pull/4927#discussion_r994201149 [06:19:18] [url] Add new rejection canned responses by AgentIsai · Pull Request #4927 · miraheze/mw-config · GitHub | github.com [06:20:36] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [06:20:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:21:01] @Agent left some reviews on your PR. [06:23:45] [discord] looking [06:24:10] [02mw-config] 07AgentIsai synchronize pull request 03#4927: Add new rejection canned responses - 13https://github.com/miraheze/mw-config/pull/4927 [06:24:11] [url] Page not found · GitHub · GitHub | github.com [06:24:29] [02mw-config] 07AgentIsai synchronize pull request 03#4927: Add new rejection canned responses - 13https://github.com/miraheze/mw-config/pull/4927 [06:24:30] [url] Page not found · GitHub · GitHub | github.com [06:24:41] [02mw-config] 07AgentIsai synchronize pull request 03#4927: Add new rejection canned responses - 13https://github.com/miraheze/mw-config/pull/4927 [06:24:42] [url] Page not found · GitHub · GitHub | github.com [06:24:45] [02miraheze/landing] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/styfle/cancel-workflow-action-0.11.0 [+0/-0/±1] 13https://github.com/miraheze/landing/commit/6dc7cc5f6ab8 [06:24:48] [02miraheze/landing] 07dependabot[bot] 036dc7cc5 - Bump styfle/cancel-workflow-action from 0.10.1 to 0.11.0 [06:24:50] [02landing] 07dependabot[bot] opened pull request 03#89: Bump styfle/cancel-workflow-action from 0.10.1 to 0.11.0 - 13https://github.com/miraheze/landing/pull/89 [06:24:51] [url] Page not found · GitHub · GitHub | github.com [06:24:51] [02landing] 07dependabot[bot] created branch 03dependabot/github_actions/styfle/cancel-workflow-action-0.11.0 - 13https://github.com/miraheze/landing [06:24:52] [url] Page not found · GitHub · GitHub | github.com [06:24:54] [02landing] 07dependabot[bot] labeled pull request 03#89: Bump styfle/cancel-workflow-action from 0.10.1 to 0.11.0 - 13https://github.com/miraheze/landing/pull/89 [06:24:55] ... [06:25:27] miraheze/mw-config - AgentIsai the build passed. [06:25:36] miraheze/landing - dependabot[bot] the build passed. [06:25:39] [02miraheze/phabricator-extensions] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/styfle/cancel-workflow-action-0.11.0 [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/commit/f4bf5a3973de [06:25:41] miraheze/mw-config - AgentIsai the build passed. [06:25:42] [02miraheze/phabricator-extensions] 07dependabot[bot] 03f4bf5a3 - Bump styfle/cancel-workflow-action from 0.10.1 to 0.11.0 [06:25:43] miraheze/mw-config - AgentIsai the build passed. [06:25:45] [02phabricator-extensions] 07dependabot[bot] created branch 03dependabot/github_actions/styfle/cancel-workflow-action-0.11.0 - 13https://github.com/miraheze/phabricator-extensions [06:25:46] ... [06:25:48] [02phabricator-extensions] 07dependabot[bot] opened pull request 03#31: Bump styfle/cancel-workflow-action from 0.10.1 to 0.11.0 - 13https://github.com/miraheze/phabricator-extensions/pull/31 [06:25:49] ... [06:25:49] [02phabricator-extensions] 07dependabot[bot] labeled pull request 03#31: Bump styfle/cancel-workflow-action from 0.10.1 to 0.11.0 - 13https://github.com/miraheze/phabricator-extensions/pull/31 [06:25:52] [02phabricator-extensions] 07dependabot[bot] labeled pull request 03#31: Bump styfle/cancel-workflow-action from 0.10.1 to 0.11.0 - 13https://github.com/miraheze/phabricator-extensions/pull/31 [06:25:55] [02mw-config] 07AgentIsai synchronize pull request 03#4927: Add new rejection canned responses - 13https://github.com/miraheze/mw-config/pull/4927 [06:26:04] [discord] CosmicAlpha: Thanks for the suggestions, I've committed them now [06:26:36] miraheze/phabricator-extensions - dependabot[bot] the build passed. [06:26:38] Thanks @Agent [06:26:47] [02phabricator-extensions] 07Universal-Omega closed pull request 03#31: Bump styfle/cancel-workflow-action from 0.10.1 to 0.11.0 - 13https://github.com/miraheze/phabricator-extensions/pull/31 [06:26:48] [02miraheze/phabricator-extensions] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/fb4fab899f17...4c17076a2abd [06:26:48] [url] Comparing fb4fab899f17...4c17076a2abd · miraheze/phabricator-extensions · GitHub | github.com [06:26:50] [02miraheze/phabricator-extensions] 07dependabot[bot] 034c17076 - Bump styfle/cancel-workflow-action from 0.10.1 to 0.11.0 (#31) [06:26:52] [02phabricator-extensions] 07Universal-Omega deleted branch 03dependabot/github_actions/styfle/cancel-workflow-action-0.11.0 - 13https://github.com/miraheze/phabricator-extensions [06:26:52] [url] Page not found · GitHub · GitHub | github.com [06:26:52] miraheze/mw-config - AgentIsai the build passed. [06:26:55] [02miraheze/phabricator-extensions] 07Universal-Omega deleted branch 03dependabot/github_actions/styfle/cancel-workflow-action-0.11.0 [06:27:55] miraheze/phabricator-extensions - Universal-Omega the build passed. [06:28:01] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.82, 3.73, 2.05 [06:28:20] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.21, 3.08, 2.13 [06:29:13] [02landing] 07Universal-Omega closed pull request 03#89: Bump styfle/cancel-workflow-action from 0.10.1 to 0.11.0 - 13https://github.com/miraheze/landing/pull/89 [06:29:14] [url] Page not found · GitHub · GitHub | github.com [06:29:16] [02miraheze/landing] 07Universal-Omega deleted branch 03dependabot/github_actions/styfle/cancel-workflow-action-0.11.0 [06:29:17] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/5c5f272c1601...a241b40d282b [06:29:18] [url] Comparing 5c5f272c1601...a241b40d282b · miraheze/landing · GitHub | github.com [06:29:18] [02miraheze/landing] 07dependabot[bot] 03a241b40 - Bump styfle/cancel-workflow-action from 0.10.1 to 0.11.0 (#89) [06:29:20] [02landing] 07Universal-Omega deleted branch 03dependabot/github_actions/styfle/cancel-workflow-action-0.11.0 - 13https://github.com/miraheze/landing [06:29:21] [url] Page not found · GitHub · GitHub | github.com [06:30:00] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 2.00, 3.11, 2.03 [06:30:02] miraheze/landing - Universal-Omega the build passed. [06:30:18] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.92, 2.30, 1.96 [06:30:55] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [06:31:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:37:02] @Agent want me to merge it? [06:37:55] [discord] CosmicAlpha: Yes please, thanks [06:38:48] [02mw-config] 07Universal-Omega closed pull request 03#4927: Add new rejection canned responses - 13https://github.com/miraheze/mw-config/pull/4927 [06:38:49] [url] Page not found · GitHub · GitHub | github.com [06:38:51] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5ab8f225316b...e9c13beaa3f3 [06:38:52] [url] Comparing 5ab8f225316b...e9c13beaa3f3 · miraheze/mw-config · GitHub | github.com [06:38:54] [02miraheze/mw-config] 07AgentIsai 03e9c13be - Add new rejection canned responses (#4927) [06:39:02] done @Agent. No problem. [06:40:05] !log [@mwtask141] starting deploy of {'config': True} to all [06:40:09] miraheze/mw-config - Universal-Omega the build passed. [06:40:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:40:12] !log [@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw142.miraheze.org [06:40:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:40:24] why is mw142 failing. [06:40:57] ? [06:42:33] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'servers': 'mw142'} to mw142 - SUCCESS in 0s [06:42:37] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync],Exec[Landing Sync] [06:42:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:43:15] !log reboot mw142 [06:43:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:45:45] !log [universalomega@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw142.miraheze.org [06:45:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:45:52] !log [universalomega@mwtask141] finished deploy of {'landing': True, 'errorpages': True, 'force': True, 'servers': 'mw142'} to mw142 - SUCCESS in 0s [06:45:56] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: connect to address 2a10:6740::6:503 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [06:45:56] [discord] Thank you \:) [06:45:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:46:07] PROBLEM - mw142 Check Gluster Clients on mw142 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [06:46:12] huh why is mw142 down...? [06:47:17] @Reception123 do you mind looking why mw142 is down, I am going off for the night. [06:47:25] ? [06:47:27] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 29409 bytes in 1.326 second response time [06:47:56] RECOVERY - mw142 HTTPS on mw142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.012 second response time [06:48:02] still throwing 503 so definitely not "OK".... [06:48:27] (rebooting did not fix) [06:48:57] I'm mobile only until the afternoon :( [06:49:49] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [06:50:01] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [06:50:07] RECOVERY - mw142 Check Gluster Clients on mw142 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [06:51:02] @Reception123 thats probably fine. It *should* be depooled, but throws 503 when proxying to it (even now when it says all backends are fine) I think it has been for hours. Didn't catch earlier as was busy. But this afternoon for you is still sooner than I could look also. [06:51:31] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [06:51:46] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [06:52:15] oh huh, I guess I had to manually pool it back in. Works now after depooling and repooling manually @Reception123 [06:52:27] Reception123: it went down at 5am yesterday [06:52:28] !log depool and then repool mw142 [06:52:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:52:54] Oh [06:53:20] It's a bit worrying that we've had a server offline for 26 hours and not noticed [06:53:31] I should've caught it earlier but I was so busy it didn't even cross my mind. [06:53:41] But then also congrats to the rest for staying up, they'd have struggled a few months ago [06:54:09] CosmicAlpha: it's not just 1 persons responsibility [06:55:12] I guess we have canary checks for a reason. I forced some of them as I didn't just think the last server would fail so wrote it off as a false failure, but when it does again I realised it must have been down for a while. So thats why I should've realised when the canary checks failed RhinosF1. [06:55:32] I was busy when doing it I just didn't think about it. [06:55:36] CosmicAlpha: the canary checks are more to detect a code failure [06:55:52] The long awaited next version of the deploy tool stops checking all servers [06:56:12] But I also want it to potentially restart php-fpm to clear cache on the canary [06:56:23] RhinosF1: yes, but still should've realised when it failed with 503s... [06:56:35] CosmicAlpha: true [06:57:02] !log [@test131] starting deploy of {'config': True} to all [06:57:03] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [06:57:05] If the deploy tool is catching an in progress outage though and the first we find out, something has still failed [06:57:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:57:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:57:23] It's not a monitoring tool, it's a guard to prevent bad code rolling out [06:57:30] And a fairly weak one at that [06:57:39] yeah I know. [06:58:55] I was busy working on swift and SocialProfile patch to support it at the time, I just wanted to deploy a CW change and ignored it not even thinking about it.... I guess I won't do that again, but also should've caught it by icinga, but again was busy and didn't even look at that. So yeah that is kinda on me as well. [06:59:21] But even by then it had been out for hours as well... [07:00:25] Yeah [07:00:30] It was down 26 hours [07:00:39] Not just gone down while you were doing work [07:00:50] You also should be asleep [07:01:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.13, 3.60, 2.62 [07:01:45] * RhinosF1 is going up the hill so his signal will go soon [07:04:29] RhinosF1: I probably should be asleep *right now* But yeah we shouldve caught long ago. [07:05:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.13, 3.22, 2.72 [07:05:58] * CosmicAlpha really needs to stop staying up until 1-3AM every single night [07:06:26] I will probably be going to sleep a bit earlier from now on also. [07:10:32] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:15:22] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.67, 5.58, 4.03 [07:17:22] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.23, 3.97, 3.62 [07:17:31] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:19:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.93, 2.99, 3.31 [07:31:20] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.3600461781 secs [07:37:21] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.18, 3.24, 2.90 [07:41:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.75, 3.61, 3.17 [07:42:44] PROBLEM - cp32 APT on cp32 is UNKNOWN: NRPE: Command timed out after 60 seconds [07:43:20] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.0160433948 secs [07:43:21] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.26, 4.26, 3.46 [07:44:41] PROBLEM - cp32 APT on cp32 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (4 critical updates). [07:45:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.24, 3.13, 3.15 [07:47:20] PROBLEM - ping6 on cp33 is WARNING: PING WARNING - Packet loss = 0%, RTA = 122.04 ms [07:48:14] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 121.64 ms [07:48:39] PROBLEM - ping6 on cp32 is WARNING: PING WARNING - Packet loss = 0%, RTA = 122.17 ms [07:51:18] RECOVERY - ping6 on cp33 is OK: PING OK - Packet loss = 0%, RTA = 102.57 ms [07:52:18] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 104.05 ms [07:52:38] RECOVERY - ping6 on cp32 is OK: PING OK - Packet loss = 0%, RTA = 102.86 ms [07:58:35] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 25036 bytes in 3.228 second response time [07:59:24] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 25035 bytes in 0.380 second response time [07:59:24] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 7.26, 3.91, 2.25 [07:59:31] PROBLEM - mw121 nutcracker process on mw121 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 116 (nutcracker), command name 'nutcracker' [07:59:36] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 15.73, 14.24, 8.85 [07:59:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.42, 11.44, 7.82 [07:59:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.57, 11.08, 7.53 [07:59:39] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 16.51, 11.47, 6.94 [07:59:45] PROBLEM - mw132 nutcracker process on mw132 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 115 (nutcracker), command name 'nutcracker' [08:01:14] PROBLEM - mw122 nutcracker process on mw122 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 116 (nutcracker), command name 'nutcracker' [08:01:36] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 4.74, 10.75, 8.21 [08:01:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.15, 9.98, 7.73 [08:01:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.89, 10.11, 7.66 [08:01:39] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 5.95, 9.11, 6.62 [08:01:39] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 1 backends are down. mw132 [08:03:36] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 4.22, 8.55, 7.71 [08:03:39] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [08:05:14] RECOVERY - mw122 nutcracker process on mw122 is OK: PROCS OK: 1 process with UID = 116 (nutcracker), command name 'nutcracker' [08:05:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.00, 12.82, 9.34 [08:05:38] RECOVERY - mw132 nutcracker process on mw132 is OK: PROCS OK: 1 process with UID = 115 (nutcracker), command name 'nutcracker' [08:07:14] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 29522 bytes in 0.253 second response time [08:07:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.88, 10.50, 8.89 [08:09:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.57, 9.26, 8.64 [08:10:37] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29522 bytes in 0.401 second response time [08:11:31] RECOVERY - mw121 nutcracker process on mw121 is OK: PROCS OK: 1 process with UID = 116 (nutcracker), command name 'nutcracker' [08:13:24] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 0.70, 3.38, 3.82 [08:15:32] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [08:17:23] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 0.74, 1.90, 3.10 [08:25:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.20, 2.52 [08:27:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.64, 2.77 [08:29:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.67, 2.89 [08:31:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.47, 2.86, 2.69 [08:43:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.96, 3.52, 2.94 [08:45:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.62, 3.07 [08:47:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.77, 3.31, 3.02 [09:08:15] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.82, 3.55, 2.54 [09:12:15] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.57, 3.97, 2.98 [09:14:15] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.15, 3.95, 3.09 [09:16:15] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.90, 3.81, 3.16 [09:18:15] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.84, 2.74, 2.85 [09:37:40] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.61, 3.06, 2.24 [09:39:40] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 3.00, 2.95, 2.30 [10:01:46] PROBLEM - mabi.myomi.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query myomi.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:30:19] PROBLEM - mabi.myomi.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mabi.myomi.net All nameservers failed to answer the query. [10:52:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.59, 3.63, 2.70 [10:54:25] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.38, 3.11, 2.62 [10:58:53] PROBLEM - mabi.myomi.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query myomi.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:09:43] [02miraheze/GlobalNewFiles] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/GlobalNewFiles/compare/92da9f8d1bef...3deffaa43f32 [11:09:44] [url] Comparing 92da9f8d1bef...3deffaa43f32 · miraheze/GlobalNewFiles · GitHub | github.com [11:09:44] [02miraheze/GlobalNewFiles] 07translatewiki 033deffaa - Localisation updates from https://translatewiki.net. [11:09:45] [url] Main page - translatewiki.net | translatewiki.net [11:09:46] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/47c92c3f2ffa...5d344e9319a6 [11:09:46] [url] Comparing 47c92c3f2ffa...5d344e9319a6 · miraheze/ManageWiki · GitHub | github.com [11:09:49] [02miraheze/ManageWiki] 07translatewiki 035d344e9 - Localisation updates from https://translatewiki.net. [11:09:49] [url] Main page - translatewiki.net | translatewiki.net [11:09:52] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/bf119040e5d0...3d74b06f1edd [11:09:52] [url] Comparing bf119040e5d0...3d74b06f1edd · miraheze/MirahezeMagic · GitHub | github.com [11:09:53] [02miraheze/MirahezeMagic] 07translatewiki 033d74b06 - Localisation updates from https://translatewiki.net. [11:09:53] [url] Main page - translatewiki.net | translatewiki.net [11:14:06] miraheze/GlobalNewFiles - translatewiki the build passed. [11:16:12] miraheze/ManageWiki - translatewiki the build passed. [11:23:01]