[00:00:27] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.12, 8.75, 8.90 [00:03:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.53, 8.36, 8.70 [00:05:44] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 276 failures. Last run 4 minutes ago with 276 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [00:05:58] PROBLEM - mem121 Puppet on mem121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:06:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.08, 9.58, 9.68 [00:06:34] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.39, 7.94, 8.49 [00:09:23] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.10, 9.80, 9.74 [00:10:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.57, 9.04, 9.03 [00:12:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.37, 9.36, 8.98 [00:14:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.21, 9.13, 9.08 [00:14:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.15, 3.63, 3.30 [00:14:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.02, 8.68, 8.21 [00:15:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.35, 9.15, 8.99 [00:16:48] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.62, 10.22, 9.35 [00:17:04] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.72, 10.09, 9.45 [00:17:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.23, 3.94, 3.45 [00:18:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.03, 9.25, 9.04 [00:19:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.67, 9.62, 9.29 [00:20:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.51, 9.93, 9.51 [00:21:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.92, 9.28, 9.11 [00:22:42] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.22, 9.94, 9.46 [00:23:06] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 9.12, 10.21, 9.74 [00:23:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.88, 2.84, 3.14 [00:23:34] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.01, 9.71, 8.81 [00:26:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.83, 9.01, 8.71 [00:27:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a60fd9101e02...086f607f0c31 [00:27:19] [02miraheze/mw-config] 07Universal-Omega 03086f607 - Add QuickInstantCommons to extension-list [00:27:51] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c067a7a82bbe...ca450249940c [00:27:53] [02miraheze/mw-config] 07Universal-Omega 03ca45024 - Update LocalSettings.php [00:27:54] [02mw-config] 07Universal-Omega synchronize pull request 03#4424: Configure QuickInstantCommons - 13https://github.com/miraheze/mw-config/pull/4424 [00:27:59] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.28, 1.72, 1.55 [00:28:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ca450249940c...38576488769d [00:28:21] [02miraheze/mw-config] 07Universal-Omega 033857648 - Update ManageWikiSettings.php [00:28:22] [02mw-config] 07Universal-Omega synchronize pull request 03#4424: Configure QuickInstantCommons - 13https://github.com/miraheze/mw-config/pull/4424 [00:28:26] miraheze/mw-config - Universal-Omega the build passed. [00:28:59] miraheze/mw-config - Universal-Omega the build passed. [00:29:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.42, 9.04, 9.40 [00:29:23] miraheze/mw-config - Universal-Omega the build passed. [00:29:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.46, 9.62, 8.99 [00:30:53] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.23, 1.82, 1.61 [00:32:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.30, 8.99, 8.86 [00:33:32] RECOVERY - mem121 Puppet on mem121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:33:32] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:33:55] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.43, 1.72, 1.61 [00:35:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.97, 9.66, 9.14 [00:36:49] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.23, 7.88, 8.44 [00:37:02] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.86, 1.45, 1.54 [00:38:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.11, 9.47, 9.18 [00:39:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.62, 9.17, 9.86 [00:40:18] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.09, 9.64, 9.88 [00:41:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.50, 3.28, 3.17 [00:42:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.71, 10.63, 10.29 [00:43:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.54, 10.07, 10.00 [00:44:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/38576488769d...6221d0b72074 [00:44:21] [02miraheze/mw-config] 07Universal-Omega 036221d0b - Update Defines.php [00:44:22] [02mw-config] 07Universal-Omega synchronize pull request 03#4424: Configure QuickInstantCommons - 13https://github.com/miraheze/mw-config/pull/4424 [00:44:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.56, 3.26, 3.21 [00:45:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.32, 9.71, 9.98 [00:45:27] miraheze/mw-config - Universal-Omega the build passed. [00:47:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.84, 9.04, 8.73 [00:47:41] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.20, 9.76, 9.37 [00:47:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.20, 9.53, 9.84 [00:50:06] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 9.90, 10.10, 9.80 [00:50:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.58, 9.40, 9.31 [00:52:50] !log [@mwtask111] starting deploy of {'config': True} to scsvg [00:52:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:53:01] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 10s [00:53:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6221d0b72074...9ff4d81022cc [00:55:05] [02miraheze/mw-config] 07Universal-Omega 039ff4d81 - Update Defines.php [00:55:06] [02mw-config] 07Universal-Omega synchronize pull request 03#4424: Configure QuickInstantCommons - 13https://github.com/miraheze/mw-config/pull/4424 [00:56:07] miraheze/mw-config - Universal-Omega the build passed. [00:56:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.66, 9.57, 9.79 [00:56:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.38, 10.86, 9.54 [00:57:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9ff4d81022cc...7bdfbfcd39d4 [00:57:09] [02miraheze/mw-config] 07Universal-Omega 037bdfbfc - Update Defines.php [00:57:11] [02mw-config] 07Universal-Omega synchronize pull request 03#4424: Configure QuickInstantCommons - 13https://github.com/miraheze/mw-config/pull/4424 [00:57:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7bdfbfcd39d4...d3e23ae5efb3 [00:57:58] [02miraheze/mw-config] 07Universal-Omega 03d3e23ae - Update LocalSettings.php [00:58:00] [02mw-config] 07Universal-Omega synchronize pull request 03#4424: Configure QuickInstantCommons - 13https://github.com/miraheze/mw-config/pull/4424 [00:58:10] miraheze/mw-config - Universal-Omega the build passed. [00:58:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.90, 3.49, 3.32 [00:58:18] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.99, 9.35, 9.77 [00:58:55] miraheze/mw-config - Universal-Omega the build passed. [00:59:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.05, 9.63, 9.76 [00:59:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.70, 9.85, 9.37 [01:01:10] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.89, 10.91, 10.29 [01:01:12] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.64, 2.97, 3.13 [01:02:55] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.49, 10.63, 10.01 [01:06:00] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.04, 9.74, 9.80 [01:06:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d3e23ae5efb3...979c4895bb0c [01:06:41] [02miraheze/mw-config] 07Universal-Omega 03979c489 - Update Defines.php [01:06:43] [02mw-config] 07Universal-Omega synchronize pull request 03#4424: Configure QuickInstantCommons - 13https://github.com/miraheze/mw-config/pull/4424 [01:07:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.27, 9.63, 9.94 [01:07:45] miraheze/mw-config - Universal-Omega the build passed. [01:08:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/979c4895bb0c...88134f35bfad [01:08:25] [02miraheze/mw-config] 07Universal-Omega 0388134f3 - Update LocalSettings.php [01:08:27] [02mw-config] 07Universal-Omega synchronize pull request 03#4424: Configure QuickInstantCommons - 13https://github.com/miraheze/mw-config/pull/4424 [01:09:21] miraheze/mw-config - Universal-Omega the build passed. [01:10:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.68, 9.84, 9.95 [01:11:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/88134f35bfad...da1c3c8a0459 [01:11:42] [02miraheze/mw-config] 07Universal-Omega 03da1c3c8 - Update LocalSettings.php [01:11:43] [02mw-config] 07Universal-Omega synchronize pull request 03#4424: Configure QuickInstantCommons - 13https://github.com/miraheze/mw-config/pull/4424 [01:12:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.37, 10.60, 9.56 [01:12:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.78, 10.72, 10.05 [01:12:41] [02mw-config] 07Universal-Omega synchronize pull request 03#4424: Configure QuickInstantCommons - 13https://github.com/miraheze/mw-config/pull/4424 [01:12:43] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/da1c3c8a0459...4cf12f943c46 [01:12:43] miraheze/mw-config - Universal-Omega the build passed. [01:12:44] [02miraheze/mw-config] 07Universal-Omega 034cf12f9 - Update Defines.php [01:13:43] miraheze/mw-config - Universal-Omega the build passed. [01:18:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.96, 3.55 [01:18:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.11, 9.70, 9.89 [01:19:42] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.39, 8.99, 9.68 [01:20:38] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.71, 1.78, 1.64 [01:21:11] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.02, 9.61, 9.97 [01:21:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.84, 3.95, 3.63 [01:21:34] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.98, 9.00, 9.35 [01:23:51] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.10, 1.61, 1.61 [01:24:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.43, 3.17, 3.37 [01:27:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.16, 10.11, 9.98 [01:27:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.91, 10.30, 9.76 [01:31:05] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.43, 9.42, 9.28 [01:31:49] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.46, 10.02, 9.64 [01:33:49] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.51, 8.85, 9.93 [01:34:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.27, 9.51, 9.65 [01:34:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.72, 8.69, 9.07 [01:34:48] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.46, 8.81, 9.25 [01:36:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:39:51] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.68, 9.75, 9.65 [01:42:13] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4cf12f943c46...4fc3aa871e53 [01:42:14] [02miraheze/mw-config] 07Universal-Omega 034fc3aa8 - Update Defines.php [01:42:16] [02mw-config] 07Universal-Omega synchronize pull request 03#4424: Configure QuickInstantCommons - 13https://github.com/miraheze/mw-config/pull/4424 [01:42:34] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.59, 10.48, 10.15 [01:43:14] miraheze/mw-config - Universal-Omega the build passed. [01:43:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4fc3aa871e53...f57213df5682 [01:43:23] [02miraheze/mw-config] 07Universal-Omega 03f57213d - Update LocalWiki.php [01:43:25] [02mw-config] 07Universal-Omega synchronize pull request 03#4424: Configure QuickInstantCommons - 13https://github.com/miraheze/mw-config/pull/4424 [01:44:26] miraheze/mw-config - Universal-Omega the build passed. [01:45:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.50, 9.62, 9.94 [01:45:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.34, 9.82, 9.94 [01:46:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.85, 9.21, 9.46 [01:46:11] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.30, 9.55, 9.17 [01:48:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.29, 4.13, 3.36 [01:48:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.99, 10.86, 10.29 [01:48:44] [02mw-config] 07Universal-Omega closed pull request 03#4424: Configure QuickInstantCommons - 13https://github.com/miraheze/mw-config/pull/4424 [01:48:46] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8 - 13https://github.com/miraheze/mw-config [01:48:47] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8 [01:48:49] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/mw-config/compare/086f607f0c31...6a2f7643ca29 [01:48:50] [02miraheze/mw-config] 07Universal-Omega 036a2f764 - Configure QuickInstantCommons (#4424) [01:49:25] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.99, 8.73, 8.92 [01:49:52] miraheze/mw-config - Universal-Omega the build passed. [01:51:17] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.70, 3.05, 3.09 [01:52:46] !log [@mwtask111] starting deploy of {'config': True} to scsvg [01:53:01] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 15s [01:53:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:53:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:55:56] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.32, 9.69, 9.35 [01:59:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.14, 9.15, 9.18 [02:00:35] [02puppet] 07Universal-Omega opened pull request 03#2377: Remove hack for T217669 - 13https://github.com/miraheze/puppet/pull/2377 [02:01:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.41, 9.02, 9.75 [02:02:19] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.38, 9.42, 9.27 [02:02:20] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.69, 7.96, 6.55 [02:05:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.20, 9.28, 9.14 [02:05:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.85, 9.52, 9.33 [02:05:34] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.00, 7.65, 6.72 [02:07:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.53, 9.20, 9.26 [02:08:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.83, 9.60, 9.29 [02:08:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.72, 3.48, 3.30 [02:08:33] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.46, 10.49, 9.75 [02:08:42] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.46, 8.27, 7.16 [02:10:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.84, 8.56, 8.97 [02:10:48] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/commit/978e36b5eab2 [02:10:49] [02miraheze/ManageWiki] 07Universal-Omega 03978e36b - ManageWikiNamespacesBlacklistedNames: case insensitive [02:10:51] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ManageWiki [02:10:52] [02ManageWiki] 07Universal-Omega opened pull request 03#326: ManageWikiNamespacesBlacklistedNames: case insensitive - 13https://github.com/miraheze/ManageWiki/pull/326 [02:11:40] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.84, 9.95, 9.46 [02:11:41] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.81, 3.25, 3.24 [02:12:00] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.84, 6.61, 6.73 [02:13:53] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.62, 9.79, 9.35 [02:14:58] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.84, 9.98, 9.43 [02:16:16] miraheze/ManageWiki - Universal-Omega the build passed. [02:16:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.44, 9.86, 9.47 [02:18:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.00, 9.94, 9.69 [02:21:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.89, 10.02, 9.74 [02:22:45] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.78, 10.30, 9.83 [02:24:11] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.57, 9.89, 9.74 [02:24:45] [02ManageWiki] 07Universal-Omega closed pull request 03#326: ManageWikiNamespacesBlacklistedNames: case insensitive - 13https://github.com/miraheze/ManageWiki/pull/326 [02:24:47] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/33d0aa91145f...753cae08aa5f [02:24:48] [02miraheze/ManageWiki] 07Universal-Omega 03753cae0 - ManageWikiNamespacesBlacklistedNames: case insensitive (#326) [02:24:50] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ManageWiki [02:24:51] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [02:25:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.01, 9.97, 9.80 [02:26:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.28, 9.57, 9.44 [02:27:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.89, 10.43, 9.95 [02:29:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.94, 9.46, 9.41 [02:29:36] miraheze/ManageWiki - Universal-Omega the build passed. [02:30:21] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.95, 9.70, 9.72 [02:40:29] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.72, 9.74, 9.61 [02:41:03] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.26, 10.10, 9.68 [02:43:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.48, 8.68, 9.23 [02:44:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.85, 9.96, 9.68 [02:46:13] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.09, 9.33, 9.40 [02:49:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.21, 10.60, 9.94 [02:51:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.27, 9.77, 9.40 [02:52:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.37, 9.66, 9.58 [02:54:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.31, 9.18, 9.80 [02:54:38] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.96, 2.21, 1.74 [02:57:11] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.88, 9.41, 9.44 [03:03:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.92, 10.13, 9.67 [03:03:32] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.22, 1.75, 1.74 [03:04:48] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.47, 8.90, 9.65 [03:06:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.50, 8.80, 9.23 [03:06:26] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.09, 1.50, 1.65 [03:08:59] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.72, 10.10, 9.65 [03:13:35] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.11, 9.82, 9.60 [03:15:29] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.61, 9.44, 9.20 [03:18:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.61, 9.61, 9.99 [03:18:34] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.99, 7.60, 8.48 [03:19:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.48, 9.09, 9.91 [03:21:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.25, 8.62, 8.89 [03:21:47] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.11, 12.11, 10.91 [03:30:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.93, 9.72, 9.19 [03:33:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.28, 9.35, 9.15 [03:34:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.72, 8.42, 9.76 [03:34:52] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.98, 8.53, 8.17 [03:37:52] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.67, 10.63, 9.82 [03:39:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.31, 10.40, 9.62 [03:45:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.20, 9.56, 9.49 [03:51:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.15, 10.08, 9.73 [03:53:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.99, 9.86, 9.72 [03:56:53] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.66, 10.64, 10.08 [04:01:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.22, 9.82, 9.57 [04:03:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.63, 9.35, 9.71 [04:04:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.31, 9.61, 9.54 [04:06:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.82, 3.91, 3.20 [04:09:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 9.96, 10.04, 9.76 [04:10:51] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.76, 9.84, 9.16 [04:11:44] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.86, 9.68, 9.64 [04:12:12] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.24, 3.82, 3.36 [04:13:51] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.20, 9.31, 9.10 [04:14:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.87, 9.79, 9.68 [04:15:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.18, 3.30, 3.21 [04:17:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.11, 9.72, 9.67 [04:19:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.04, 9.59, 9.97 [04:22:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.01, 3.44, 3.45 [04:22:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 9.77, 10.17, 10.16 [04:24:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.39, 9.61, 9.75 [04:25:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.00, 3.10, 3.34 [04:25:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.84, 9.70, 9.33 [04:26:17] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.52, 9.88, 9.89 [04:27:31] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.28, 9.96, 9.85 [04:28:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.68, 9.12, 9.16 [04:31:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.44, 9.38, 9.93 [04:33:32] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.62, 9.71, 9.80 [04:34:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.11, 10.62, 10.30 [04:35:34] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.59, 2.46, 1.88 [04:38:08] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.98, 9.95, 9.64 [04:38:35] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.05, 1.93, 1.79 [04:41:31] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.46, 1.66, 1.70 [04:42:18] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.84, 10.01, 9.84 [04:45:09] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.84, 9.39, 9.62 [04:50:53] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.71, 9.84, 9.70 [04:53:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.96, 9.26, 9.51 [04:55:18] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.53, 8.97, 9.78 [04:58:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.84, 9.48, 9.92 [04:58:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.23, 9.43, 9.77 [05:00:59] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.53, 9.90, 10.01 [05:01:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.94, 8.99, 9.52 [05:01:17] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.06, 9.36, 9.84 [05:01:48] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.88, 9.25, 9.86 [05:04:34] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.64, 7.90, 8.43 [05:04:43] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.35, 10.70, 10.27 [05:06:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.97, 9.73, 9.95 [05:10:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.05, 9.50, 9.43 [05:11:16] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.15, 8.93, 9.10 [05:12:36] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.03, 9.49, 9.74 [05:13:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.09, 9.62, 9.53 [05:14:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.20, 8.48, 8.89 [05:14:10] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 8.92, 10.14, 9.33 [05:15:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.31, 9.64, 9.78 [05:15:37] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.00, 7.40, 8.44 [05:16:04] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.07, 9.74, 9.58 [05:16:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.01, 9.63, 10.00 [05:17:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.97, 9.90, 9.36 [05:19:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.72, 9.51, 9.53 [05:19:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.56, 10.08, 10.08 [05:27:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.48, 8.62, 8.66 [05:27:17] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.63, 9.37, 9.39 [05:27:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.04, 9.54, 9.91 [05:27:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.22, 9.61, 9.48 [05:29:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.30, 8.86, 8.71 [05:30:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.86, 9.62, 9.83 [05:32:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.25, 3.59, 2.84 [05:32:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.34, 9.10, 8.88 [05:33:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.06, 9.56, 9.79 [05:33:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.88, 9.14, 9.38 [05:35:03] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.76, 3.20, 2.81 [05:36:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.31, 9.71, 9.69 [05:36:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:36:47] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.39, 9.74, 9.53 [05:39:23] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.62, 9.89, 9.88 [05:40:37] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.53, 9.12, 8.86 [05:42:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.44, 9.96, 9.74 [05:43:30] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.25, 8.38, 8.60 [05:45:33] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.93, 10.02, 9.38 [05:46:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.06, 8.94, 8.76 [05:48:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.78, 8.77, 9.00 [05:50:32] PROBLEM - wiki.sanctuary-rts.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.sanctuary-rts.com' expires in 7 day(s) (Sun 30 Jan 2022 05:28:55 GMT +0000). [05:52:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.88, 8.85, 8.83 [05:52:51] [02ManageWiki] 07lens0021 opened pull request 03#327: Merge ExtensionCredits only if given - 13https://github.com/miraheze/ManageWiki/pull/327 [05:53:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.84, 9.91, 9.52 [05:55:12] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.33, 10.05, 9.27 [05:56:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.44, 9.61, 9.52 [05:58:34] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.59, 9.73, 9.28 [06:00:04] miraheze/ManageWiki - Universal-Omega the build passed. [06:00:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.80, 9.10, 9.27 [06:01:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.66, 8.69, 8.94 [06:04:14] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.36, 9.54, 9.24 [06:06:24] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.04, 1.78, 1.06 [06:07:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.91, 9.20, 9.16 [06:08:33] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.16, 7.93, 8.37 [06:09:38] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.24, 7.72, 8.45 [06:10:43] [02ManageWiki] 07lens0021 synchronize pull request 03#327: Merge ExtensionCredits only if given - 13https://github.com/miraheze/ManageWiki/pull/327 [06:10:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.61, 7.91, 8.50 [06:12:05] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 2.00, 1.94, 1.36 [06:14:00] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=mogaiwiki --username-prefix wikia:mogai --report 1 --no-updates /home/reception/mogai_pages_full.xml (START) [06:14:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:14:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.38, 9.66, 9.13 [06:14:57] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.25, 2.07, 1.51 [06:15:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.02, 8.96, 8.72 [06:17:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.07, 9.28, 8.85 [06:17:56] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.59, 8.30, 8.50 [06:17:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.65, 8.80, 8.86 [06:18:17] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.14, 9.39, 9.15 [06:18:53] [02puppet] 07Reception123 closed pull request 03#2376: CSP: remove `mc-uc.netease.com` (per T7910) - 13https://github.com/miraheze/puppet/pull/2376 [06:18:55] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/73e7c630cb8a...e49021d7165c [06:18:56] [02miraheze/puppet] 07Universal-Omega 03e49021d - CSP: remove `mc-uc.netease.com` (per T7910) (#2376) [06:21:26] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=mogaiwiki --username-prefix wikia:mogai --report 1 --no-updates /home/reception/mogai_pages_full.xml (END - exit=0) [06:21:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:21:52] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=mogaiwiki (START) [06:22:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:23:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.44, 9.36, 9.09 [06:23:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.42, 9.78, 9.46 [06:26:55] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.37, 9.95, 9.59 [06:27:23] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=mogaiwiki (END - exit=0) [06:27:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:29:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.71, 10.03, 9.30 [06:31:18] [02puppet] 07Reception123 closed pull request 03#2377: Remove hack for T217669 - 13https://github.com/miraheze/puppet/pull/2377 [06:31:19] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e49021d7165c...aeef0a4adbd3 [06:31:21] [02miraheze/puppet] 07Universal-Omega 03aeef0a4 - Remove hack for T217669 (#2377) [06:31:35] [02miraheze/MatomoAnalytics] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/nanoid-3.2.0 [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/commit/7ea0cf70ec26 [06:31:36] [02miraheze/MatomoAnalytics] 07dependabot[bot] 037ea0cf7 - Bump nanoid from 3.1.25 to 3.2.0 [06:31:38] [02MatomoAnalytics] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/nanoid-3.2.0 - 13https://github.com/miraheze/MatomoAnalytics [06:31:39] [02MatomoAnalytics] 07dependabot[bot] opened pull request 03#67: Bump nanoid from 3.1.25 to 3.2.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/67 [06:31:41] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#67: Bump nanoid from 3.1.25 to 3.2.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/67 [06:31:42] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#67: Bump nanoid from 3.1.25 to 3.2.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/67 [06:31:49] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/aeef0a4adbd3...c4f808fb67a5 [06:31:50] [02miraheze/puppet] 07Universal-Omega 03c4f808f - MediaWiki: Add StructuredNavigation composer (#2375) [06:31:52] [02puppet] 07Reception123 closed pull request 03#2375: MediaWiki: Add StructuredNavigation composer - 13https://github.com/miraheze/puppet/pull/2375 [06:32:37] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.72, 9.35, 9.59 [06:32:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.21, 8.67, 8.90 [06:33:46] [02puppet] 07Reception123 closed pull request 03#2371: Revert "Revert "Revert "absent manageInactiveWikis cron""" - 13https://github.com/miraheze/puppet/pull/2371 [06:33:47] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c4f808fb67a5...0fd4cc8df11c [06:33:49] [02miraheze/puppet] 07Universal-Omega 030fd4cc8 - Revert "Revert "Revert "absent manageInactiveWikis cron (#2348)" (#2352)" (#2366)" (#2371) [06:34:59] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.42, 10.05, 9.49 [06:36:11] miraheze/MatomoAnalytics - dependabot[bot] the build passed. [06:36:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:37:26] miraheze/ManageWiki - Universal-Omega the build passed. [06:37:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.22, 9.74, 9.47 [06:40:35] [02ManageWiki] 07Universal-Omega closed pull request 03#327: Merge ExtensionCredits only if given - 13https://github.com/miraheze/ManageWiki/pull/327 [06:40:37] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/753cae08aa5f...2495d9a95b54 [06:40:38] [02miraheze/ManageWiki] 07lens0021 032495d9a - Merge ExtensionCredits only if given (#327) [06:41:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.69, 9.73, 9.12 [06:41:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:44:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.87, 9.54, 9.16 [06:45:59] miraheze/ManageWiki - Universal-Omega the build passed. [06:46:27] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.86, 1.95, 2.00 [06:46:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.05, 7.65, 8.36 [06:47:21] !log [@test101] starting deploy of {'config': True} to skip [06:47:22] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [06:47:27] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [06:47:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:47:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:47:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:48:22] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 43 seconds ago with 2 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync],Exec[Landing Sync] [06:50:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.21, 9.24, 9.09 [06:51:33] !log MariaDB [phabricator_herald]> UPDATE herald_rule SET isDisabled = 1 WHERE id = '74'; [06:51:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:52:27] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.11, 2.09, 2.04 [06:52:34] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 314 bytes in 0.022 second response time [06:53:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.35, 8.85, 8.98 [06:53:36] !log [@test101] starting deploy of {'config': True} to skip [06:53:37] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [06:53:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.92, 10.47, 9.48 [06:53:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:54:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:56:42] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.52, 9.94, 9.44 [06:58:10] [02miraheze/ManageWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/nanoid-3.2.0 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/commit/c53038075534 [06:58:11] [02miraheze/ManageWiki] 07dependabot[bot] 03c530380 - Bump nanoid from 3.1.25 to 3.2.0 [06:58:13] [02ManageWiki] 07dependabot[bot] opened pull request 03#328: Bump nanoid from 3.1.25 to 3.2.0 - 13https://github.com/miraheze/ManageWiki/pull/328 [06:58:14] [02ManageWiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/nanoid-3.2.0 - 13https://github.com/miraheze/ManageWiki [06:58:16] [02ManageWiki] 07dependabot[bot] labeled pull request 03#328: Bump nanoid from 3.1.25 to 3.2.0 - 13https://github.com/miraheze/ManageWiki/pull/328 [06:58:17] [02ManageWiki] 07dependabot[bot] labeled pull request 03#328: Bump nanoid from 3.1.25 to 3.2.0 - 13https://github.com/miraheze/ManageWiki/pull/328 [06:58:47] !log [@test101] starting deploy of {'config': True} to skip [06:58:48] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [06:58:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:59:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:59:07] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[postgresql-contrib-13] [07:00:20] !log [reception@test101] test [07:00:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:01:33] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.05, 7.94, 8.45 [07:02:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.94, 7.59, 8.36 [07:02:57] miraheze/ManageWiki - dependabot[bot] the build passed. [07:03:20] !log [@test101] starting deploy of {'config': True} to skip [07:03:21] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [07:03:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:03:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:03:52] [02puppet] 07Universal-Omega opened pull request 03#2378: logsalmsg: fix if $USER is undefined - 13https://github.com/miraheze/puppet/pull/2378 [07:04:29] [02puppet] 07Reception123 closed pull request 03#2378: logsalmsg: fix if $USER is undefined - 13https://github.com/miraheze/puppet/pull/2378 [07:04:31] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0fd4cc8df11c...2ba98a45aa3a [07:04:32] [02miraheze/puppet] 07Universal-Omega 032ba98a4 - logsalmsg: fix if $USER is undefined (#2378) [07:05:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.61, 9.41, 8.66 [07:05:23] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.67, 9.26, 9.13 [07:06:28] !log [@test101] starting deploy of {'config': True} to skip [07:06:29] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [07:06:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:06:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:08:17] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.55, 8.63, 8.97 [07:09:31] !log [@test101] starting deploy of {'config': True} to skip [07:09:32] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [07:09:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:09:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:10:48] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.80, 8.45, 8.45 [07:16:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.03, 9.25, 9.09 [07:17:20] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.38, 8.60, 8.50 [07:17:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.98, 9.39, 9.92 [07:18:12] !log [@test101] starting deploy of {'config': True} to skip [07:18:13] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [07:18:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:18:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:19:08] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 22230 bytes in 0.815 second response time [07:19:42] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.04, 9.32, 9.14 [07:20:17] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.44, 7.59, 8.13 [07:20:37] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.71, 10.98, 10.45 [07:21:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.00, 9.54, 8.72 [07:23:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.93, 10.39, 9.03 [07:25:57] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:27:22] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.83, 9.73, 8.90 [07:31:04] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.51, 7.66, 8.24 [07:34:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.02, 7.78, 8.34 [07:47:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.32, 9.94, 9.78 [07:47:46] !log [@test101] starting deploy of {'config': True} to skip [07:47:47] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [07:47:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:47:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:48:46] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:50:24] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 9.88, 10.18, 9.91 [07:51:34] !log [@test101] starting deploy of {'config': True} to skip [07:51:35] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [07:51:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:51:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:53:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.23, 9.27, 9.60 [07:53:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.72, 9.08, 8.65 [07:54:17] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.57, 8.77, 8.33 [07:55:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.43, 9.29, 9.91 [07:56:04] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.02, 9.44, 9.59 [07:57:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.21, 8.36, 8.26 [07:59:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.66, 8.65, 9.26 [07:59:10] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.95, 9.63, 8.96 [08:01:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:01:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.75, 8.89, 8.80 [08:03:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.81, 9.26, 8.68 [08:04:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:04:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.19, 9.66, 9.11 [08:08:01] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.16, 1.31, 1.85 [08:10:50] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.26, 0.94, 1.62 [08:12:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.28, 9.39, 8.84 [08:13:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.78, 8.02, 8.47 [08:13:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.29, 8.98, 9.14 [08:15:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.95, 8.46, 7.80 [08:17:38] !log [@test101] starting deploy of {'config': True} to skip [08:17:39] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [08:17:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:17:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:17:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.04, 9.35, 9.08 [08:18:28] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.30, 8.16, 7.82 [08:18:35] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.62, 8.67, 8.84 [08:19:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.25, 8.44, 8.54 [08:22:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.07, 7.47, 8.15 [08:24:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.51, 8.97, 8.92 [08:24:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:29:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:30:17] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.20, 7.47, 8.27 [08:32:47] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.55, 9.03, 9.81 [08:35:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.44, 9.77, 9.98 [08:38:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.75, 9.84, 9.97 [08:39:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:43:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:44:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.19, 9.27, 8.97 [08:46:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.52, 9.17, 9.01 [08:46:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.94, 9.71, 9.73 [08:47:45] !log [@test101] starting deploy of {'config': True} to skip [08:47:46] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [08:47:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:47:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:49:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.02, 9.68, 9.73 [08:52:43] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.78, 11.04, 10.23 [08:53:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.11, 8.89, 8.44 [08:56:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.21, 9.82, 9.18 [08:58:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.96, 9.37, 9.83 [08:58:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:59:14] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.58, 7.75, 8.13 [08:59:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.49, 8.27, 8.68 [09:02:30] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.13, 7.65, 8.40 [09:03:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.36, 7.89, 8.50 [09:11:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:14:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.18, 8.91, 8.28 [09:15:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.26, 7.85, 8.44 [09:16:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:17:10] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.30, 8.21, 8.11 [09:17:56] !log [@test101] starting deploy of {'config': True} to skip [09:17:57] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [09:18:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:18:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:19:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:27:21] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.08, 10.60, 9.29 [09:29:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:29:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.14, 8.95, 8.40 [09:30:04] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.82, 9.84, 8.92 [09:30:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.03, 9.46, 8.69 [09:31:16] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.18, 9.89, 8.90 [09:32:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.68, 8.36, 8.26 [09:34:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.80, 9.39, 8.88 [09:36:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.57, 9.78, 8.80 [09:36:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:37:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.28, 9.71, 9.08 [09:39:05] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.65, 10.33, 9.19 [09:39:16] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.12, 8.87, 8.60 [09:46:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:47:27] !log [@test101] starting deploy of {'config': True} to skip [09:47:28] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [09:47:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:47:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:50:17] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.59, 10.29, 9.30 [09:51:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:52:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.37, 9.63, 9.22 [09:53:38] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.28, 9.51, 9.66 [09:55:45] [02puppet] 07RhinosF1 commented on pull request 03#2299: varnish: don't pass cookies to backend for load.php - 13https://github.com/miraheze/puppet/pull/2299#issuecomment-1019144592 [09:56:36] [02puppet] 07RhinosF1 commented on pull request 03#2298: varnish: allow proxying beta - 13https://github.com/miraheze/puppet/pull/2298#issuecomment-1019145275 [10:01:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.06, 7.17, 8.21 [10:01:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.26, 8.92, 9.86 [10:01:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.25, 8.51, 9.64 [10:03:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.21, 8.36, 9.83 [10:04:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.89, 9.16, 9.97 [10:04:43] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.80, 7.01, 8.38 [10:06:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:10:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 4.52, 6.18, 8.05 [10:15:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.56, 6.59, 8.19 [10:15:55] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.86, 7.56, 8.37 [10:18:01] !log [@test101] starting deploy of {'config': True} to skip [10:18:02] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [10:18:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:18:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:20:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.48, 7.14, 8.27 [10:29:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:30:52] [02mw-config] 07AgentIsai opened pull request 03#4425: Add wgWikiSeoDefaultImage - 13https://github.com/miraheze/mw-config/pull/4425 [10:31:55] miraheze/mw-config - AgentIsai the build passed. [10:34:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:43:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.56, 9.59, 8.37 [10:43:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.54, 8.19, 7.61 [10:44:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:46:48] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.98, 8.39, 7.81 [10:47:36] !log [@test101] starting deploy of {'config': True} to skip [10:47:37] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [10:47:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:47:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:49:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.15, 9.86, 8.92 [10:54:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.71, 8.67, 7.79 [10:54:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:59:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:59:48] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.04, 7.91, 7.74 [11:03:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.75, 7.34, 8.10 [11:04:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:14:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.08, 8.53, 7.91 [11:17:37] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.86, 8.32, 7.95 [11:17:53] !log [@test101] starting deploy of {'config': True} to skip [11:17:54] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [11:18:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:18:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:22:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:23:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.79, 8.65, 8.06 [11:26:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.17, 8.67, 8.07 [11:27:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.18, 8.85, 8.54 [11:28:53] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.47, 8.41, 8.05 [11:29:49] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.12, 8.30, 8.40 [11:37:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:40:35] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.48, 8.16, 8.35 [11:46:10] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.96, 8.79, 8.46 [11:47:51] !log [@test101] starting deploy of {'config': True} to skip [11:47:52] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [11:47:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:48:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:48:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:48:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.02, 8.33, 8.33 [11:53:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:58:18] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.38, 8.93, 8.57 [11:59:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:01:08] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.41, 7.89, 8.24 [12:04:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:10:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.54, 9.18, 8.48 [12:10:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:13:17] PROBLEM - wiki.zergmafia.top - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve [12:13:17] e, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404752731323242 seconds [12:13:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.18, 10.28, 9.02 [12:14:28] PROBLEM - wiki.homiedb.party - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve [12:14:28] e, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404509544372559 seconds [12:16:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.12, 8.72, 8.62 [12:16:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.41, 8.78, 8.43 [12:17:38] !log [@test101] starting deploy of {'config': True} to skip [12:17:39] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [12:17:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:18:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:19:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.40, 8.32, 8.47 [12:19:50] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.42, 10.73, 9.22 [12:26:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.76, 8.76, 8.68 [12:28:28] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.68, 9.38, 9.34 [12:32:34] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.89, 8.04, 8.46 [12:34:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.86, 9.11, 8.41 [12:36:45] PROBLEM - wiki.cyberfurs.org - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve [12:36:45] e, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.408142328262329 seconds [12:39:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.79, 9.32, 9.06 [12:40:24] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.03, 8.21, 8.27 [12:42:49] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.32, 9.00, 8.96 [12:43:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.30, 8.62, 8.37 [12:44:18] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.49, 9.87, 9.10 [12:45:44] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.19, 9.52, 9.18 [12:47:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.16, 9.62, 9.14 [12:47:52] !log [@test101] starting deploy of {'config': True} to skip [12:47:53] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [12:48:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:48:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:48:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.28, 9.16, 9.08 [12:49:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.94, 8.33, 8.34 [12:51:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.74, 8.85, 8.57 [12:53:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.33, 9.79, 9.29 [12:56:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.90, 9.08, 8.66 [12:57:08] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.07, 9.53, 9.36 [12:57:26] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.12, 8.53, 8.12 [12:57:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.53, 7.89, 8.28 [12:59:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.40, 9.13, 9.29 [12:59:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.06, 9.24, 8.78 [13:00:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.66, 8.89, 9.12 [13:00:36] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.14, 8.74, 8.23 [13:02:11] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.03, 8.49, 8.60 [13:03:33] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.11, 8.53, 8.24 [13:09:25] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.58, 7.75, 8.04 [13:10:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:11:39] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.87, 7.63, 8.34 [13:16:37] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.53, 9.74, 9.02 [13:17:29] !log [@test101] starting deploy of {'config': True} to skip [13:17:30] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [13:17:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:17:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:17:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:17:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.67, 8.51, 8.48 [13:18:10] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.47, 10.65, 9.47 [13:19:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.59, 9.09, 9.08 [13:20:42] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.23, 9.35, 8.82 [13:23:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.95, 8.89, 8.75 [13:25:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.00, 9.69, 9.31 [13:26:26] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.76, 9.58, 9.00 [13:29:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.50, 9.14, 8.91 [13:30:45] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.43, 8.53, 8.46 [13:31:08] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.10, 10.13, 9.64 [13:32:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.89, 9.82, 9.89 [13:34:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.66, 9.19, 9.52 [13:36:40] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.04, 8.20, 8.44 [13:37:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.72, 9.77, 9.64 [13:38:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.28, 9.30, 9.56 [13:41:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.14, 8.84, 9.35 [13:43:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.54, 9.48, 9.06 [13:43:48] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.47, 8.12, 8.44 [13:44:03] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.34, 9.56, 9.54 [13:46:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.11, 9.60, 9.77 [13:46:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.24, 9.04, 9.36 [13:47:39] !log [@test101] starting deploy of {'config': True} to skip [13:47:40] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [13:47:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:47:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:49:38] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.45, 10.14, 9.69 [13:52:04] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.71, 9.53, 9.11 [13:52:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.93, 9.59, 9.56 [13:53:11] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.32, 8.58, 8.46 [13:54:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.07, 9.07, 8.98 [13:55:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.34, 9.93, 9.70 [13:58:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.19, 9.37, 9.39 [13:58:48] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.33, 3.72, 2.97 [13:59:04] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.69, 7.85, 8.17 [14:01:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.31, 9.28, 9.35 [14:01:47] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.30, 2.78, 2.76 [14:04:14] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.17, 9.66, 9.48 [14:05:29] RECOVERY - wiki.cyberfurs.org - reverse DNS on sslhost is OK: SSL OK - wiki.cyberfurs.org reverse DNS resolves to cp20.miraheze.org - CNAME OK [14:05:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.54, 8.54, 8.49 [14:08:52] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.33, 9.47, 8.88 [14:10:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.96, 9.76, 9.63 [14:10:06] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.56, 9.60, 9.26 [14:11:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.16, 9.51, 9.02 [14:12:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:12:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.19, 10.00, 9.71 [14:13:08] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.87, 9.95, 9.52 [14:13:32] RECOVERY - wiki.homiedb.party - reverse DNS on sslhost is OK: SSL OK - wiki.homiedb.party reverse DNS resolves to cp20.miraheze.org - CNAME OK [14:15:42] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.01, 9.35, 9.52 [14:16:17] RECOVERY - wiki.zergmafia.top - reverse DNS on sslhost is OK: SSL OK - wiki.zergmafia.top reverse DNS resolves to cp21.miraheze.org - CNAME OK [14:17:57] !log [@test101] starting deploy of {'config': True} to skip [14:17:58] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [14:18:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:18:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:18:37] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.48, 10.91, 10.06 [14:19:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:23:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.62, 8.90, 9.83 [14:24:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.47, 9.53, 9.76 [14:27:29] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.34, 10.85, 10.19 [14:32:22] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.53, 9.97, 9.24 [14:33:23] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.20, 10.16, 9.94 [14:36:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.43, 9.51, 9.81 [14:40:15] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.16, 9.82, 9.21 [14:42:18] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.54, 10.34, 10.05 [14:42:26] [02ManageWiki] 07lens0021 opened pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329 [14:43:15] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.70, 9.80, 9.31 [14:44:13] [02ManageWiki] 07lens0021 edited pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329 [14:45:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.37, 9.94, 9.99 [14:46:15] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.69, 10.10, 9.50 [14:46:47] miraheze/ManageWiki - lens0021 the build passed. [14:47:52] !log [@test101] starting deploy of {'config': True} to skip [14:47:53] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [14:48:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:48:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.46, 10.86, 10.30 [14:48:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:49:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.09, 9.69, 10.00 [14:49:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.01, 9.32, 9.53 [14:52:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.42, 9.70, 9.58 [14:52:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.53, 10.07, 10.08 [14:52:57] [02ManageWiki] 07lens0021 edited pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329 [14:55:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.33, 9.96, 9.67 [14:55:31] [02ManageWiki] 07lens0021 edited pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329 [14:55:56] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.47, 9.72, 9.53 [14:58:32] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.95, 9.61, 9.64 [15:00:50] [02ManageWiki] 07lens0021 edited pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329 [15:02:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.83, 9.49, 9.90 [15:02:31] [02ManageWiki] 07lens0021 edited pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329 [15:04:26] [02ManageWiki] 07lens0021 edited pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329 [15:04:50] [02ManageWiki] 07lens0021 edited pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329 [15:05:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.14, 9.59, 9.82 [15:07:57] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.95, 9.40, 9.32 [15:08:09] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.52, 10.31, 10.01 [15:08:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.85, 10.89, 10.14 [15:09:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.75, 8.93, 9.82 [15:11:10] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.91, 9.47, 9.79 [15:14:16] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.48, 9.16, 9.39 [15:15:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.93, 9.14, 9.88 [15:16:16] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.10, 9.96, 9.87 [15:18:00] !log [@test101] starting deploy of {'config': True} to skip [15:18:01] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [15:18:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:18:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:18:44] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.26, 9.69, 9.92 [15:19:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.71, 9.28, 9.65 [15:19:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.62, 3.03 [15:21:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.61, 9.86, 9.98 [15:22:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.09, 2.92, 2.87 [15:24:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.44, 11.70, 10.68 [15:28:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.15, 9.01, 9.29 [15:30:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.05, 9.10, 9.75 [15:33:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.02, 9.82, 9.91 [15:33:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.49, 9.10, 9.91 [15:36:06] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.03, 7.37, 8.29 [15:36:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.28, 8.97, 9.55 [15:39:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.22, 9.42, 9.61 [15:42:35] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.29, 9.56, 9.79 [15:42:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.75, 9.78, 9.78 [15:44:19] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.05, 9.00, 8.92 [15:45:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.11, 9.81, 9.88 [15:45:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.14, 10.76, 10.16 [15:46:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.91, 8.59, 8.60 [15:47:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.00, 8.75, 8.86 [15:47:54] !log [@test101] starting deploy of {'config': True} to skip [15:47:55] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [15:48:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:48:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.04, 10.73, 10.19 [15:48:32] [02ManageWiki] 07lens0021 commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1019295828 [15:48:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:49:15] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.78, 8.18, 8.47 [15:49:23] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2ba98a45aa3a...18fbdb19048e [15:49:25] [02miraheze/puppet] 07JohnFLewis 0318fbdb1 - bastion: Allow port 22 to be proxied [15:54:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.83, 9.55, 9.87 [15:56:26] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.58, 8.85, 8.69 [15:58:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.83, 10.21, 10.06 [16:02:18] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.26, 10.88, 9.76 [16:05:02] [02ManageWiki] 07lens0021 edited pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329 [16:06:18] [02ManageWiki] 07lens0021 edited pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329 [16:07:15] !log install socat on mwtask111 [16:07:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:07:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.79, 8.61, 9.50 [16:08:48] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.71, 9.72, 9.66 [16:10:09] PROBLEM - mwtask111 Puppet on mwtask111 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 17 minutes ago with 0 failures [16:10:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.09, 10.27, 9.95 [16:11:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.80, 9.56, 9.98 [16:12:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.18, 9.58, 9.14 [16:14:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.32, 9.88, 10.02 [16:14:52] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.99, 9.68, 9.60 [16:15:30] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.11, 9.15, 9.04 [16:17:39] !log [@test101] starting deploy of {'config': True} to skip [16:17:40] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [16:17:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:18:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:18:36] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.77, 1.33, 0.98 [16:18:42] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.06, 9.78, 9.31 [16:20:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.72, 9.69, 9.82 [16:21:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.40, 9.29, 9.47 [16:21:38] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.78, 1.02, 0.91 [16:23:44] !log uninstall socat on mwtask111 [16:24:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:24:49] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.04, 8.52, 8.95 [16:26:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.07, 9.52, 9.40 [16:29:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.37, 10.07, 9.87 [16:30:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.49, 9.36, 9.35 [16:32:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.15, 9.80, 9.81 [16:32:54] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.62, 9.91, 9.56 [16:35:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.30, 9.09, 9.27 [16:38:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.11, 9.91, 9.78 [16:47:47] !log [@test101] starting deploy of {'config': True} to skip [16:47:48] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [16:47:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:48:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:50:42] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.11, 9.29, 9.14 [16:54:58] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.21, 9.61, 9.08 [16:55:35] I didn't know my all minor edits on PR are logged in here 🤣 [16:56:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.22, 9.66, 9.40 [16:58:06] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.99, 9.41, 9.13 [16:59:04] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://github.com/miraheze/phabricator-extensions/commit/c40defa0369a [16:59:05] [02miraheze/phabricator-extensions] 07paladox 03c40defa - Add custom GitHub auth [16:59:07] [02phabricator-extensions] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/phabricator-extensions [16:59:08] [02phabricator-extensions] 07paladox opened pull request 03#16: Add custom GitHub auth - 13https://github.com/miraheze/phabricator-extensions/pull/16 [16:59:28] [02miraheze/phabricator-extensions] 07github-actions[bot] pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/c40defa0369a...733a66727fa8 [16:59:29] [02miraheze/phabricator-extensions] 07github-actions 03733a667 - CI: lint code [16:59:31] [02phabricator-extensions] 07github-actions[bot] synchronize pull request 03#16: Add custom GitHub auth - 13https://github.com/miraheze/phabricator-extensions/pull/16 [17:00:04] miraheze/phabricator-extensions - paladox the build passed. [17:00:06] [02phabricator-extensions] 07JohnFLewis commented on commit 03c40defa0369a1369851db2a51f041c4ca3970051 - 13https://github.com/miraheze/phabricator-extensions/commit/c40defa0369a1369851db2a51f041c4ca3970051#r64397189 [17:00:12] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://github.com/miraheze/phabricator-extensions/compare/733a66727fa8...9c9a04b14a9e [17:00:13] [02miraheze/phabricator-extensions] 07paladox 039c9a04b - Create PhabricatorCustomGitHubAuthProvider.php [17:00:15] [02phabricator-extensions] 07paladox synchronize pull request 03#16: Add custom GitHub auth - 13https://github.com/miraheze/phabricator-extensions/pull/16 [17:00:28] [02miraheze/phabricator-extensions] 07github-actions[bot] pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/9c9a04b14a9e...11489dd72aa8 [17:00:29] [02miraheze/phabricator-extensions] 07github-actions 0311489dd - CI: lint code [17:00:31] [02phabricator-extensions] 07github-actions[bot] synchronize pull request 03#16: Add custom GitHub auth - 13https://github.com/miraheze/phabricator-extensions/pull/16 [17:01:20] miraheze/phabricator-extensions - paladox the build passed. [17:01:28] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/11489dd72aa8...48671fa20580 [17:01:29] [02miraheze/phabricator-extensions] 07paladox 0348671fa - Update PhabricatorCustomGitHubAuthProvider.php [17:01:31] [02phabricator-extensions] 07paladox synchronize pull request 03#16: Add custom GitHub auth - 13https://github.com/miraheze/phabricator-extensions/pull/16 [17:02:17] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/48671fa20580...659d41614470 [17:02:19] [02miraheze/phabricator-extensions] 07paladox 03659d416 - Update PhutilCustomGitHubAuthAdapter.php [17:02:20] [02phabricator-extensions] 07paladox synchronize pull request 03#16: Add custom GitHub auth - 13https://github.com/miraheze/phabricator-extensions/pull/16 [17:02:25] miraheze/phabricator-extensions - paladox the build passed. [17:02:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.81, 9.46, 9.98 [17:03:02] [02phabricator-extensions] 07paladox closed pull request 03#16: Add custom GitHub auth - 13https://github.com/miraheze/phabricator-extensions/pull/16 [17:03:03] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03master [+2/-0/±0] 13https://github.com/miraheze/phabricator-extensions/compare/76c31a281c33...2740033ba7d0 [17:03:05] [02miraheze/phabricator-extensions] 07paladox 032740033 - Add custom GitHub auth (#16) [17:03:18] [02phabricator-extensions] 07supertassu reviewed pull request 03#16 commit - 13https://github.com/miraheze/phabricator-extensions/pull/16#discussion_r790161807 [17:03:21] miraheze/phabricator-extensions - paladox the build passed. [17:03:56] [02puppet] 07paladox closed pull request 03#2368: cp20/21/30/31: set nginx::worker_processes to 4 - 13https://github.com/miraheze/puppet/pull/2368 [17:03:57] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [17:03:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/18fbdb19048e...4a59a015f873 [17:03:59] miraheze/phabricator-extensions - paladox the build passed. [17:04:00] [02miraheze/puppet] 07paladox 034a59a01 - cp20/21/30/31: set nginx::worker_processes to 4 (#2368) [17:04:02] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [17:04:09] [02puppet] 07paladox closed pull request 03#2332: varnish: use round_robin - 13https://github.com/miraheze/puppet/pull/2332 [17:04:11] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [17:04:12] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [17:05:59] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.29, 9.82, 10.02 [17:07:24] [02miraheze/phabricator-extensions] 07paladox deleted branch 03paladox-patch-1 [17:07:25] [02phabricator-extensions] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/phabricator-extensions [17:07:31] [02phabricator-extensions] 07paladox reviewed pull request 03#16 commit - 13https://github.com/miraheze/phabricator-extensions/pull/16#discussion_r790162115 [17:07:59] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/2740033ba7d0...44c5776e5bf5 [17:08:00] [02miraheze/phabricator-extensions] 07paladox 0344c5776 - Fix [17:08:39] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.30, 7.82, 8.41 [17:08:57] miraheze/phabricator-extensions - paladox the build passed. [17:09:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.31, 9.20, 9.74 [17:09:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.78, 8.89, 9.81 [17:10:39] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.59, 9.52, 9.21 [17:12:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.09, 9.59, 9.78 [17:12:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.73, 8.95, 9.79 [17:12:57] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.77, 9.93, 10.03 [17:13:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.80, 9.55, 9.31 [17:15:05] [02puppet] 07JohnFLewis reviewed pull request 03#2298 commit - 13https://github.com/miraheze/puppet/pull/2298#discussion_r790162808 [17:15:54] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.34, 9.30, 8.86 [17:18:09] !log [@test101] starting deploy of {'config': True} to skip [17:18:10] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [17:18:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:18:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.25, 10.31, 9.97 [17:18:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:19:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.82, 3.74, 3.42 [17:20:17] [02puppet] 07JohnFLewis reviewed pull request 03#2299 commit - 13https://github.com/miraheze/puppet/pull/2299#discussion_r790163352 [17:21:32] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4a59a015f873...71d190879c73 [17:21:34] [02miraheze/puppet] 07Universal-Omega 0371d1908 - check_reverse_dns: add support for `for.uz` tld (#2312) [17:21:35] [02puppet] 07JohnFLewis closed pull request 03#2312: check_reverse_dns: add support for `for.uz` tld - 13https://github.com/miraheze/puppet/pull/2312 [17:21:42] [02ManageWiki] 07Universal-Omega commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1019313149 [17:21:48] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.28, 9.50, 9.07 [17:22:01] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/71d190879c73...e45b629a0c31 [17:22:02] [02miraheze/puppet] 07Universal-Omega 03e45b629 - Move prometheus::es_exporter to graylog121 (#2350) [17:22:04] [02puppet] 07JohnFLewis closed pull request 03#2350: Add prometheus::es_exporter to graylog121 - 13https://github.com/miraheze/puppet/pull/2350 [17:22:55] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.53, 2.97, 3.17 [17:24:05] [02puppet] 07JohnFLewis synchronize pull request 03#2367: Remove support for ufw - 13https://github.com/miraheze/puppet/pull/2367 [17:24:29] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-4/±37] 13https://github.com/miraheze/puppet/compare/e45b629a0c31...ac66869e380c [17:24:30] [02miraheze/puppet] 07Universal-Omega 03ac66869 - Remove support for ufw (#2367) [17:24:32] [02puppet] 07JohnFLewis closed pull request 03#2367: Remove support for ufw - 13https://github.com/miraheze/puppet/pull/2367 [17:26:36] [02puppet] 07JohnFLewis closed pull request 03#2373: Increase memcached memory to 11GB - 13https://github.com/miraheze/puppet/pull/2373 [17:26:37] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/ac66869e380c...5638dce2ecac [17:26:39] [02miraheze/puppet] 07Universal-Omega 035638dce - Increase memcached memory to 11GB (#2373) [17:27:54] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.46, 9.13, 9.10 [17:29:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.53, 9.37, 9.23 [17:29:21] [02puppet] 07Universal-Omega reviewed pull request 03#2298 commit - 13https://github.com/miraheze/puppet/pull/2298#discussion_r790164135 [17:30:51] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.21, 9.48, 9.20 [17:32:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.34, 9.40, 9.26 [17:33:30] [02puppet] 07JohnFLewis reviewed pull request 03#2298 commit - 13https://github.com/miraheze/puppet/pull/2298#discussion_r790164562 [17:33:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.66, 9.95, 9.46 [17:36:16] [02puppet] 07Universal-Omega synchronize pull request 03#2351: ssl-acme: proxy through bastion - 13https://github.com/miraheze/puppet/pull/2351 [17:40:39] [02puppet] 07JohnFLewis closed pull request 03#2351: ssl-acme: proxy through bastion - 13https://github.com/miraheze/puppet/pull/2351 [17:40:41] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5638dce2ecac...677fb801fc9d [17:40:42] [02miraheze/puppet] 07Universal-Omega 03677fb80 - ssl-acme: proxy through bastion (#2351) [17:41:50] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.10, 9.09, 8.97 [17:42:42] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.55, 9.09, 9.74 [17:43:44] [02puppet] 07supertassu synchronize pull request 03#2369: bastion: move firewall rule to role class - 13https://github.com/miraheze/puppet/pull/2369 [17:45:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.49, 9.43, 9.15 [17:45:23] [02puppet] 07JohnFLewis commented on pull request 03#2369: bastion: move firewall rule to role class - 13https://github.com/miraheze/puppet/pull/2369#issuecomment-1019317969 [17:45:26] [02puppet] 07JohnFLewis closed pull request 03#2369: bastion: move firewall rule to role class - 13https://github.com/miraheze/puppet/pull/2369 [17:45:28] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/677fb801fc9d...355508cd88bd [17:45:29] [02miraheze/puppet] 07supertassu 03355508c - bastion: move firewall rule to role class (#2369) [17:47:09] [02puppet] 07Universal-Omega reviewed pull request 03#2299 commit - 13https://github.com/miraheze/puppet/pull/2299#discussion_r790167071 [17:48:18] !log [@test101] starting deploy of {'config': True} to skip [17:48:19] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [17:48:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:49:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.95, 3.78, 3.27 [17:49:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:49:36] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.11, 9.73, 9.65 [17:50:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.76, 9.07, 9.89 [17:53:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.08, 10.86, 10.43 [17:55:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.14, 9.88, 9.79 [17:55:57] [02puppet] 07supertassu opened pull request 03#2379: base::firewall: fix ssh rule - 13https://github.com/miraheze/puppet/pull/2379 [17:56:00] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.29, 9.69, 9.22 [17:56:29] [02puppet] 07JohnFLewis closed pull request 03#2379: base::firewall: fix ssh rule - 13https://github.com/miraheze/puppet/pull/2379 [17:56:30] JohnLewis: ^ sorry about that [17:56:31] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/355508cd88bd...b50510430b2e [17:56:32] [02miraheze/puppet] 07supertassu 03b505104 - base::firewall: fix ssh rule (#2379) [17:58:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.90, 3.10, 3.14 [17:58:57] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.59, 9.07, 9.09 [17:59:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:01:49] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.42, 10.56, 9.65 [18:02:26] PROBLEM - graylog121 Puppet on graylog121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[prometheus-es-exporter] [18:06:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:10:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.46, 8.97, 8.81 [18:10:44] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.45, 9.73, 9.71 [18:13:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.88, 9.75, 9.19 [18:13:15] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.30, 9.84, 9.76 [18:17:34] !log [@test101] starting deploy of {'config': True} to skip [18:17:35] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [18:17:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:18:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:19:21] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.22, 10.25, 9.56 [18:19:52] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.90, 9.78, 9.62 [18:24:51] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb [18:28:08] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:30:32] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:30:32] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/74f0cdce282e [18:30:34] [02miraheze/mw-config] 07Universal-Omega 0374f0cdc - Lower `$wgRevisionCacheExpiry` to 4 days [18:30:35] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [18:30:37] [02mw-config] 07Universal-Omega opened pull request 03#4426: Lower `$wgRevisionCacheExpiry` to 4 days - 13https://github.com/miraheze/mw-config/pull/4426 [18:31:40] miraheze/mw-config - Universal-Omega the build passed. [18:32:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.71, 9.23, 9.82 [18:32:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.41, 3.32 [18:32:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.18, 3.42, 3.20 [18:33:49] [02puppet] 07JohnFLewis commented on pull request 03#2299: varnish: don't pass cookies to backend for load.php - 13https://github.com/miraheze/puppet/pull/2299#issuecomment-1019330315 [18:33:51] [02puppet] 07JohnFLewis closed pull request 03#2299: varnish: don't pass cookies to backend for load.php - 13https://github.com/miraheze/puppet/pull/2299 [18:34:40] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [18:35:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.40, 8.49, 9.70 [18:36:06] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.23, 3.17, 3.16 [18:36:32] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [18:36:41] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:39:04] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:39:53] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.99, 2.80, 3.12 [18:40:34] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:40:35] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:40:54] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:41:06] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:42:28] RECOVERY - enc.for.uz - reverse DNS on sslhost is OK: SSL OK - enc.for.uz reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK [18:42:31] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw101 [18:42:51] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:42:52] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw101 [18:43:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.39, 7.25, 8.39 [18:43:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.50, 7.15, 8.45 [18:43:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.24, 9.08, 9.76 [18:44:37] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:44:56] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:45:44] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:45:57] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:46:04] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.101 second response time [18:46:53] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:47:52] !log [@test101] starting deploy of {'config': True} to skip [18:47:53] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [18:48:28] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.973 second response time [18:48:36] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.984 second response time [18:48:47] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.419 second response time [18:48:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:49:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:49:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 5.95, 8.50, 9.85 [18:49:19] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.017 second response time [18:49:26] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.363 second response time [18:49:47] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 11 backends are healthy [18:49:56] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 11 backends are healthy [18:50:15] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.823 second response time [18:50:23] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.376 second response time [18:50:26] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.413 second response time [18:50:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.63, 8.92, 9.97 [18:51:11] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:51:37] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.712 second response time [18:51:40] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.316 second response time [18:51:40] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.021 second response time [18:51:53] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.313 second response time [18:52:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.10, 8.94, 9.74 [18:52:41] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:54:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.12, 8.84, 8.49 [18:54:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.06, 9.35, 8.73 [18:56:47] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.44, 8.90, 9.22 [18:56:52] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.64, 9.71, 9.89 [18:59:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.92, 8.88, 9.13 [18:59:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.28, 9.29, 9.69 [19:00:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.46, 9.50, 8.80 [19:01:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.11, 9.78, 9.91 [19:03:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.56, 9.67, 8.99 [19:04:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.03, 10.78, 10.32 [19:07:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.23, 4.70, 3.60 [19:08:13] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.11, 4.83, 3.84 [19:08:56] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:09:58] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.46, 9.00, 9.21 [19:10:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.12, 4.76, 3.96 [19:12:07] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.796 second response time [19:13:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.14, 3.80, 3.57 [19:14:28] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.36, 3.74, 3.74 [19:15:32] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.01, 9.50, 9.20 [19:16:05] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.61, 9.87, 9.61 [19:16:11] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.20, 10.38, 9.65 [19:17:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.58, 3.18, 3.36 [19:17:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.66, 9.71, 9.99 [19:17:41] !log [@test101] starting deploy of {'config': True} to skip [19:17:42] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [19:18:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.38, 9.37, 9.24 [19:20:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.36, 9.92, 10.02 [19:21:52] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.55, 9.52, 9.30 [19:22:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.13, 3.30, 3.70 [19:24:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.97, 9.84, 9.47 [19:29:56] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.77, 2.88, 3.22 [19:30:27] [02mw-config] 07Universal-Omega closed pull request 03#4426: Lower `$wgRevisionCacheExpiry` to 4 days - 13https://github.com/miraheze/mw-config/pull/4426 [19:30:29] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [19:30:30] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 [19:30:32] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6a2f7643ca29...33b3b03350ce [19:30:33] [02miraheze/mw-config] 07Universal-Omega 0333b3b03 - Lower `$wgRevisionCacheExpiry` to 4 days (#4426) [19:30:59] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.31, 10.78, 9.98 [19:31:30] miraheze/mw-config - Universal-Omega the build passed. [19:32:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.71, 2.72, 3.23 [19:34:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.44, 9.97, 9.85 [19:37:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.03, 9.59, 9.69 [19:37:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.91, 9.21, 9.81 [19:38:06] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.48, 9.65, 9.51 [19:40:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.78, 9.57, 9.66 [19:40:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.15, 10.55, 10.23 [19:40:52] [02ManageWiki] 07Universal-Omega commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1019345834 [19:43:08] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.34, 9.94, 9.79 [19:44:29] [02ManageWiki] 07Universal-Omega reviewed pull request 03#329 commit - 13https://github.com/miraheze/ManageWiki/pull/329#discussion_r790178207 [19:46:06] [02ManageWiki] 07Universal-Omega edited a comment on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1019345834 [19:47:26] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.84, 9.70, 9.76 [19:48:26] !log [@test101] starting deploy of {'config': True} to skip [19:48:27] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [19:48:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.84, 3.68, 3.48 [19:49:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:06] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.89, 3.33, 3.37 [19:52:18] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.64, 9.97, 9.99 [20:01:05] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.79, 9.04, 9.92 [20:01:36] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.57, 9.40, 9.56 [20:02:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.81, 8.88, 9.86 [20:02:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.94, 9.37, 9.91 [20:03:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.52, 3.22 [20:04:07] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.39, 11.19, 10.61 [20:06:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.59, 9.35, 9.28 [20:06:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.20, 3.93, 3.44 [20:07:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.96, 8.93, 9.31 [20:08:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.53, 9.63, 9.80 [20:09:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.95, 9.51, 9.33 [20:09:47] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 3.40, 3.33 [20:11:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.31, 10.17, 10.02 [20:13:55] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.62, 10.08, 9.72 [20:17:49] !log [@test101] starting deploy of {'config': True} to skip [20:17:50] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [20:18:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:18:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:18:54] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.62, 9.91, 9.46 [20:21:28] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.98, 2.75, 2.09 [20:23:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.59, 3.64, 3.21 [20:23:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.88, 9.74, 9.81 [20:26:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.19, 3.19, 3.09 [20:26:20] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.53, 10.02, 9.89 [20:27:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.47, 1.48, 1.76 [20:29:26] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.38, 9.77, 9.83 [20:30:28] [02puppet] 07Universal-Omega opened pull request 03#2380: ssl-acme: use -6 to force IPV6 for ProxyCommand - 13https://github.com/miraheze/puppet/pull/2380 [20:30:45] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.81, 1.25, 1.62 [20:31:22] [02puppet] 07JohnFLewis closed pull request 03#2380: ssl-acme: use -6 to force IPV6 for ProxyCommand - 13https://github.com/miraheze/puppet/pull/2380 [20:31:24] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b50510430b2e...b7582b449d25 [20:31:25] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.33, 9.43, 9.75 [20:31:25] [02miraheze/puppet] 07Universal-Omega 03b7582b4 - ssl-acme: use -6 to force IPV6 for ProxyCommand (#2380) [20:31:32] paladox: can I enable puppet on mwtask111? [20:31:59] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.50, 3.36, 3.08 [20:32:21] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.39, 9.85, 9.83 [20:35:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.74, 8.80, 9.42 [20:35:44] Yes [20:38:12] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.03, 4.75, 3.72 [20:39:37] !log [@mwtask111] starting deploy of {'config': True} to scsvg [20:39:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:39:50] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 13s [20:40:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:40:26] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [20:41:36] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.69, 10.30, 9.77 [20:41:49] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a64f59c884a3...e83196819aff [20:41:50] [02miraheze/ssl] 07MirahezeSSLBot 03e831968 - Bot: Update SSL cert for wiki.sanctuary-rts.com [20:42:46] !log manually triggering ssl-acme on mwtask111 for previously missed certificates [20:42:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:43:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.07, 3.55, 3.43 [20:44:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.67, 3.91, 3.72 [20:44:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e83196819aff...233fd88b5fb6 [20:44:24] [02miraheze/ssl] 07MirahezeSSLBot 03233fd88 - Bot: Update SSL cert for www.erikapedia.com [20:45:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/233fd88b5fb6...0f41e5de3f8a [20:45:53] [02miraheze/ssl] 07MirahezeSSLBot 030f41e5d - Bot: Update SSL cert for thestarsareright.org [20:46:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.30, 2.86, 3.18 [20:46:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0f41e5de3f8a...5d8a44c730d8 [20:46:54] [02miraheze/ssl] 07MirahezeSSLBot 035d8a44c - Bot: Update SSL cert for www.johanloopmans.nl [20:47:14] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5d8a44c730d8...29618af697bb [20:47:16] [02miraheze/ssl] 07MirahezeSSLBot 0329618af - Bot: Update SSL cert for patternarchive.online [20:47:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/29618af697bb...d555b639c543 [20:47:38] [02miraheze/ssl] 07MirahezeSSLBot 03d555b63 - Bot: Update SSL cert for www.bluepageswiki.org [20:47:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d555b639c543...5a5fa9323db5 [20:47:56] [02miraheze/ssl] 07MirahezeSSLBot 035a5fa93 - Bot: Update SSL cert for electrowiki.org [20:47:57] !log [@test101] starting deploy of {'config': True} to skip [20:47:58] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [20:48:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:27] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5a5fa9323db5...7689bc6825a8 [20:48:28] [02miraheze/ssl] 07MirahezeSSLBot 037689bc6 - Bot: Update SSL cert for wikimoma.art [20:48:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:05] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7689bc6825a8...3df0c483ee3c [20:49:06] [02miraheze/ssl] 07MirahezeSSLBot 033df0c48 - Bot: Update SSL cert for www.christipedia.nl [20:49:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3df0c483ee3c...cc677acc0e15 [20:49:42] [02miraheze/ssl] 07MirahezeSSLBot 03cc677ac - Bot: Update SSL cert for www.baharna.org [20:50:02] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cc677acc0e15...fb036a4b45a7 [20:50:03] [02miraheze/ssl] 07MirahezeSSLBot 03fb036a4 - Bot: Update SSL cert for www.mcpk.wiki [20:50:25] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.08, 3.14, 3.39 [20:50:46] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.65, 9.45, 9.81 [20:51:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/ssl/compare/fb036a4b45a7...08e69afc5299 [20:51:28] [02miraheze/ssl] 07MirahezeSSLBot 0308e69af - Bot: Update SSL cert for puzzles.wiki [20:52:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/08e69afc5299...d518f6da4884 [20:52:23] [02miraheze/ssl] 07MirahezeSSLBot 03d518f6d - Bot: Update SSL cert for wiki.meregos.com [20:52:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d518f6da4884...2ebaa8c34303 [20:52:44] [02miraheze/ssl] 07MirahezeSSLBot 032ebaa8c - Bot: Update SSL cert for wiki-asterix.cf [20:53:02] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2ebaa8c34303...a1766728bb20 [20:53:04] [02miraheze/ssl] 07MirahezeSSLBot 03a176672 - Bot: Update SSL cert for wiki.worldsofweary.com [20:53:12] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.65, 9.50, 9.14 [20:53:33] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a1766728bb20...72ea843f7f7e [20:53:35] [02miraheze/ssl] 07MirahezeSSLBot 0372ea843 - Bot: Update SSL cert for grc.repository.archiopedia.org [20:53:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/72ea843f7f7e...df5970b68519 [20:53:58] [02miraheze/ssl] 07MirahezeSSLBot 03df5970b - Bot: Update SSL cert for civwiki.org [20:54:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/df5970b68519...c5f372828fe5 [20:54:23] [02miraheze/ssl] 07MirahezeSSLBot 03c5f3728 - Bot: Update SSL cert for wiki.yumeka.icu [20:54:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c5f372828fe5...8b020bcd9529 [20:54:45] [02miraheze/ssl] 07MirahezeSSLBot 038b020bc - Bot: Update SSL cert for reviwiki.info [20:55:09] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8b020bcd9529...88d843e1e8f2 [20:55:10] [02miraheze/ssl] 07MirahezeSSLBot 0388d843e - Bot: Update SSL cert for otcg.ml [20:55:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/88d843e1e8f2...49f74243a6d1 [20:55:23] [02miraheze/ssl] 07MirahezeSSLBot 0349f7424 - Bot: Update SSL cert for rww2.org [20:55:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/49f74243a6d1...314d5a70f5e4 [20:55:54] [02miraheze/ssl] 07MirahezeSSLBot 03314d5a7 - Bot: Update SSL cert for www.zenbuddhism.info [20:56:13] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/314d5a70f5e4...e93b9f8fe4ef [20:56:15] [02miraheze/ssl] 07MirahezeSSLBot 03e93b9f8 - Bot: Update SSL cert for www.arru.xyz [20:56:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.05, 9.78, 9.36 [20:56:31] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e93b9f8fe4ef...0a0905baa8d0 [20:56:32] [02miraheze/ssl] 07MirahezeSSLBot 030a0905b - Bot: Update SSL cert for www.iceria.org [20:56:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0a0905baa8d0...d75547c47a55 [20:56:50] [02miraheze/ssl] 07MirahezeSSLBot 03d75547c - Bot: Update SSL cert for www.lab612.at [20:57:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.33, 10.54, 10.12 [20:57:17] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d75547c47a55...9f8bba32cc74 [20:57:19] [02miraheze/ssl] 07MirahezeSSLBot 039f8bba3 - Bot: Update SSL cert for www.rothwell-leeds.co.uk [20:58:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9f8bba32cc74...fd95dede76c7 [20:58:27] [02miraheze/ssl] 07MirahezeSSLBot 03fd95ded - Bot: Update SSL cert for pornwiki.org [20:58:44] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fd95dede76c7...c7e154a80acb [20:58:45] [02miraheze/ssl] 07MirahezeSSLBot 03c7e154a - Bot: Update SSL cert for en.pornwiki.org [20:58:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c7e154a80acb...854f39379642 [20:59:00] [02miraheze/ssl] 07MirahezeSSLBot 03854f393 - Bot: Update SSL cert for files.pornwiki.org [20:59:27] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/854f39379642...fec191cf8071 [20:59:28] [02miraheze/ssl] 07MirahezeSSLBot 03fec191c - Bot: Update SSL cert for wiki.vinesh.eu.org [20:59:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fec191cf8071...65956eabd449 [20:59:55] [02miraheze/ssl] 07MirahezeSSLBot 0365956ea - Bot: Update SSL cert for sdiy.info [21:00:08] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.60, 9.42, 9.74 [21:02:39] RECOVERY - erikapedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.erikapedia.com' will expire on Fri 22 Apr 2022 19:44:17 GMT +0000. [21:02:54] RECOVERY - www.electrowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electrowiki.org' will expire on Fri 22 Apr 2022 19:47:49 GMT +0000. [21:04:53] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/65956eabd449...0381e8762664 [21:04:54] [02miraheze/ssl] 07MirahezeSSLBot 030381e87 - Bot: Update SSL cert for www.pornwiki.org [21:05:21] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.16, 8.91, 9.83 [21:06:08] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0381e8762664...104e324e5567 [21:06:09] [02miraheze/ssl] 07MirahezeSSLBot 03104e324 - Bot: Update SSL cert for wiki.meeusen.net [21:06:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/104e324e5567...aaf0084902b4 [21:06:36] [02miraheze/ssl] 07MirahezeSSLBot 03aaf0084 - Bot: Update SSL cert for wiki.autocountsoft.com [21:07:00] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/aaf0084902b4...78ef656175d5 [21:07:02] [02miraheze/ssl] 07MirahezeSSLBot 0378ef656 - Bot: Update SSL cert for www.portalsofphereon.com [21:09:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.12, 9.61, 9.64 [21:09:14] RECOVERY - wikimoma.art - LetsEncrypt on sslhost is OK: OK - Certificate 'wikimoma.art' will expire on Fri 22 Apr 2022 19:48:22 GMT +0000. [21:11:38] RECOVERY - www.bluepageswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.bluepageswiki.org' will expire on Fri 22 Apr 2022 19:47:31 GMT +0000. [21:11:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.76, 9.87, 9.85 [21:12:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.81, 8.93, 9.42 [21:13:22] RECOVERY - johanloopmans.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.johanloopmans.nl' will expire on Fri 22 Apr 2022 19:46:47 GMT +0000. [21:15:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.14, 8.50, 9.28 [21:16:03] RECOVERY - thestarsareright.org - LetsEncrypt on sslhost is OK: OK - Certificate 'thestarsareright.org' will expire on Fri 22 Apr 2022 19:45:46 GMT +0000. [21:18:11] !log [@test101] starting deploy of {'config': True} to skip [21:18:12] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [21:18:30] RECOVERY - www.johanloopmans.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.johanloopmans.nl' will expire on Fri 22 Apr 2022 19:46:47 GMT +0000. [21:18:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:18:57] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.18, 9.64, 9.49 [21:19:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:19:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.73, 8.76, 9.62 [21:20:22] RECOVERY - electrowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electrowiki.org' will expire on Fri 22 Apr 2022 19:47:49 GMT +0000. [21:20:54] RECOVERY - wiki.sanctuary-rts.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.sanctuary-rts.com' will expire on Fri 22 Apr 2022 19:41:43 GMT +0000. [21:22:21] RECOVERY - patternarchive.online - LetsEncrypt on sslhost is OK: OK - Certificate 'patternarchive.online' will expire on Fri 22 Apr 2022 19:47:09 GMT +0000. [21:22:47] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.77, 2.24, 1.59 [21:23:28] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.36, 2.26, 1.14 [21:25:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.27, 9.67, 9.31 [21:25:18] RECOVERY - www.erikapedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.erikapedia.com' will expire on Fri 22 Apr 2022 19:44:17 GMT +0000. [21:26:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.64, 9.81, 9.79 [21:26:32] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.39, 1.80, 1.16 [21:26:52] RECOVERY - wiki.yumeka.icu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yumeka.icu' will expire on Fri 22 Apr 2022 19:54:16 GMT +0000. [21:27:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/78ef656175d5...0c6eb873f6a0 [21:27:12] [02miraheze/ssl] 07MirahezeSSLBot 030c6eb87 - Bot: Update SSL cert for wiki.gesamtschule-nordkirchen.de [21:27:25] RECOVERY - www.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.pornwiki.org' will expire on Fri 22 Apr 2022 20:04:48 GMT +0000. [21:28:27] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.87, 9.96, 9.67 [21:29:18] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildImages.php --wiki=2b2twiki --missing (END - exit=0) [21:29:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:29:35] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.20, 1.53, 1.17 [21:29:42] RECOVERY - wiki-asterix.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki-asterix.cf' will expire on Fri 22 Apr 2022 19:52:34 GMT +0000. [21:30:53] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/rebuildImages.php --missing (START) [21:31:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:31:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.94, 9.88, 9.58 [21:31:53] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.16, 9.02, 8.86 [21:32:06] RECOVERY - files.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'files.pornwiki.org' will expire on Fri 22 Apr 2022 19:58:53 GMT +0000. [21:32:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.12, 9.59, 9.77 [21:32:52] RECOVERY - arru.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'www.arru.xyz' will expire on Fri 22 Apr 2022 19:56:08 GMT +0000. [21:34:15] RECOVERY - wiki.autocountsoft.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.autocountsoft.com' will expire on Fri 22 Apr 2022 20:06:29 GMT +0000. [21:35:08] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.16, 8.33, 8.64 [21:35:18] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.93, 1.53, 1.73 [21:35:43] RECOVERY - www.rothwell-leeds.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rothwell-leeds.co.uk' will expire on Fri 22 Apr 2022 19:57:11 GMT +0000. [21:35:46] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 9.71, 10.32, 10.07 [21:35:54] RECOVERY - rothwell-leeds.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rothwell-leeds.co.uk' will expire on Fri 22 Apr 2022 19:57:11 GMT +0000. [21:35:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.54, 3.18 [21:36:12] RECOVERY - otcg.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'otcg.ml' will expire on Fri 22 Apr 2022 19:55:04 GMT +0000. [21:37:08] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0c6eb873f6a0...c3b6ca6d9662 [21:37:09] [02miraheze/ssl] 07MirahezeSSLBot 03c3b6ca6 - Bot: Update SSL cert for www.zenbuddhism.info [21:37:24] RECOVERY - pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'pornwiki.org' will expire on Fri 22 Apr 2022 19:58:20 GMT +0000. [21:37:48] RECOVERY - www.portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Fri 22 Apr 2022 20:06:55 GMT +0000. [21:38:21] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.50, 9.46, 9.41 [21:38:24] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.26, 7.45, 8.24 [21:38:30] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.86, 1.32, 1.61 [21:41:05] RECOVERY - www.zenbuddhism.info - LetsEncrypt on sslhost is OK: OK - Certificate 'www.zenbuddhism.info' will expire on Fri 22 Apr 2022 19:55:47 GMT +0000. [21:41:06] RECOVERY - wiki.vinesh.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.vinesh.eu.org' will expire on Fri 22 Apr 2022 19:59:21 GMT +0000. [21:42:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.17, 9.46, 9.76 [21:42:28] RECOVERY - www.iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Fri 22 Apr 2022 19:56:25 GMT +0000. [21:42:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.52, 3.78, 3.39 [21:43:09] RECOVERY - www.mcpk.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.mcpk.wiki' will expire on Fri 22 Apr 2022 19:49:57 GMT +0000. [21:43:16] RECOVERY - en.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.pornwiki.org' will expire on Fri 22 Apr 2022 19:58:38 GMT +0000. [21:44:09] RECOVERY - www.lab612.at - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lab612.at' will expire on Fri 22 Apr 2022 19:56:43 GMT +0000. [21:45:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.28, 9.39, 9.65 [21:46:16] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:46:16] RECOVERY - www.baharna.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.baharna.org' will expire on Fri 22 Apr 2022 19:49:35 GMT +0000. [21:46:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.70, 3.89, 3.30 [21:46:41] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:46:52] RECOVERY - grc.repository.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'grc.repository.archiopedia.org' will expire on Fri 22 Apr 2022 19:53:28 GMT +0000. [21:47:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb [21:48:01] !log [@test101] starting deploy of {'config': True} to skip [21:48:02] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [21:48:53] RECOVERY - wiki.meeusen.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meeusen.net' will expire on Fri 22 Apr 2022 20:06:03 GMT +0000. [21:49:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:49:28] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.38, 4.03, 3.56 [21:49:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:37] RECOVERY - reviwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Fri 22 Apr 2022 19:54:38 GMT +0000. [21:49:38] RECOVERY - www.arru.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'www.arru.xyz' will expire on Fri 22 Apr 2022 19:56:08 GMT +0000. [21:49:46] RECOVERY - www.christipedia.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Fri 22 Apr 2022 19:48:59 GMT +0000. [21:49:50] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.07, 3.27, 3.20 [21:50:21] [02miraheze/MirahezeMagic] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/90d0cffc4097...2fda23401213 [21:50:22] [02miraheze/MirahezeMagic] 07RhinosF1 032fda234 - i18n: indicate error should be quoted in form [21:50:23] [02MirahezeMagic] 07RhinosF1 synchronize pull request 03#313: Recaptcha: use enterprise version - 13https://github.com/miraheze/MirahezeMagic/pull/313 [21:50:39] RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Fri 22 Apr 2022 19:59:48 GMT +0000. [21:50:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:51:39] RECOVERY - iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Fri 22 Apr 2022 19:56:25 GMT +0000. [21:52:03] RECOVERY - portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Fri 22 Apr 2022 20:06:55 GMT +0000. [21:52:14] RECOVERY - baharna.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.baharna.org' will expire on Fri 22 Apr 2022 19:49:35 GMT +0000. [21:53:12] RECOVERY - wiki.meregos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meregos.com' will expire on Fri 22 Apr 2022 19:52:15 GMT +0000. [21:53:12] RECOVERY - christipedia.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Fri 22 Apr 2022 19:48:59 GMT +0000. [21:53:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.58, 3.60 [21:53:44] RECOVERY - rww2.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rww2.org' will expire on Fri 22 Apr 2022 19:55:16 GMT +0000. [21:53:46] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.110 second response time [21:54:11] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.321 second response time [21:54:34] RECOVERY - www.sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Fri 22 Apr 2022 19:59:48 GMT +0000. [21:54:34] RECOVERY - civwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'civwiki.org' will expire on Fri 22 Apr 2022 19:53:52 GMT +0000. [21:55:52] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:56:40] RECOVERY - wiki.worldsofweary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worldsofweary.com' will expire on Fri 22 Apr 2022 19:52:57 GMT +0000. [21:56:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.29, 10.34, 9.59 [21:58:10] miraheze/MirahezeMagic - RhinosF1 the build passed. [21:59:02] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.17, 8.72, 8.81 [21:59:47] !log root@mwtask111:/var/lib/nagios# ./ssl-acme -s WARNING -t HARD -u for 36 certificates that were missed during the time ssl-acme was broken [21:59:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:00:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:01:43] [02mw-config] 07Universal-Omega closed pull request 03#4401: Add StructuredNavigation to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4401 [22:01:44] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [22:01:46] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [22:01:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/33b3b03350ce...73fe4cdf7cb6 [22:01:49] [02miraheze/mw-config] 07Universal-Omega 0373fe4cd - Add StructuredNavigation to ManageWikiExtensions (#4401) [22:02:06] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:02:37] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.282 second response time [22:02:44] miraheze/mw-config - Universal-Omega the build passed. [22:03:18] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.56, 3.81, 3.75 [22:05:27] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.32, 8.89, 8.98 [22:06:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:06:39] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.01, 4.22, 3.90 [22:06:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.67, 9.85, 10.00 [22:06:46] RECOVERY - mw102 APT on mw102 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:07:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.71, 4.25, 3.85 [22:08:45] RECOVERY - mw112 APT on mw112 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:08:49] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:08:57] RECOVERY - mw111 APT on mw111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:08:58] RECOVERY - mw101 APT on mw101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:09:10] RECOVERY - mw122 APT on mw122 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:09:15] !log apt-get upgrade on mw*, mwtask111, and test101 [22:09:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:09:25] RECOVERY - test101 APT on test101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:09:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.49, 9.56, 9.43 [22:09:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.52, 3.46, 3.66 [22:10:12] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.60, 1.87, 1.77 [22:10:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.78, 3.67, 3.70 [22:10:33] RECOVERY - mw121 APT on mw121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:10:56] RECOVERY - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gesamtschule-nordkirchen.de' will expire on Fri 22 Apr 2022 20:27:04 GMT +0000. [22:11:54] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.24, 8.94, 8.87 [22:12:10] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.99, 9.51, 9.93 [22:13:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.42, 11.24, 10.44 [22:15:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.81, 8.85, 8.84 [22:15:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.44, 10.65, 10.28 [22:16:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.45, 2.85, 3.30 [22:16:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.94, 9.87, 9.73 [22:16:23] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.67, 9.56, 8.61 [22:16:39] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.50, 1.62, 1.69 [22:18:20] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.37, 10.64, 9.56 [22:19:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.52, 10.17, 9.88 [22:22:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.45, 9.86, 9.18 [22:23:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.92, 2.81, 3.24 [22:23:29] !log [@mwtask111] starting deploy of {'config': True} to scsvg [22:23:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:23:44] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 15s [22:23:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:27:28] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.52, 1.81, 1.80 [22:27:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.87, 8.75, 9.30 [22:29:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.95, 9.09, 9.85 [22:32:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.50, 10.33, 10.21 [22:33:41] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.43, 1.46, 1.63 [22:34:02] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/07598925f1cc [22:34:03] [02miraheze/mw-config] 07Universal-Omega 030759892 - Update Defines.php [22:34:05] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [22:34:27] [02mw-config] 07Universal-Omega opened pull request 03#4427: use `::class` - 13https://github.com/miraheze/mw-config/pull/4427 [22:35:33] miraheze/mw-config - Universal-Omega the build passed. [22:35:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.57, 9.46, 9.98 [22:37:37] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/07598925f1cc...e475dc225f73 [22:37:39] [02miraheze/mw-config] 07Universal-Omega 03e475dc2 - Update LocalSettings.php [22:37:40] [02mw-config] 07Universal-Omega synchronize pull request 03#4427: use `::class` - 13https://github.com/miraheze/mw-config/pull/4427 [22:38:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e475dc225f73...6e03d6b17d5f [22:38:24] [02miraheze/mw-config] 07Universal-Omega 036e03d6b - Update Defines.php [22:38:25] [02mw-config] 07Universal-Omega synchronize pull request 03#4427: use `::class` - 13https://github.com/miraheze/mw-config/pull/4427 [22:38:47] miraheze/mw-config - Universal-Omega the build passed. [22:39:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6e03d6b17d5f...159ed34ee505 [22:39:04] [02miraheze/mw-config] 07Universal-Omega 03159ed34 - Update GlobalCache.php [22:39:06] [02mw-config] 07Universal-Omega synchronize pull request 03#4427: use `::class` - 13https://github.com/miraheze/mw-config/pull/4427 [22:39:11] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.97, 7.86, 8.36 [22:39:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.97, 9.79, 9.96 [22:39:21] miraheze/mw-config - Universal-Omega the build passed. [22:39:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/159ed34ee505...abb50cb0b248 [22:39:30] [02miraheze/mw-config] 07Universal-Omega 03abb50cb - Update Database.php [22:39:31] [02mw-config] 07Universal-Omega synchronize pull request 03#4427: use `::class` - 13https://github.com/miraheze/mw-config/pull/4427 [22:39:35] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:39:54] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:03] miraheze/mw-config - Universal-Omega the build passed. [22:40:33] miraheze/mw-config - Universal-Omega the build passed. [22:42:45] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.498 second response time [22:43:01] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.828 second response time [22:43:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.07, 3.16, 2.97 [22:46:43] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.53, 9.94, 9.39 [22:46:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.97, 1.79, 1.52 [22:49:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.24, 9.76, 9.45 [22:50:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.19, 1.78, 1.55 [22:50:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.82, 3.39 [22:51:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.98, 3.66, 3.32 [22:51:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.47, 9.48, 9.98 [22:52:55] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.10, 2.16, 1.83 [22:52:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.12, 8.95, 8.66 [22:54:35] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.07, 3.38, 3.27 [22:55:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.21, 9.58, 9.98 [22:56:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/79ade17fa46d [22:56:12] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.57, 9.87, 9.07 [22:56:13] [02miraheze/mw-config] 07Universal-Omega 0379ade17 - cleanup Wikibase.php [22:56:15] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [22:56:16] [02mw-config] 07Universal-Omega opened pull request 03#4428: cleanup Wikibase.php - 13https://github.com/miraheze/mw-config/pull/4428 [22:56:35] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.71, 1.30, 1.43 [22:56:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.12, 4.48, 3.68 [22:57:10] miraheze/mw-config - Universal-Omega the build passed. [22:58:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/73fe4cdf7cb6...6aa2cfd06226 [22:58:22] [02miraheze/mw-config] 07Universal-Omega 036aa2cfd - use ldap111 [22:59:24] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.34, 1.86, 1.86 [22:59:28] miraheze/mw-config - Universal-Omega the build passed. [22:59:29] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.61, 9.17, 8.94 [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:04] !log [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg [23:00:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.84, 3.81, 3.58 [23:00:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:04:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.53, 9.44, 9.53 [23:05:23] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.31, 1.98, 1.91 [23:05:34] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.17, 9.20, 8.89 [23:06:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.18, 3.04, 3.29 [23:07:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.20, 8.51, 9.16 [23:08:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.84, 9.40, 9.19 [23:08:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.38, 9.26, 9.89 [23:08:39] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to scsvg [23:08:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:08:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.29, 8.94, 8.83 [23:08:58] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to scsvg - SUCCESS in 19s [23:09:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:10:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.56, 9.22, 9.29 [23:11:39] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 9.77, 10.02, 10.10 [23:13:56] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.31, 9.07, 9.20 [23:14:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.10, 9.22, 9.31 [23:14:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.94, 9.35, 9.80 [23:15:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.76, 3.47 [23:16:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:18:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.52, 9.82, 9.58 [23:18:11] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.97, 10.15, 10.03 [23:18:42] !log [@test101] starting deploy of {'config': True} to skip [23:18:43] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [23:18:43] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.39, 9.84, 9.36 [23:19:01] !log [@test101] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 1139s [23:19:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:19:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:20:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:20:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.08, 9.49, 9.34 [23:20:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.87, 4.09, 3.74 [23:21:54] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.34, 8.99, 9.11 [23:23:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:23:42] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.32, 9.15, 9.25 [23:24:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.57, 3.96, 3.76 [23:24:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.01, 9.12, 9.65 [23:24:38] !log root@mwtask111:/mnt/mediawiki-static# chown -R www-data:www-data 2b2twiki [23:24:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.17, 9.64, 9.59 [23:25:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:25:17] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.35, 9.52, 9.28 [23:27:04] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.96, 9.82, 9.49 [23:28:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.99, 9.24, 9.43 [23:28:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.28, 9.43, 9.29 [23:29:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/79ade17fa46d...bf334668dce6 [23:29:21] [02miraheze/mw-config] 07Universal-Omega 03bf33466 - Update Wikibase.php [23:29:22] [02mw-config] 07Universal-Omega synchronize pull request 03#4428: cleanup Wikibase.php - 13https://github.com/miraheze/mw-config/pull/4428 [23:30:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.91, 9.78, 9.53 [23:30:24] miraheze/mw-config - Universal-Omega the build passed. [23:30:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.32, 2.98, 3.39 [23:31:02] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.51, 10.08, 9.83 [23:31:38] [02mw-config] 07Universal-Omega closed pull request 03#4428: cleanup Wikibase.php - 13https://github.com/miraheze/mw-config/pull/4428 [23:31:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6aa2cfd06226...52b56b10e8af [23:31:41] [02miraheze/mw-config] 07Universal-Omega 0352b56b1 - cleanup Wikibase.php (#4428) [23:31:42] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [23:31:44] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 [23:32:45] miraheze/mw-config - Universal-Omega the build passed. [23:33:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.03, 10.42, 9.79 [23:34:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.57, 2.91, 3.22 [23:37:08] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.44, 9.89, 9.86 [23:37:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/abb50cb0b248...0c3c08bd565c [23:37:18] [02miraheze/mw-config] 07Universal-Omega 030c3c08b - Update LocalSettings.php [23:37:19] [02mw-config] 07Universal-Omega synchronize pull request 03#4427: use `::class` - 13https://github.com/miraheze/mw-config/pull/4427 [23:37:36] [02mw-config] 07Universal-Omega closed pull request 03#4427: use `::class` - 13https://github.com/miraheze/mw-config/pull/4427 [23:37:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/mw-config/compare/52b56b10e8af...b571973ce431 [23:37:39] [02miraheze/mw-config] 07Universal-Omega 03b571973 - use `::class` (#4427) [23:37:41] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [23:37:42] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [23:38:19] miraheze/mw-config - Universal-Omega the build passed. [23:38:41] miraheze/mw-config - Universal-Omega the build passed. [23:38:52] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to skip [23:38:53] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to skip - SUCCESS in 0s [23:39:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:39:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:40:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.23, 9.84, 9.32 [23:41:15] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.31, 10.27, 9.60 [23:43:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.31, 9.41, 9.58 [23:44:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.39, 9.53, 9.46 [23:45:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.87, 4.12, 3.71 [23:46:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.39, 9.52, 9.60 [23:47:55] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.81, 10.28, 9.77 [23:49:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.91, 3.52, 3.17 [23:49:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.18, 4.25, 3.89 [23:51:00] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03varnish-rewrite-config [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/2ffce709023f [23:51:02] [02miraheze/puppet] 07JohnFLewis 032ffce70 - Rewrite Varnish Configuration [23:51:03] [02puppet] 07JohnFLewis created branch 03varnish-rewrite-config - 13https://github.com/miraheze/puppet [23:51:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.28, 9.46, 9.52 [23:51:16] [02puppet] 07JohnFLewis opened pull request 03#2381: Rewrite Varnish Configuration - 13https://github.com/miraheze/puppet/pull/2381 [23:51:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.43, 3.32, 3.53 [23:54:57] !log [@mwtask111] finished deploy of {'l10nupdate': True} to scsvg - SUCCESS in 3292s [23:55:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:55:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.84, 2.66, 2.91 [23:55:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.12, 3.17, 3.54 [23:55:57] !log [@mwtask111] starting deploy of {'config': True} to scsvg [23:56:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:56:15] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 17s [23:56:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:57:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.00, 2.93, 3.30