[00:02:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.13, 8.14, 8.08 [00:04:22] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 7742 MB (19% inode=96%); [00:05:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:05:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.95, 7.71, 7.92 [00:07:01] PROBLEM - mem121 Puppet on mem121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:07:55] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 775 failures. Last run 6 minutes ago with 775 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [00:08:20] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.83, 8.71, 8.37 [00:14:26] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.15, 9.23, 8.71 [00:17:32] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.90, 9.25, 8.85 [00:22:50] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.62, 8.46, 7.94 [00:23:39] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.84, 9.61, 9.04 [00:24:00] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.96, 9.49, 8.61 [00:24:23] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.33, 9.47, 8.71 [00:27:28] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.44, 9.69, 8.95 [00:28:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.42, 8.78, 8.34 [00:28:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.19, 8.62, 8.18 [00:30:26] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.06, 10.59, 9.39 [00:31:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.18, 8.47, 8.28 [00:31:51] RECOVERY - mem121 Puppet on mem121 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [00:33:04] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:35:32] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.83, 9.38, 9.39 [00:36:20] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.44, 9.80, 9.43 [00:37:31] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.29, 7.83, 8.07 [00:38:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.47, 9.87, 9.05 [00:39:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.90, 9.67, 9.58 [00:40:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:41:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.67, 9.64, 8.87 [00:41:25] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.13, 10.36, 9.79 [00:41:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.59, 9.37, 8.99 [00:41:58] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.43, 9.93, 9.68 [00:43:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:44:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.91, 10.33, 9.41 [00:47:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.30, 9.07, 8.92 [00:47:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.06, 9.69, 9.33 [00:47:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.33, 9.61, 8.74 [00:48:09] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.59, 9.58, 9.43 [00:50:10] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.04, 10.34, 9.43 [00:50:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.62, 9.18, 8.74 [00:51:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.14, 9.20, 9.29 [00:53:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.44, 8.88, 9.62 [00:56:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.61, 9.00, 9.18 [00:56:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.56, 10.05, 9.93 [00:57:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.66, 9.63, 9.82 [01:02:20] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.25, 9.46, 9.82 [01:05:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.36, 9.80, 9.56 [01:05:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.53, 9.92, 9.65 [01:08:10] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.02, 9.65, 9.79 [01:08:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.84, 9.17, 9.41 [01:11:16] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.50, 10.18, 9.30 [01:11:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.73, 9.84, 9.72 [01:14:54] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.73, 9.49, 9.43 [01:17:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.24, 10.21, 9.88 [01:17:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.29, 10.02, 9.38 [01:20:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.78, 9.33, 9.60 [01:23:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.03, 9.69, 9.63 [01:23:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.96, 10.12, 9.82 [01:27:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.33, 9.10, 9.37 [01:29:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.07, 9.77, 9.81 [01:30:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.91, 9.18, 9.53 [01:35:08] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.05, 9.51, 9.96 [01:38:42] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.04, 8.17, 8.49 [01:39:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.59, 9.37, 9.36 [01:41:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.21, 9.51, 9.08 [01:41:02] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.80, 9.75, 9.82 [01:42:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.99, 8.84, 9.20 [01:43:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.13, 8.77, 8.87 [01:46:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.28, 9.53, 9.76 [01:47:14] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.23, 9.66, 9.60 [01:47:53] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.15, 9.45, 9.33 [01:48:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.24, 10.08, 9.54 [01:50:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.73, 9.22, 9.45 [01:50:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.19, 9.06, 8.75 [01:50:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.68, 9.39, 9.32 [01:52:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.88, 8.13, 8.48 [01:52:57] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.35, 9.02, 9.39 [01:54:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.18, 9.86, 9.75 [01:55:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.85, 8.87, 9.31 [01:56:55] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.85, 7.93, 8.39 [02:00:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.49, 9.16, 9.36 [02:03:12] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.16, 8.76, 9.16 [02:04:40] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.20, 9.44, 9.24 [02:07:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.17, 9.11, 9.14 [02:08:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:10:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.89, 9.93, 9.42 [02:11:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.16, 7.67, 8.43 [02:13:24] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.65, 9.89, 9.51 [02:13:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:14:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.50, 7.14, 8.09 [02:15:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.38, 8.56, 8.51 [02:18:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.89, 8.72, 8.67 [02:18:04] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.32, 8.96, 8.92 [02:18:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.10, 8.26, 8.42 [02:19:24] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.05, 9.50, 9.39 [02:21:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.78, 8.01, 8.57 [02:22:24] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.43, 8.82, 9.14 [02:23:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:24:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.78, 7.96, 8.46 [02:27:08] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.04, 9.49, 8.93 [02:28:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:29:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.43, 2.35, 2.04 [02:30:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.69, 8.48, 8.67 [02:32:16] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.85, 1.65, 1.81 [02:34:35] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.38, 9.96, 9.09 [02:35:15] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.78, 1.24, 1.62 [02:36:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.42, 8.63, 8.53 [02:37:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.84, 8.84, 8.79 [02:38:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.41, 8.76, 8.59 [02:38:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:38:54] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.26, 8.07, 8.42 [02:39:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.19, 7.79, 8.20 [02:40:11] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.12, 8.91, 8.89 [02:41:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.12, 10.48, 9.30 [02:43:02] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.42, 9.37, 9.13 [02:44:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.32, 9.76, 9.25 [02:45:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:51:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.62, 9.27, 8.79 [02:56:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.42, 10.15, 9.48 [02:57:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.42, 9.60, 9.04 [02:58:32] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.58, 8.92, 8.66 [02:59:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.19, 8.33, 8.89 [03:00:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.97, 8.47, 8.71 [03:01:02] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.38, 8.95, 9.04 [03:01:33] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.07, 8.44, 8.51 [03:04:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.99, 8.36, 8.82 [03:05:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.00, 7.58, 8.44 [03:07:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.15, 8.21, 8.37 [03:09:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.71, 7.63, 8.36 [03:14:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.34, 8.68, 8.79 [03:19:46] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.12, 10.33, 9.31 [03:20:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.32, 9.54, 9.21 [03:23:27] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.03, 9.40, 8.97 [03:23:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.47, 8.67, 8.36 [03:25:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.84, 9.65, 9.39 [03:26:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.11, 8.71, 8.47 [03:26:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.07, 9.08, 8.55 [03:31:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.02, 9.99, 9.64 [03:32:00] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.12, 8.21, 8.39 [03:32:35] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.34, 10.04, 9.43 [03:32:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.08, 8.80, 8.70 [03:33:00] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.08, 9.10, 9.11 [03:34:45] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.44, 9.17, 9.38 [03:35:28] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.62, 8.77, 9.03 [03:35:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.99, 8.70, 8.97 [03:37:37] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.93, 9.54, 9.46 [03:38:22] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.14, 9.01, 9.02 [03:38:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.47, 7.95, 8.44 [03:40:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.24, 9.14, 9.34 [03:41:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.02, 8.72, 8.92 [03:44:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.42, 7.27, 8.21 [03:45:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:55:17] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.95, 9.28, 9.11 [03:58:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.44, 8.94, 8.99 [03:58:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:59:04] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.13, 8.25, 8.44 [04:06:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.53, 9.91, 8.98 [04:09:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.67, 8.98, 8.76 [04:12:11] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.13, 1.92, 1.48 [04:13:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.45, 8.82, 8.67 [04:13:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:15:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.46, 9.47, 8.78 [04:15:50] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.52, 9.96, 9.38 [04:17:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:18:09] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.25, 1.45, 1.42 [04:18:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.78, 9.26, 8.81 [04:18:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.48, 9.00, 9.17 [04:18:55] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.36, 1.82, 1.43 [04:21:53] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.87, 1.64, 1.45 [04:21:54] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.11, 7.82, 8.29 [04:22:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:24:07] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.17, 7.71, 8.25 [04:30:34] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.02, 7.45, 8.32 [04:35:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:38:56] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.42, 7.05, 8.07 [04:45:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:42:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.75, 8.10, 7.46 [05:45:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.02, 7.59, 7.37 [06:19:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:23:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.16, 8.66, 7.85 [06:24:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:29:06] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.22, 7.44, 7.55 [06:49:00] https://usercontent.irccloud-cdn.com/file/DJf0QtZO/image.png [06:49:04] Oh wow [06:49:08] That is a very interestingfeed [06:53:41] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.03, 2.11, 1.42 [06:57:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:02:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:05:46] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.28, 1.76, 1.72 [07:08:41] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.07, 1.43, 1.60 [07:20:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:25:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:56:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:01:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:49:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:52:59] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.60, 2.98, 1.92 [08:54:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:24:20] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.24, 1.58, 1.96 [10:26:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:31:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:33:17] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.13, 1.33, 1.67 [10:34:35] PROBLEM - wiki.yapsavun.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.yapsavun.xyz' expires in 7 day(s) (Sat 19 Feb 2022 10:26:20 GMT +0000). [11:20:10] PROBLEM - miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'miraheze.wiki' expires in 7 day(s) (Sat 19 Feb 2022 11:08:45 GMT +0000). [12:03:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:13:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:26:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:29:52] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.68, 8.48, 7.48 [12:35:48] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.29, 7.72, 7.45 [12:41:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:08:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:14:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.38, 8.69, 8.15 [13:15:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.63, 8.76, 8.26 [13:15:50] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.73, 1.33, 1.08 [13:15:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.73, 8.22, 7.77 [13:18:14] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.93, 8.07, 8.06 [13:18:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:19:02] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.62, 8.52, 7.90 [13:21:55] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.74, 1.30, 1.18 [13:22:05] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.86, 7.86, 7.77 [13:24:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:38:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.75, 7.81, 8.45 [13:45:24] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.47, 8.62, 8.72 [13:46:18] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.42, 8.90, 8.38 [13:48:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.27, 7.74, 8.38 [13:49:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:52:17] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.30, 9.46, 8.79 [13:55:20] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.02, 9.01, 8.75 [13:57:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:58:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.08, 9.32, 8.56 [14:00:32] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.07, 8.71, 8.59 [14:01:23] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.11, 9.57, 9.05 [14:01:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.45, 8.60, 8.42 [14:02:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:02:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.57, 8.36, 8.10 [14:04:20] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.51, 9.60, 9.16 [14:04:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.91, 8.38, 8.38 [14:05:48] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.04, 8.29, 8.16 [14:06:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:09:34] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.39, 9.97, 9.13 [14:18:14] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.47, 9.77, 8.95 [14:19:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.15, 9.71, 9.38 [14:21:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.37, 9.14, 8.86 [14:22:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.03, 8.59, 8.47 [14:22:50] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.21, 9.34, 8.42 [14:25:47] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.42, 9.76, 9.57 [14:26:01] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.54, 8.28, 8.18 [14:30:52] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.12, 9.52, 10.00 [14:31:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.50, 8.64, 8.48 [14:31:46] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.19, 10.56, 9.95 [14:33:16] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.14, 9.88, 9.29 [14:34:40] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.91, 9.81, 9.78 [14:38:06] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.61, 9.20, 8.70 [14:39:41] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.37, 9.91, 9.90 [14:40:06] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.08, 7.52, 8.13 [14:40:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.20, 9.11, 8.89 [14:40:35] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.22, 9.67, 9.72 [14:42:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.45, 9.23, 9.35 [14:43:33] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.68, 9.72, 9.74 [14:44:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.22, 8.17, 8.39 [14:45:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.20, 9.17, 9.26 [14:46:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.10, 9.34, 9.16 [14:46:34] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.18, 10.15, 9.90 [14:49:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.33, 9.66, 9.78 [14:51:15] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.05, 10.38, 9.44 [14:52:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.52, 8.61, 8.39 [14:54:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.63, 8.36, 8.99 [14:54:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.84, 9.45, 9.26 [14:55:04] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 4.97, 7.34, 7.96 [14:58:32] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.33, 10.49, 9.88 [15:00:07] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.11, 9.64, 9.33 [15:03:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.83, 8.83, 9.09 [15:03:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.67, 7.73, 8.50 [15:04:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.50, 9.26, 9.62 [15:06:11] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.72, 9.00, 9.83 [15:06:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:06:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.11, 9.04, 9.00 [15:07:19] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.29, 9.17, 9.48 [15:09:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.57, 8.38, 8.75 [15:10:19] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.70, 8.62, 9.23 [15:10:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:11:50] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.71, 7.89, 8.50 [15:13:13] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.76, 9.11, 9.29 [15:15:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:15:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.52, 7.13, 8.14 [15:16:09] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.52, 8.91, 9.18 [15:17:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.37, 9.52, 9.54 [15:19:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:20:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.99, 9.05, 9.35 [15:22:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.17, 8.67, 8.24 [15:25:17] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.27, 8.05, 8.09 [15:26:25] PROBLEM - gluster111 APT on gluster111 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [15:28:16] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.63, 9.36, 9.05 [15:29:27] PROBLEM - gluster111 APT on gluster111 is CRITICAL: APT CRITICAL: 21 packages available for upgrade (12 critical updates). [15:29:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:32:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.64, 7.63, 8.50 [15:33:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:34:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.52, 9.28, 9.13 [15:40:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.43, 9.25, 9.08 [15:46:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.34, 9.55, 9.30 [15:49:24] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.23, 9.98, 9.48 [15:52:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.92, 9.16, 9.27 [15:54:11] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.85, 9.11, 8.16 [15:55:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.82, 9.02, 8.24 [15:55:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.27, 8.93, 8.68 [15:58:46] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.23, 9.31, 9.26 [16:00:21] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.15, 6.93, 6.85 [16:00:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.25, 8.10, 8.10 [16:01:49] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.76, 8.39, 8.22 [16:03:22] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.37, 6.75, 6.79 [16:04:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.17, 9.36, 9.48 [16:07:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.14, 9.03, 8.66 [16:07:50] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.16, 10.23, 9.79 [16:08:20] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.79, 1.73, 1.43 [16:09:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.13, 8.74, 8.35 [16:10:53] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.04, 10.36, 9.43 [16:11:31] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.92, 1.39, 1.35 [16:12:51] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.52, 7.88, 8.10 [16:15:34] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.61, 9.69, 8.80 [16:16:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.16, 9.72, 9.50 [16:18:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.42, 7.89, 8.27 [16:20:48] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.22, 6.88, 6.90 [16:22:21] [02miraheze/MatomoAnalytics] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.12.0 [+0/-0/±2] 13https://github.com/miraheze/MatomoAnalytics/commit/589cbb66aa84 [16:22:22] [02miraheze/MatomoAnalytics] 07dependabot[bot] 03589cbb6 - Bump stylelint-config-wikimedia from 0.11.1 to 0.12.0 [16:22:24] [02MatomoAnalytics] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.12.0 - 13https://github.com/miraheze/MatomoAnalytics [16:22:25] [02MatomoAnalytics] 07dependabot[bot] opened pull request 03#71: Bump stylelint-config-wikimedia from 0.11.1 to 0.12.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/71 [16:22:27] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#71: Bump stylelint-config-wikimedia from 0.11.1 to 0.12.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/71 [16:22:28] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#71: Bump stylelint-config-wikimedia from 0.11.1 to 0.12.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/71 [16:23:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.20, 10.04, 9.73 [16:23:47] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.11, 6.07, 6.58 [16:25:00] miraheze/MatomoAnalytics - dependabot[bot] the build has errored. [16:25:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.19, 10.00, 9.79 [16:26:18] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.38, 9.31, 9.87 [16:28:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.91, 8.58, 8.46 [16:32:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.91, 7.25, 8.20 [16:34:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.70, 8.25, 8.34 [16:35:00] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.06, 9.57, 9.48 [16:38:05] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.21, 9.35, 9.44 [16:38:27] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.18, 10.58, 9.99 [16:39:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.80, 8.96, 8.67 [16:41:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.05, 10.04, 9.69 [16:42:37] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.41, 8.19, 8.43 [16:42:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.12, 8.82, 8.67 [16:47:43] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.14, 9.18, 9.70 [16:49:02] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.82, 7.80, 8.32 [16:50:20] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.64, 9.89, 9.99 [16:52:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.18, 8.78, 8.68 [16:55:22] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.37, 8.18, 8.47 [16:58:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:02:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:05:54] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.99, 9.55, 9.14 [17:07:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:08:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 5.82, 7.93, 8.60 [17:11:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.87, 8.99, 8.86 [17:12:04] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.13, 7.79, 8.43 [17:14:29] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.00, 8.79, 8.76 [17:15:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:20:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:20:54] [02miraheze/ManageWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.12.0 [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/commit/c2cece26df9d [17:20:55] [02miraheze/ManageWiki] 07dependabot[bot] 03c2cece2 - Bump stylelint-config-wikimedia from 0.11.1 to 0.12.0 [17:20:57] [02ManageWiki] 07dependabot[bot] opened pull request 03#340: Bump stylelint-config-wikimedia from 0.11.1 to 0.12.0 - 13https://github.com/miraheze/ManageWiki/pull/340 [17:20:58] [02ManageWiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.12.0 - 13https://github.com/miraheze/ManageWiki [17:21:00] [02ManageWiki] 07dependabot[bot] labeled pull request 03#340: Bump stylelint-config-wikimedia from 0.11.1 to 0.12.0 - 13https://github.com/miraheze/ManageWiki/pull/340 [17:21:01] [02ManageWiki] 07dependabot[bot] labeled pull request 03#340: Bump stylelint-config-wikimedia from 0.11.1 to 0.12.0 - 13https://github.com/miraheze/ManageWiki/pull/340 [17:23:32] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.83, 9.06, 8.81 [17:23:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:25:01] miraheze/ManageWiki - dependabot[bot] the build has errored. [17:26:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.14, 9.26, 8.95 [17:27:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.68, 9.06, 8.71 [17:29:38] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.79, 9.54, 9.08 [17:30:28] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.24, 9.60, 8.95 [17:32:40] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.09, 9.05, 8.95 [17:33:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:36:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/79bc003f353e [17:36:13] [02miraheze/puppet] 07paladox 0379bc003 - mon111: increase MaxConcurrentChecks to 12 [17:36:15] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [17:36:16] [02puppet] 07paladox opened pull request 03#2420: mon111: increase MaxConcurrentChecks to 12 - 13https://github.com/miraheze/puppet/pull/2420 [17:38:35] [02puppet] 07paladox closed pull request 03#2420: mon111: increase MaxConcurrentChecks to 12 - 13https://github.com/miraheze/puppet/pull/2420 [17:38:40] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [17:38:42] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [17:39:38] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.45, 8.51, 8.89 [17:40:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:41:49] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.28, 9.51, 8.97 [17:44:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.62, 8.83, 8.79 [17:51:06] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.52, 7.74, 8.27 [17:52:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.14, 6.90, 7.98 [17:52:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 16.80, 13.74, 10.12 [17:58:42] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.83, 1.60, 1.19 [17:58:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.91, 8.77, 8.12 [17:59:36] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.02, 8.83, 8.54 [18:01:46] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.19, 9.52, 8.85 [18:01:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.40, 8.99, 8.35 [18:01:54] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.53, 1.48, 1.21 [18:02:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.40, 7.98, 7.91 [18:04:51] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.44, 9.47, 8.96 [18:04:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.54, 9.18, 8.36 [18:08:49] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.89, 9.37, 8.95 [18:09:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.62, 8.89, 8.35 [18:11:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.11, 10.80, 9.68 [18:11:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.93, 9.72, 9.18 [18:11:04] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.87, 10.00, 8.90 [18:14:06] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.88, 9.66, 9.42 [18:15:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.87, 8.77, 8.89 [18:15:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.03, 9.27, 8.69 [18:17:11] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.53, 9.72, 9.46 [18:18:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.56, 8.33, 8.46 [18:20:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.22, 9.53, 9.42 [18:20:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.40, 8.69, 8.97 [18:23:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.71, 7.18, 8.35 [18:24:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.01, 9.21, 9.98 [18:24:35] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.99, 7.53, 8.37 [18:26:40] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.58, 9.54, 9.37 [18:26:41] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.43, 7.68, 8.42 [18:30:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:30:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.81, 8.63, 8.64 [18:31:52] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.71, 9.85, 9.11 [18:32:50] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.95, 9.23, 9.36 [18:33:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.04, 9.17, 8.81 [18:34:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 5.51, 8.00, 8.55 [18:36:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:36:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.18, 8.87, 8.64 [18:37:58] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.50, 7.02, 8.08 [18:40:04] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.63, 8.05, 8.38 [18:42:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.98, 9.68, 9.75 [18:46:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:47:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.13, 7.94, 8.47 [18:48:58] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.24, 8.98, 8.80 [18:51:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:52:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.46, 8.59, 8.69 [18:52:03] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.87, 6.48, 8.16 [18:52:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.54, 8.41, 7.89 [18:55:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.65, 9.16, 8.25 [18:58:12] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.14, 9.77, 8.87 [18:58:13] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.07, 8.06, 7.95 [19:01:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.77, 9.38, 8.88 [19:04:31] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.87, 8.06, 8.44 [19:06:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.03, 8.60, 8.37 [19:10:23] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.32, 9.68, 9.03 [19:11:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:12:42] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.50, 8.43, 8.39 [19:13:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.08, 8.84, 8.85 [19:18:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:19:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.12, 7.91, 8.39 [19:27:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.82, 8.94, 8.43 [19:28:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:33:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.98, 7.63, 8.03 [19:33:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.97, 8.42, 7.96 [19:34:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.17, 8.49, 8.04 [19:35:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:36:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.20, 8.66, 8.58 [19:37:41] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.32, 7.04, 7.56 [19:39:15] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.13, 8.22, 8.41 [19:39:29] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.63, 7.50, 7.74 [19:50:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:51:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.17, 9.24, 8.50 [19:53:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:54:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.68, 9.59, 8.78 [19:54:53] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.72, 9.49, 8.54 [19:58:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.93, 9.98, 8.61 [19:59:34] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.97, 10.37, 9.09 [20:00:42] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.36, 10.35, 9.27 [20:01:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.56, 9.75, 9.02 [20:01:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.33, 9.57, 8.72 [20:02:44] RECOVERY - cp30 APT on cp30 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:03:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:03:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.07, 9.65, 9.18 [20:04:35] RECOVERY - db101 APT on db101 is OK: APT OK: 9 packages available for upgrade (0 critical updates). [20:04:46] RECOVERY - cp31 APT on cp31 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:05:05] RECOVERY - cp21 APT on cp21 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:05:44] RECOVERY - db121 APT on db121 is OK: APT OK: 9 packages available for upgrade (0 critical updates). [20:05:59] RECOVERY - cp20 APT on cp20 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:06:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.92, 8.42, 8.75 [20:06:54] RECOVERY - bast121 APT on bast121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:06:59] RECOVERY - bast101 APT on bast101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:07:08] RECOVERY - ns1 APT on ns1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:07:55] RECOVERY - gluster101 APT on gluster101 is OK: APT OK: 9 packages available for upgrade (0 critical updates). [20:08:04] RECOVERY - phab121 APT on phab121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:08:04] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.50, 9.07, 8.70 [20:08:11] RECOVERY - jobchron121 APT on jobchron121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:08:12] RECOVERY - mem121 APT on mem121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:08:30] RECOVERY - mem101 APT on mem101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:08:31] RECOVERY - gluster111 APT on gluster111 is OK: APT OK: 9 packages available for upgrade (0 critical updates). [20:08:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:08:52] RECOVERY - gluster121 APT on gluster121 is OK: APT OK: 9 packages available for upgrade (0 critical updates). [20:08:55] RECOVERY - mon111 APT on mon111 is OK: APT OK: 6 packages available for upgrade (0 critical updates). [20:09:07] PROBLEM - puppet111 Current Load on puppet111 is CRITICAL: CRITICAL - load average: 9.32, 20.95, 12.34 [20:09:26] RECOVERY - db111 APT on db111 is OK: APT OK: 9 packages available for upgrade (0 critical updates). [20:09:28] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.52, 7.82, 8.44 [20:10:00] RECOVERY - ns2 APT on ns2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:10:15] RECOVERY - ldap111 APT on ldap111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:10:25] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:10:34] RECOVERY - mail121 APT on mail121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:10:57] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.91, 10.25, 9.36 [20:12:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.17, 9.51, 8.84 [20:13:37] RECOVERY - cloud11 SMART on cloud11 is OK: OK: [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean [20:13:41] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.34, 10.68, 9.60 [20:16:41] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.10, 8.52, 8.60 [20:16:57] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.36, 8.92, 8.82 [20:17:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.40, 10.00, 9.53 [20:17:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.74, 9.27, 9.11 [20:20:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.54, 8.39, 9.03 [20:20:26] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.15, 7.42, 8.15 [20:20:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.28, 9.23, 9.00 [20:22:20] PROBLEM - puppet111 conntrack_table_size on puppet111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:22:58] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:23:42] PROBLEM - puppet111 NTP time on puppet111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:24:11] PROBLEM - puppet111 Disk Space on puppet111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:24:26] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.78, 6.69, 8.16 [20:24:46] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:24:54] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.70, 6.59, 8.03 [20:25:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 3.94, 5.93, 7.64 [20:25:21] PROBLEM - puppet111 ferm_active on puppet111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:25:53] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:03] PROBLEM - puppet111 PowerDNS Recursor on puppet111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:26:15] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:19] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:36] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:39] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:26:58] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.79, 9.65, 8.97 [20:27:03] PROBLEM - puppet111 SSH on puppet111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:20] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:29:00] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.69, 9.33, 9.00 [20:29:02] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:29:22] RECOVERY - puppet111 ferm_active on puppet111 is OK: OK ferm input default policy is set [20:29:29] PROBLEM - mw112 Puppet on mw112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:29:33] PROBLEM - db121 Puppet on db121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:29:33] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:29:43] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.850 second response time [20:29:52] RECOVERY - puppet111 PowerDNS Recursor on puppet111 is OK: DNS OK: 1.082 second response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [20:29:57] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.557 second response time [20:30:00] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 0.830 second response time [20:30:13] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.569 second response time [20:30:18] PROBLEM - mem101 Puppet on mem101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:30:18] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:30:21] RECOVERY - puppet111 conntrack_table_size on puppet111 is OK: OK: nf_conntrack is 0 % full [20:30:30] RECOVERY - puppet111 SSH on puppet111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:30:41] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:30:47] RECOVERY - cloud11 SMART on cloud11 is OK: OK: [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean [20:30:53] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:31:09] RECOVERY - prometheus101 APT on prometheus101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:31:09] PROBLEM - prometheus101 Puppet on prometheus101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:31:18] RECOVERY - puppet111 NTP time on puppet111 is OK: NTP OK: Offset -0.000880420208 secs [20:31:31] PROBLEM - db101 Puppet on db101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:31:32] RECOVERY - puppet111 Disk Space on puppet111 is OK: DISK OK - free space: / 20912 MB (78% inode=91%); [20:31:40] PROBLEM - es111 Puppet on es111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:32:00] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:32:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.30, 8.46, 8.73 [20:32:08] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:32:16] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:33:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.76, 9.64, 9.27 [20:33:35] PROBLEM - mon111 Puppet on mon111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:33:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:33:53] PROBLEM - es101 Puppet on es101 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 4 minutes ago with 4 failures. Failed resources (up to 3 shown): File[/home/reception],File[/home/void],File[wildcard.miraheze.org-2020-2],File[wildcard.miraheze.org-2020-2_private] [20:34:00] PROBLEM - graylog121 Puppet on graylog121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:34:34] PROBLEM - gluster111 Puppet on gluster111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:35:38] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:35:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.19, 8.71, 8.35 [20:36:06] PROBLEM - gluster121 Puppet on gluster121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:38:33] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.54, 8.59, 8.11 [20:40:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.75, 9.65, 8.53 [20:41:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.52, 9.49, 8.95 [20:41:10] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.99, 10.24, 8.96 [20:41:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:43:35] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.74, 10.37, 9.03 [20:44:10] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.86, 9.61, 9.09 [20:44:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.53, 9.64, 8.70 [20:46:10] RECOVERY - es101 Puppet on es101 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [20:47:21] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.05, 9.98, 9.32 [20:48:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.88, 10.19, 9.20 [20:50:18] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:50:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.02, 10.19, 9.42 [20:50:27] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:50:46] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:50:50] RECOVERY - mw112 Puppet on mw112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:50:53] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:51:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.33, 9.77, 9.21 [20:51:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.05, 9.07, 9.01 [20:51:53] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:51:55] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [20:51:56] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:51:57] RECOVERY - mem101 Puppet on mem101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:52:14] RECOVERY - prometheus101 Puppet on prometheus101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:52:54] RECOVERY - es111 Puppet on es111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:53:13] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:53:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.08, 9.78, 9.36 [20:53:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.68, 9.62, 9.30 [20:53:46] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.96, 9.31, 9.16 [20:54:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.15, 9.73, 9.27 [20:54:44] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:54:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.04, 9.00, 8.99 [20:54:58] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:55:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.34, 9.48, 9.60 [20:56:19] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [20:56:33] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:56:41] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.90, 9.83, 9.36 [20:56:49] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [20:58:11] RECOVERY - mon111 Puppet on mon111 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [20:58:47] RECOVERY - graylog121 Puppet on graylog121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:59:01] RECOVERY - gluster111 Puppet on gluster111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:59:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.21, 9.02, 9.25 [21:00:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.43, 8.86, 9.06 [21:00:25] RECOVERY - gluster121 Puppet on gluster121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:03:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.72, 9.60, 9.30 [21:03:27] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:03:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.56, 7.78, 8.38 [21:05:42] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.04, 9.38, 9.40 [21:09:48] PROBLEM - puppet111 Current Load on puppet111 is WARNING: WARNING - load average: 1.31, 1.28, 5.71 [21:10:21] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.47, 10.33, 9.62 [21:11:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.93, 9.33, 9.02 [21:11:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.95, 9.89, 9.68 [21:12:45] RECOVERY - puppet111 Current Load on puppet111 is OK: OK - load average: 1.31, 1.44, 5.01 [21:13:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.64, 9.93, 9.59 [21:13:44] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [21:14:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.48, 10.81, 10.09 [21:16:08] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.88, 9.90, 9.60 [21:17:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.29, 9.34, 9.05 [21:20:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.87, 9.32, 9.33 [21:20:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.18, 9.17, 8.79 [21:21:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.09, 9.31, 9.70 [21:23:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.12, 9.70, 9.79 [21:26:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:26:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.01, 9.49, 9.28 [21:26:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.04, 9.51, 9.58 [21:26:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.25, 9.33, 9.62 [21:29:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.51, 8.60, 8.97 [21:29:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.16, 9.66, 9.70 [21:31:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.85, 9.44, 9.82 [21:32:32] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.35, 9.53, 9.28 [21:32:48] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.23, 8.05, 8.44 [21:32:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.73, 9.27, 9.54 [21:34:12] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.65, 10.31, 10.11 [21:35:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.55, 8.96, 9.10 [21:35:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:37:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.19, 9.44, 9.81 [21:38:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.88, 10.01, 9.47 [21:38:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.16, 9.49, 9.44 [21:39:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.69, 9.62, 9.13 [21:40:02] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.14, 9.18, 9.61 [21:41:17] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:41:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.80, 9.70, 9.44 [21:41:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.87, 8.70, 9.18 [21:43:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.88, 10.40, 9.61 [21:44:22] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.29, 9.86, 9.55 [21:44:55] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.83, 9.47, 9.38 [21:45:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.40, 7.70, 8.46 [21:45:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.28, 9.50, 9.72 [21:47:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.58, 9.36, 9.42 [21:50:37] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.34, 10.55, 9.83 [21:50:57] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.32, 8.94, 9.25 [21:52:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.66, 9.68, 9.68 [21:55:39] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.56, 8.82, 8.47 [21:57:00] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.17, 9.63, 9.47 [21:57:49] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.01, 10.04, 9.85 [21:58:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.51, 9.69, 9.64 [21:58:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.42, 9.17, 8.68 [21:59:16] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.18, 9.21, 9.24 [22:01:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.93, 9.29, 9.46 [22:01:43] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.19, 10.41, 9.25 [22:02:16] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.66, 9.38, 9.30 [22:02:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.76, 9.72, 9.87 [22:03:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.02, 8.79, 9.24 [22:03:51] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.56, 9.91, 9.90 [22:06:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 9.84, 10.07, 9.98 [22:07:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.31, 8.42, 8.80 [22:09:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.69, 9.79, 9.90 [22:11:02] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.84, 9.79, 9.40 [22:12:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.08, 9.79, 9.37 [22:12:39] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 9.89, 10.25, 10.09 [22:14:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.69, 9.14, 9.25 [22:15:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.99, 9.34, 9.27 [22:16:47] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.75, 9.53, 9.04 [22:17:15] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.32, 9.77, 9.43 [22:17:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.36, 4.58, 3.48 [22:18:18] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.63, 10.39, 9.66 [22:20:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.56, 9.53, 9.21 [22:23:20] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.82, 9.79, 9.60 [22:23:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.12, 8.92, 9.08 [22:23:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.68, 3.44 [22:25:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.57, 9.18, 8.91 [22:25:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.63, 9.44, 9.36 [22:26:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.09, 9.69, 9.58 [22:26:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.50, 2.68, 3.09 [22:27:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.60, 9.80, 9.74 [22:27:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.56, 8.97, 9.80 [22:28:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.55, 9.29, 8.99 [22:28:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.03, 9.60, 9.38 [22:29:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.72, 8.13, 8.99 [22:31:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.97, 9.33, 9.34 [22:40:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.78, 8.68, 8.68 [22:41:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.30, 9.39, 8.94 [22:41:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.28, 8.98, 8.96 [22:42:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.66, 10.21, 9.64 [22:43:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.12, 8.58, 8.68 [22:44:08] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.98, 8.78, 8.79 [22:44:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.96, 8.82, 8.93 [22:45:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.56, 8.79, 9.21 [22:45:33] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.20, 9.81, 9.45 [22:48:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.71, 8.17, 8.48 [22:51:25] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.96, 9.80, 9.57 [22:55:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:57:13] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.31, 7.62, 8.41 [22:58:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.64, 7.43, 8.14 [22:58:56] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.89, 7.91, 8.44 [23:00:01] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:02] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:07:24] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.60, 9.16, 8.96 [23:07:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:10:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.89, 8.52, 8.29 [23:10:27] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.84, 9.13, 9.01 [23:11:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:11:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.67, 8.57, 8.60 [23:11:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.36, 10.17, 9.28 [23:13:26] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.77, 9.67, 9.18 [23:14:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.60, 8.59, 8.42 [23:16:14] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.96, 9.39, 8.70 [23:16:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.92, 9.06, 9.09 [23:17:39] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.63, 8.41, 8.40 [23:17:49] !log [@test101] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 1067s [23:17:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:21:00] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1258s [23:21:03] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.61, 9.63, 9.60 [23:21:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.85, 9.73, 9.06 [23:26:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:26:51] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.83, 8.16, 8.43 [23:28:27] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.50, 7.71, 8.33 [23:28:27] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.17, 8.91, 8.77 [23:31:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.61, 8.38, 8.59 [23:33:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:35:43] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.08, 7.71, 8.50 [23:36:07] PROBLEM - cp31 Disk Space on cp31 is WARNING: DISK WARNING - free space: / 4286 MB (10% inode=96%); [23:38:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.83, 8.49, 8.55 [23:40:15] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.07, 8.16, 8.48 [23:41:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.63, 8.88, 8.36 [23:42:29] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.12, 8.85, 8.75 [23:44:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.56, 9.82, 9.08 [23:45:24] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.30, 9.21, 8.89 [23:47:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.81, 9.43, 9.07 [23:48:24] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.37, 9.29, 9.02 [23:49:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.68, 8.86, 8.51 [23:51:25] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.83, 10.08, 9.34 [23:53:17] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.16, 9.27, 9.09 [23:54:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.62, 9.62, 9.32 [23:56:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.73, 9.15, 9.05 [23:56:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.57, 8.02, 8.23 [23:58:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki