[00:00:10] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.72, 1.27, 1.29 [00:02:32] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 8340 MB (21% inode=96%); [00:05:26] PROBLEM - mon111 Current Load on mon111 is WARNING: WARNING - load average: 4.65, 5.13, 4.60 [00:05:32] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 533 failures. Last run 4 minutes ago with 533 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [00:08:12] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.59, 9.15, 8.52 [00:08:28] RECOVERY - mon111 Current Load on mon111 is OK: OK - load average: 4.11, 4.61, 4.48 [00:11:20] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.74, 9.42, 8.75 [00:14:36] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.57, 10.32, 9.23 [00:17:49] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.82, 9.91, 9.25 [00:30:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:30:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.60, 9.34, 9.85 [00:31:37] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [00:33:26] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.06, 8.55, 8.08 [00:33:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.39, 10.45, 10.18 [00:34:00] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.97, 7.73, 8.38 [00:36:32] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.33, 7.98, 7.95 [00:40:31] PROBLEM - mon111 Current Load on mon111 is CRITICAL: CRITICAL - load average: 6.91, 5.39, 4.94 [00:43:37] RECOVERY - mon111 Current Load on mon111 is OK: OK - load average: 4.32, 4.76, 4.77 [00:48:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.64, 9.36, 9.88 [00:50:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.19, 9.02, 9.76 [00:51:16] PROBLEM - mon111 Current Load on mon111 is CRITICAL: CRITICAL - load average: 6.11, 5.85, 5.34 [00:52:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.22, 9.33, 9.74 [00:53:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.62, 9.64, 9.85 [00:54:19] PROBLEM - mon111 Current Load on mon111 is WARNING: WARNING - load average: 4.27, 5.17, 5.16 [00:55:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.46, 9.11, 9.59 [00:55:08] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [00:56:30] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.76, 8.56, 8.16 [00:59:53] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.42, 7.36, 7.80 [01:06:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.44, 9.45, 9.84 [01:07:07] RECOVERY - mon111 Current Load on mon111 is OK: OK - load average: 2.75, 4.28, 4.81 [01:10:47] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.64, 9.82, 9.36 [01:15:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:19:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.96, 9.92, 9.73 [01:20:26] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [01:21:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:29:03] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.42, 10.14, 9.53 [01:32:13] PROBLEM - mon111 Current Load on mon111 is WARNING: WARNING - load average: 4.15, 5.41, 5.06 [01:32:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.03, 9.69, 9.44 [01:35:14] RECOVERY - mon111 Current Load on mon111 is OK: OK - load average: 3.26, 4.86, 4.93 [01:35:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.20, 8.96, 9.17 [01:38:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.16, 9.21, 9.22 [01:42:38] PROBLEM - mon111 Current Load on mon111 is WARNING: WARNING - load average: 5.41, 5.16, 5.01 [01:44:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.51, 9.67, 9.79 [01:45:38] RECOVERY - mon111 Current Load on mon111 is OK: OK - load average: 3.95, 4.94, 4.98 [01:50:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.83, 9.36, 9.44 [01:51:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:02:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:03:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.36, 11.51, 10.27 [02:05:13] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.91, 3.49, 2.25 [02:10:11] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.63, 9.66, 9.95 [02:13:17] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.88, 10.60, 10.28 [02:19:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.08, 9.33, 9.90 [02:22:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:23:31] PROBLEM - mon111 Current Load on mon111 is WARNING: WARNING - load average: 5.71, 5.34, 5.06 [02:24:09] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.65, 7.99, 8.45 [02:26:31] RECOVERY - mon111 Current Load on mon111 is OK: OK - load average: 4.17, 4.79, 4.89 [02:29:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:31:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.23, 9.14, 8.99 [02:31:37] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.78, 9.36, 9.40 [02:33:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.04, 8.42, 7.99 [02:36:52] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.46, 7.63, 7.77 [02:37:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.57, 9.31, 9.02 [02:41:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.11, 8.73, 9.34 [02:43:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.32, 9.66, 9.29 [02:44:00] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.30, 10.16, 9.78 [02:47:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.97, 9.60, 9.66 [02:56:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.17, 8.98, 9.17 [03:01:48] PROBLEM - mon111 Current Load on mon111 is WARNING: WARNING - load average: 4.50, 5.40, 5.07 [03:03:11] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.93, 9.62, 9.45 [03:04:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:04:52] RECOVERY - mon111 Current Load on mon111 is OK: OK - load average: 3.66, 4.41, 4.72 [03:05:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.52, 9.35, 9.09 [03:06:16] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.07, 10.12, 9.67 [03:07:24] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.50, 8.71, 8.20 [03:08:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 8.59, 3.67, 2.11 [03:09:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:12:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.75, 9.86, 9.76 [03:14:40] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.86, 1.78, 1.74 [03:15:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.41, 11.11, 10.27 [03:17:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.22, 1.47, 1.62 [03:19:35] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.55, 8.11, 8.27 [03:21:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.41, 8.97, 9.60 [03:23:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.26, 9.38, 9.70 [03:28:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.81, 10.00, 9.82 [03:29:24] PROBLEM - mon111 Current Load on mon111 is CRITICAL: CRITICAL - load average: 6.50, 5.45, 4.86 [03:35:50] PROBLEM - mon111 Current Load on mon111 is WARNING: WARNING - load average: 5.04, 5.57, 5.17 [03:41:29] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.38, 2.25, 1.54 [03:42:23] RECOVERY - mon111 Current Load on mon111 is OK: OK - load average: 4.78, 5.02, 5.02 [03:42:47] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.93, 11.83, 10.12 [03:44:43] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.53, 1.54, 1.40 [03:50:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.20, 9.57, 10.00 [03:54:05] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.77, 10.29, 10.21 [03:57:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.09, 9.69, 9.98 [03:58:51] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.02, 1.36, 1.97 [04:01:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.50, 9.17, 9.84 [04:03:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.08, 9.04, 9.49 [04:04:50] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.09, 1.14, 1.66 [04:06:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.60, 9.11, 9.44 [04:07:10] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.42, 9.97, 9.90 [04:09:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.55, 10.20, 9.82 [04:12:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.68, 9.87, 9.80 [04:15:42] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.00, 10.34, 9.99 [04:27:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.63, 9.38, 9.94 [04:29:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.81, 8.51, 9.74 [04:30:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.27, 8.92, 9.60 [04:33:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.22, 9.66, 9.73 [04:34:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:35:53] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.29, 9.74, 9.84 [04:36:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.93, 9.09, 9.49 [04:38:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.13, 8.85, 9.50 [04:39:52] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.85, 9.86, 9.71 [04:43:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:47:45] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.23, 10.68, 8.99 [04:47:47] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.61, 9.40, 9.45 [04:51:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.47, 10.01, 9.51 [04:56:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.79, 9.85, 9.72 [04:56:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.85, 9.87, 9.60 [04:59:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.64, 10.47, 9.90 [05:13:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:15:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.22, 9.55, 9.45 [05:17:51] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.96, 8.53, 9.88 [05:19:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:23:34] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.41, 9.52, 9.98 [05:24:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.47, 9.91, 9.78 [05:27:13] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.81, 10.22, 9.91 [05:30:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.53, 9.82, 9.85 [05:32:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.11, 7.65, 8.42 [05:33:23] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.78, 10.52, 10.10 [05:34:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:39:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.52, 8.37, 9.37 [05:40:06] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.66, 8.24, 8.45 [05:40:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.41, 8.54, 9.83 [05:41:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.93, 9.81, 9.64 [05:43:04] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.74, 7.68, 8.24 [05:45:11] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.15, 9.31, 9.86 [05:46:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:47:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.11, 9.34, 9.50 [05:48:23] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.71, 7.23, 8.36 [05:58:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.62, 7.43, 8.42 [06:02:21] PROBLEM - mon111 Current Load on mon111 is WARNING: WARNING - load average: 3.93, 5.18, 4.61 [06:03:00] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.93, 9.71, 9.41 [06:04:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.13, 9.21, 8.64 [06:05:23] RECOVERY - mon111 Current Load on mon111 is OK: OK - load average: 4.12, 4.81, 4.57 [06:05:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.95, 8.76, 8.73 [06:08:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.06, 8.94, 8.77 [06:09:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.81, 9.40, 9.45 [06:10:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.88, 8.42, 8.58 [06:10:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.17, 7.81, 8.37 [06:11:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.02, 8.59, 8.72 [06:12:14] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.06, 9.87, 9.58 [06:17:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.75, 9.13, 8.80 [06:22:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.38, 9.46, 8.95 [06:23:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.01, 9.19, 8.85 [06:26:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.72, 8.69, 8.73 [06:28:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.96, 9.13, 9.04 [06:32:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.26, 8.10, 8.45 [06:32:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.62, 9.41, 9.08 [06:33:17] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.08, 9.40, 9.96 [06:36:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.45, 8.39, 8.74 [06:37:46] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.62, 9.86, 9.17 [06:39:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.74, 8.99, 8.86 [06:40:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.35, 9.67, 9.24 [06:42:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.62, 8.91, 8.90 [06:42:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.95, 8.76, 8.68 [06:43:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.87, 10.90, 9.81 [06:45:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.38, 9.91, 9.32 [06:48:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.37, 9.87, 9.41 [06:48:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.27, 10.94, 10.17 [06:50:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.33, 9.60, 9.66 [06:51:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.47, 7.84, 8.35 [06:54:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.04, 8.81, 9.53 [07:05:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.94, 8.84, 8.85 [07:06:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:06:36] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.74, 7.85, 8.45 [07:08:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.01, 8.13, 8.56 [07:11:06] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.41, 7.84, 8.37 [07:12:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.26, 7.45, 8.35 [07:47:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:52:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:08:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:13:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:16:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:19:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.26, 8.83, 7.56 [08:22:49] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.53, 7.86, 7.44 [08:26:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:33:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:38:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:45:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:50:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:51:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.16, 9.10, 8.37 [08:54:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:54:51] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.92, 7.97, 8.04 [08:59:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:01:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:06:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:31:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.51, 8.58, 7.87 [10:32:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:34:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.88, 7.53, 7.60 [10:37:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:37:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.64, 8.59, 7.92 [10:40:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.70, 7.84, 7.76 [10:47:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:52:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:09:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:19:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:35:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:45:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:57:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:07:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.59, 9.21, 8.24 [12:08:56] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.81, 1.58, 1.17 [12:10:38] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.34, 8.29, 8.06 [12:11:56] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.47, 1.36, 1.14 [12:17:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:20:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.55, 8.92, 8.31 [12:21:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:29:32] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.47, 8.04, 8.17 [12:39:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.19, 9.74, 9.06 [12:46:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:50:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:55:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:00:38] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.16, 8.01, 8.48 [13:02:34] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.77, 9.43, 8.58 [13:05:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.67, 9.01, 8.57 [13:15:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:17:35] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.21, 8.41, 8.49 [13:20:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:25:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:30:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:36:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:38:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.01, 9.37, 8.76 [13:50:29] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.62, 9.77, 9.16 [13:51:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.43, 9.28, 8.74 [13:53:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.77, 9.16, 8.53 [13:53:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.71, 9.41, 9.15 [13:54:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 9.22, 10.12, 9.19 [13:56:45] PROBLEM - mon111 Current Load on mon111 is WARNING: WARNING - load average: 5.89, 5.54, 5.03 [13:57:11] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.87, 8.01, 8.32 [13:59:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.75, 7.97, 8.22 [14:00:37] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.42, 8.81, 9.05 [14:01:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:02:43] PROBLEM - mon111 Current Load on mon111 is CRITICAL: CRITICAL - load average: 6.23, 5.69, 5.26 [14:02:59] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.32, 7.68, 8.44 [14:05:39] RECOVERY - mon111 Current Load on mon111 is OK: OK - load average: 4.22, 5.03, 5.08 [14:06:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.27, 10.01, 9.05 [14:07:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:09:39] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.54, 9.61, 9.12 [14:10:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.11, 10.81, 9.68 [14:12:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.39, 10.02, 8.92 [14:12:42] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.50, 9.58, 9.22 [14:13:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.23, 9.72, 9.45 [14:19:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.71, 9.82, 9.36 [14:19:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.97, 10.88, 10.01 [14:21:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.12, 9.04, 8.22 [14:24:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.53, 9.68, 8.61 [14:27:38] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.31, 8.40, 8.32 [14:27:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.50, 9.72, 9.78 [14:30:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.59, 10.82, 10.18 [14:42:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.83, 9.04, 8.43 [14:45:02] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.39, 8.77, 8.51 [14:51:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.47, 7.71, 8.20 [14:51:52] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.90, 9.54, 8.88 [14:58:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.70, 9.59, 9.13 [14:58:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.37, 9.23, 8.76 [15:01:08] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.76, 10.07, 9.36 [15:07:18] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.93, 9.69, 9.40 [15:11:08] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.38, 9.30, 8.97 [15:14:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.99, 9.22, 9.02 [15:23:58] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.68, 8.89, 8.82 [15:26:35] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.45, 7.86, 8.39 [15:27:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.64, 8.61, 8.71 [15:30:01] PROBLEM - mon111 Current Load on mon111 is WARNING: WARNING - load average: 4.89, 5.28, 4.84 [15:33:05] RECOVERY - mon111 Current Load on mon111 is OK: OK - load average: 3.52, 4.48, 4.60 [15:33:29] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.37, 7.97, 8.36 [15:34:08] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.90, 8.59, 8.57 [15:37:15] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.86, 8.34, 8.47 [15:44:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.42, 8.31, 8.52 [15:47:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.26, 8.91, 8.79 [15:48:02] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.35, 8.27, 8.49 [15:48:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.22, 9.26, 9.95 [15:50:32] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.59, 8.00, 8.47 [15:51:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.31, 9.85, 10.06 [15:54:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.80, 9.13, 9.76 [15:57:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.05, 8.64, 9.74 [16:00:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.68, 8.18, 9.93 [16:01:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.58, 9.10, 9.63 [16:03:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.31, 9.35, 10.02 [16:04:54] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.45, 9.29, 9.33 [16:06:07] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.27, 8.98, 8.45 [16:07:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.69, 8.90, 9.16 [16:08:33] RECOVERY - es111 Disk Space on es111 is OK: DISK OK - free space: / 261278 MB (98% inode=99%); [16:09:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.00, 9.09, 8.57 [16:12:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.87, 9.71, 9.98 [16:14:07] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 4.67, 5.52, 4.56 [16:14:23] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.77, 7.44, 8.42 [16:15:53] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.38, 9.66, 9.85 [16:17:22] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.13, 3.61, 3.94 [16:18:41] !log deleting graylog_1 index on es111 [16:19:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:22:27] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.27, 11.18, 9.79 [16:25:22] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 7.91, 6.77, 5.50 [16:27:00] [02miraheze/MatomoAnalytics] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.22.1 [+0/-0/±2] 13https://github.com/miraheze/MatomoAnalytics/commit/8c1d2ecf6bc1 [16:27:01] [02miraheze/MatomoAnalytics] 07dependabot[bot] 038c1d2ec - Bump eslint-config-wikimedia from 0.21.0 to 0.22.1 [16:27:03] [02MatomoAnalytics] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.22.1 - 13https://github.com/miraheze/MatomoAnalytics [16:27:04] [02MatomoAnalytics] 07dependabot[bot] opened pull request 03#70: Bump eslint-config-wikimedia from 0.21.0 to 0.22.1 - 13https://github.com/miraheze/MatomoAnalytics/pull/70 [16:27:06] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#70: Bump eslint-config-wikimedia from 0.21.0 to 0.22.1 - 13https://github.com/miraheze/MatomoAnalytics/pull/70 [16:27:07] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#70: Bump eslint-config-wikimedia from 0.21.0 to 0.22.1 - 13https://github.com/miraheze/MatomoAnalytics/pull/70 [16:27:09] [02MatomoAnalytics] 07dependabot[bot] commented on pull request 03#69: Bump eslint-config-wikimedia from 0.21.0 to 0.22.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/69#issuecomment-1033948231 [16:27:10] [02MatomoAnalytics] 07dependabot[bot] closed pull request 03#69: Bump eslint-config-wikimedia from 0.21.0 to 0.22.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/69 [16:27:12] [02MatomoAnalytics] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.22.0 - 13https://github.com/miraheze/MatomoAnalytics [16:27:13] [02miraheze/MatomoAnalytics] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.22.0 [16:29:04] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.46, 6.73, 7.85 [16:31:31] miraheze/MatomoAnalytics - dependabot[bot] the build passed. [16:46:52] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.81, 8.75, 8.33 [16:49:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.97, 9.26, 9.97 [16:52:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.95, 9.80, 10.02 [16:56:41] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.30, 8.43, 8.34 [16:59:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.44, 8.73, 9.97 [17:02:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.77, 9.16, 9.68 [17:07:35] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.88, 9.49, 8.76 [17:09:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.88, 10.86, 10.28 [17:11:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.71, 8.58, 8.49 [17:12:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 9.72, 10.05, 9.88 [17:13:52] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.81, 9.87, 9.17 [17:15:16] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.58, 9.25, 9.63 [17:18:28] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.18, 10.24, 9.96 [17:24:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.53, 9.82, 9.98 [17:27:36] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.24, 7.58, 8.25 [17:31:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.62, 9.47, 9.93 [17:32:30] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:32:34] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:32:59] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:33:12] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:33:20] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.64, 7.72, 8.39 [17:33:53] [02miraheze/ManageWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.22.1 [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/commit/9d3a8a06d19a [17:33:54] [02miraheze/ManageWiki] 07dependabot[bot] 039d3a8a0 - Bump eslint-config-wikimedia from 0.21.0 to 0.22.1 [17:33:56] [02ManageWiki] 07dependabot[bot] opened pull request 03#339: Bump eslint-config-wikimedia from 0.21.0 to 0.22.1 - 13https://github.com/miraheze/ManageWiki/pull/339 [17:33:57] [02ManageWiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.22.1 - 13https://github.com/miraheze/ManageWiki [17:33:59] [02ManageWiki] 07dependabot[bot] labeled pull request 03#339: Bump eslint-config-wikimedia from 0.21.0 to 0.22.1 - 13https://github.com/miraheze/ManageWiki/pull/339 [17:34:00] [02ManageWiki] 07dependabot[bot] labeled pull request 03#339: Bump eslint-config-wikimedia from 0.21.0 to 0.22.1 - 13https://github.com/miraheze/ManageWiki/pull/339 [17:34:02] [02ManageWiki] 07dependabot[bot] commented on pull request 03#338: Bump eslint-config-wikimedia from 0.21.0 to 0.22.0 - 13https://github.com/miraheze/ManageWiki/pull/338#issuecomment-1034019569 [17:34:03] [02ManageWiki] 07dependabot[bot] closed pull request 03#338: Bump eslint-config-wikimedia from 0.21.0 to 0.22.0 - 13https://github.com/miraheze/ManageWiki/pull/338 [17:34:05] [02ManageWiki] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.22.0 - 13https://github.com/miraheze/ManageWiki [17:34:06] [02miraheze/ManageWiki] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.22.0 [17:34:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.20, 9.12, 9.49 [17:35:40] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.071 second response time [17:35:44] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22220 bytes in 2.285 second response time [17:36:15] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.787 second response time [17:36:33] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.069 second response time [17:37:51] miraheze/ManageWiki - dependabot[bot] the build passed. [17:38:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.77, 9.79, 9.73 [17:40:55] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.04, 9.61, 9.53 [17:41:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.86, 10.23, 9.88 [17:44:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.82, 8.64, 9.30 [17:47:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.53, 9.76, 9.59 [17:51:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.37, 9.52, 9.55 [17:51:52] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.23, 9.70, 9.00 [17:54:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.03, 10.55, 9.96 [17:56:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.43, 9.33, 9.91 [18:02:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.21, 9.58, 9.86 [18:03:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.57, 9.04, 9.70 [18:04:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.01, 9.44, 9.84 [18:04:50] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.11, 8.71, 8.68 [18:05:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.67, 3.52, 2.95 [18:05:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.40, 9.09, 9.63 [18:06:43] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.93, 9.80, 9.82 [18:07:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.02, 10.95, 10.33 [18:07:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:07:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.48, 8.20, 8.46 [18:08:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.55, 2.87, 2.82 [18:09:22] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.07, 2.58, 1.83 [18:10:54] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.76, 2.90, 5.36 [18:12:24] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.60, 1.97, 1.71 [18:13:52] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.04, 2.09, 4.62 [18:15:30] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.14, 1.59, 1.61 [18:21:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.57, 8.20, 9.45 [18:26:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.34, 9.25, 9.86 [18:27:24] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/b99b80287c9f...ef17ad343882 [18:27:26] [02miraheze/dns] 07JohnFLewis 03ef17ad3 - add ipv6 PTR records [18:30:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.97, 7.26, 8.44 [18:32:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.63, 9.88, 9.88 [18:33:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.60, 9.62, 9.20 [18:36:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.04, 9.04, 9.03 [18:38:05] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.63, 8.04, 8.52 [18:41:14] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.01, 7.99, 8.41 [18:44:42] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.02, 9.14, 8.60 [18:45:59] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.72, 4.02, 3.17 [18:45:59] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.43, 9.75, 9.29 [18:48:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.54, 9.45, 9.96 [18:49:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.65, 9.58, 8.95 [18:49:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.09, 8.66, 8.92 [18:51:13] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.15, 9.27, 9.00 [18:51:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.44, 10.09, 10.13 [18:52:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.03, 3.36, 3.24 [18:52:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.72, 8.88, 8.81 [18:52:36] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:54:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.84, 8.56, 8.48 [18:55:01] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [18:56:10] PROBLEM - mon111 APT on mon111 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [18:57:06] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.70, 8.28, 8.40 [18:57:10] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.35, 8.86, 9.65 [18:58:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.93, 9.65, 9.93 [18:58:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.99, 9.43, 9.03 [18:59:19] PROBLEM - mon111 APT on mon111 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (3 critical updates). [18:59:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:00:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.60, 9.25, 9.62 [19:01:36] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.14, 8.80, 8.86 [19:01:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [19:04:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.62, 10.00, 9.96 [19:04:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.86, 8.96, 8.89 [19:06:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.28, 9.07, 9.41 [19:10:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.93, 9.72, 9.60 [19:10:11] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.59, 9.92, 9.34 [19:10:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.11, 9.79, 9.25 [19:13:16] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.36, 9.96, 9.49 [19:13:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.01, 9.76, 9.36 [19:17:07] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.08, 9.45, 9.28 [19:17:38] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.87, 9.73, 9.33 [19:20:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.01, 8.88, 9.07 [19:20:38] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:20:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.18, 9.11, 9.15 [19:23:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.71, 9.78, 9.38 [19:23:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.79, 9.40, 9.94 [19:24:16] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.10, 9.50, 9.25 [19:26:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.54, 9.17, 9.22 [19:27:19] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.40, 10.40, 10.23 [19:27:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.52, 9.52, 9.32 [19:28:00] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [19:34:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.63, 10.18, 9.58 [19:34:59] PROBLEM - matomo101 Puppet on matomo101 is UNKNOWN: NRPE: Unable to read output [19:35:10] PROBLEM - prometheus101 Disk Space on prometheus101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:35:12] PROBLEM - matomo101 conntrack_table_size on matomo101 is UNKNOWN: NRPE: Unable to read output [19:35:15] PROBLEM - prometheus101 APT on prometheus101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:35:25] PROBLEM - matomo101 ferm_active on matomo101 is UNKNOWN: NRPE: Unable to read output [19:35:35] PROBLEM - prometheus101 ferm_active on prometheus101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:35:37] PROBLEM - prometheus101 conntrack_table_size on prometheus101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:36:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.97, 3.64, 3.36 [19:36:27] PROBLEM - prometheus101 PowerDNS Recursor on prometheus101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:36:48] PROBLEM - prometheus101 NTP time on prometheus101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:36:50] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.21, 9.67, 9.42 [19:37:10] PROBLEM - prometheus101 Current Load on prometheus101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:37:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.08, 9.87, 9.39 [19:37:37] PROBLEM - prometheus101 Puppet on prometheus101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:38:10] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.38, 9.36, 9.95 [19:38:43] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:38:51] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [19:38:51] RECOVERY - prometheus101 Disk Space on prometheus101 is OK: DISK OK - free space: / 130404 MB (98% inode=99%); [19:38:54] RECOVERY - prometheus101 APT on prometheus101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:39:05] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [19:39:11] PROBLEM - prometheus101 conntrack_table_size on prometheus101 is UNKNOWN: NRPE: Unable to read output [19:39:12] PROBLEM - prometheus101 ferm_active on prometheus101 is UNKNOWN: NRPE: Unable to read output [19:39:55] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.55, 3.22, 3.26 [19:39:58] RECOVERY - prometheus101 PowerDNS Recursor on prometheus101 is OK: DNS OK: 0.180 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [19:40:21] RECOVERY - prometheus101 NTP time on prometheus101 is OK: NTP OK: Offset 0.002536773682 secs [19:40:39] RECOVERY - prometheus101 Current Load on prometheus101 is OK: OK - load average: 0.18, 1.00, 0.90 [19:41:05] PROBLEM - prometheus101 Puppet on prometheus101 is UNKNOWN: NRPE: Unable to read output [19:41:42] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.79, 10.36, 10.23 [19:45:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.53, 9.39, 9.56 [19:45:51] RECOVERY - prometheus101 conntrack_table_size on prometheus101 is OK: OK: nf_conntrack is 0 % full [19:46:00] RECOVERY - prometheus101 ferm_active on prometheus101 is OK: OK ferm input default policy is set [19:46:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.65, 9.65, 9.60 [19:47:29] [02miraheze/dns] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±4] 13https://github.com/miraheze/dns/compare/ef17ad343882...63684dd4c20a [19:47:30] [02miraheze/dns] 07JohnFLewis 036fd1c03 - add matomo101 [19:47:32] [02miraheze/dns] 07JohnFLewis 0363684dd - add prometheus101 [19:47:43] RECOVERY - prometheus101 Puppet on prometheus101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:48:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.19, 10.51, 9.96 [19:50:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.61, 9.66, 9.72 [19:51:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.01, 9.72, 9.80 [19:55:07] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/63684dd4c20a...15ae6d0555ba [19:55:08] [02miraheze/dns] 07JohnFLewis 0315ae6d0 - matomo101: 306 -> 106 [19:56:41] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.03, 3.44, 2.00 [19:58:03] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.52, 9.56, 9.60 [19:58:21] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.18, 7.54, 8.36 [20:01:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.83, 8.85, 9.32 [20:02:38] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.77, 7.08, 8.12 [20:02:51] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.29, 1.53, 1.62 [20:04:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.55, 8.89, 9.71 [20:04:23] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.19, 9.28, 9.35 [20:06:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.34, 7.93, 8.47 [20:07:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.29, 9.23, 9.66 [20:07:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.69, 9.33, 9.35 [20:08:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.63, 8.66, 9.62 [20:09:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.95, 8.99, 8.61 [20:10:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.81, 8.80, 8.73 [20:13:08] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±5] 13https://github.com/miraheze/puppet/compare/3bf9bd44f83a...b2aa4741e92c [20:13:10] [02miraheze/puppet] 07JohnFLewis 03b2aa474 - deploy matomo101 [20:14:11] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.54, 9.67, 9.74 [20:14:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.28, 9.89, 9.45 [20:14:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 9.81, 10.44, 9.66 [20:14:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.47, 9.40, 9.61 [20:17:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.67, 9.50, 9.67 [20:19:58] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b2aa4741e92c...68542c775aa8 [20:20:00] [02miraheze/puppet] 07JohnFLewis 0368542c7 - push matomo.mh.o to matomo101 [20:20:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.36, 9.57, 9.42 [20:21:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.05, 9.99, 9.81 [20:21:48] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/68542c775aa8...2101c6103d7f [20:21:48] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_matomo] [20:21:49] [02miraheze/puppet] 07JohnFLewis 032101c61 - add matomo to db firewall [20:22:10] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.62, 7.90, 8.35 [20:23:52] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.38, 10.75, 10.17 [20:24:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.81, 9.72, 9.81 [20:24:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.45, 9.49, 9.63 [20:26:25] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.60, 8.11, 8.43 [20:27:00] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.79, 9.57, 9.41 [20:27:18] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.91, 10.36, 10.05 [20:30:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.44, 9.97, 9.98 [20:30:53] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.49, 9.95, 9.12 [20:31:06] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.33, 9.64, 9.59 [20:31:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.78, 9.33, 9.82 [20:32:35] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2101c6103d7f...3cb9446c6a7e [20:32:37] [02miraheze/puppet] 07JohnFLewis 033cb9446 - add http_proxy [20:33:41] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 289 bytes in 0.257 second response time [20:33:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.17, 10.63, 10.24 [20:36:53] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 2828 bytes in 0.240 second response time [20:36:53] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.48, 9.84, 9.01 [20:37:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.94, 9.39, 9.84 [20:37:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.45, 3.26, 3.14 [20:38:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.87, 9.60, 9.73 [20:40:11] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.94, 9.89, 9.20 [20:40:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.23, 10.29, 10.12 [20:40:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.97, 9.38, 9.67 [20:40:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.54, 8.86, 9.09 [20:41:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.15, 9.47, 9.66 [20:43:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.90, 10.02, 9.86 [20:43:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.63, 3.12, 3.17 [20:46:20] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.34, 9.78, 9.33 [20:46:46] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.36, 9.99, 9.48 [20:47:42] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.84, 10.06, 9.74 [20:49:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.69, 9.23, 9.19 [20:49:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.36, 9.41, 9.87 [20:50:18] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:51:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.83, 9.57, 9.90 [20:53:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.50, 9.76, 9.61 [20:54:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.53, 10.33, 10.13 [20:54:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 0.90, 2.04, 1.71 [20:55:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 5.71, 8.52, 9.50 [20:57:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.08, 9.06, 9.76 [20:57:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.91, 9.14, 9.68 [20:57:59] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.16, 1.50, 1.55 [21:00:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.62, 9.76, 9.83 [21:01:49] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.55, 9.26, 9.03 [21:02:27] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.40, 9.37, 9.18 [21:04:59] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:05:31] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:06:24] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:06:44] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.32, 10.07, 9.64 [21:07:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.82, 9.82, 9.81 [21:07:13] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 3.56, 2.75, 1.56 [21:07:18] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:07:59] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:08:32] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.53, 9.33, 9.06 [21:08:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.19, 8.70, 8.85 [21:09:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.49, 9.22, 9.32 [21:09:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.18, 9.01, 9.35 [21:11:28] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.07, 6.86, 8.05 [21:11:41] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.41, 9.87, 9.25 [21:14:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:14:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.56, 8.83, 8.95 [21:14:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.32, 9.93, 9.50 [21:21:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:25:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.99, 7.11, 8.21 [21:25:54] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.04, 8.35, 8.50 [21:26:03] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.11, 9.24, 9.13 [21:29:06] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.17, 7.17, 7.99 [21:29:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.98, 9.17, 9.12 [21:31:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:33:45] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.65, 7.32, 8.39 [21:33:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.86, 8.82, 8.74 [21:36:51] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.05, 7.85, 8.38 [21:37:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.92, 8.71, 7.90 [21:40:17] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.10, 8.10, 7.82 [21:41:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.48, 7.94, 8.41 [21:51:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:52:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.42, 8.87, 8.11 [21:55:47] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.72, 7.91, 7.88 [21:56:08] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.26, 9.12, 8.50 [21:56:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.61, 9.15, 8.58 [21:59:16] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.12, 8.37, 8.34 [21:59:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.20, 8.85, 8.58 [22:06:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.42, 8.81, 8.66 [22:07:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.23, 8.99, 8.34 [22:07:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.80, 8.84, 8.27 [22:10:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.70, 9.00, 8.48 [22:10:47] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.89, 7.88, 7.99 [22:13:02] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.98, 8.33, 8.49 [22:18:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.20, 10.09, 9.26 [22:18:37] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.89, 9.62, 8.70 [22:20:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.73, 8.89, 8.76 [22:21:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.60, 9.40, 9.14 [22:21:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.15, 9.39, 8.75 [22:21:52] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.06, 9.48, 8.84 [22:22:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.66, 8.89, 8.23 [22:24:02] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.45, 9.38, 8.96 [22:24:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.63, 9.94, 9.39 [22:26:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:27:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.31, 9.07, 8.91 [22:28:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.12, 9.31, 9.25 [22:28:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.82, 9.80, 9.09 [22:30:32] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.11, 9.70, 9.17 [22:31:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.58, 9.21, 9.00 [22:32:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.75, 9.29, 8.62 [22:33:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:34:51] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.04, 2.35, 1.67 [22:35:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.52, 9.04, 8.62 [22:35:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.13, 10.09, 9.40 [22:36:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.23, 9.16, 9.23 [22:38:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:40:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.90, 8.63, 8.72 [22:41:04] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.99, 9.25, 8.98 [22:41:59] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.75, 7.87, 8.27 [22:42:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.69, 9.21, 9.23 [22:43:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:43:45] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.13, 7.91, 8.43 [22:44:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.83, 9.11, 8.99 [22:45:14] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.04, 9.22, 9.21 [22:47:25] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.94, 9.88, 9.26 [22:48:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.88, 9.69, 9.43 [22:48:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:49:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.11, 8.09, 8.33 [22:50:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.51, 8.76, 8.94 [22:51:18] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.04, 8.85, 8.67 [22:51:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.58, 10.18, 9.63 [22:53:32] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.34, 1.82, 1.92 [22:54:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.21, 8.44, 8.55 [22:54:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:56:16] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.31, 7.77, 8.43 [22:57:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.05, 8.94, 8.71 [22:57:56] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.48, 9.80, 9.66 [22:59:45] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.11, 1.32, 1.65 [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:04] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.68, 8.38, 8.54 [23:03:16] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.51, 9.78, 9.17 [23:04:05] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.57, 9.89, 9.64 [23:06:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.22, 9.35, 9.10 [23:09:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:09:36] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.13, 9.63, 9.23 [23:10:11] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.13, 8.71, 8.59 [23:10:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.83, 9.50, 9.52 [23:13:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:16:50] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.60, 2.72, 2.09 [23:17:03] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.27, 9.05, 8.47 [23:18:06] !log [@test101] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 1084s [23:18:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:19:29] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.51, 8.55, 9.18 [23:19:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.97, 8.96, 8.86 [23:20:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.01, 9.57, 8.73 [23:22:51] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.86, 8.74, 9.07 [23:23:16] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.95, 10.95, 9.16 [23:23:17] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.21, 7.93, 8.46 [23:23:40] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.57, 8.75, 8.54 [23:24:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.82, 8.97, 8.64 [23:25:47] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1543s [23:25:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:26:11] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.65, 7.99, 8.74 [23:26:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.19, 9.47, 8.97 [23:26:58] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.95, 8.00, 8.25 [23:30:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.24, 7.48, 8.27 [23:30:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.91, 9.84, 9.18 [23:35:47] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.08, 7.86, 8.41 [23:37:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.64, 10.08, 9.43 [23:37:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.53, 8.44, 8.49 [23:39:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.13, 0.78, 1.72 [23:39:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.32, 7.73, 8.40 [23:42:26] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.53, 0.57, 1.45 [23:42:28] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.61, 8.73, 8.63 [23:43:28] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.46, 9.07, 8.82 [23:45:35] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.53, 8.06, 8.40 [23:46:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.94, 9.42, 9.47