[00:03:13] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 7825 MB (20% inode=96%); [00:07:21] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 427 failures. Last run 6 minutes ago with 427 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [00:08:53] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.00, 1.52, 1.88 [00:08:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.78, 8.82, 8.27 [00:12:02] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.68, 1.96, 1.98 [00:12:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.82, 7.84, 8.00 [00:15:06] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.54, 1.44, 1.78 [00:18:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.98, 1.75, 1.79 [00:21:37] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.87, 1.75, 1.77 [00:22:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.66, 9.68, 9.93 [00:25:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.41, 10.44, 10.17 [00:28:16] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.61, 1.06, 1.46 [00:32:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.83, 9.32, 9.84 [00:32:47] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:46:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:57:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:59:06] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.96, 7.81, 8.45 [01:06:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.60, 9.19, 8.94 [01:15:08] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.53, 7.92, 8.43 [01:17:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:20:43] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.29, 2.08, 1.52 [01:23:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:27:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.17, 8.21, 9.43 [01:28:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:30:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.64, 8.73, 9.37 [01:33:23] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.77, 8.02, 9.00 [01:42:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.72, 1.21, 1.91 [01:45:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.85, 7.77, 8.31 [01:51:03] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.26, 1.81, 1.89 [01:54:02] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.57, 1.74, 1.85 [01:55:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.75, 9.37, 9.01 [01:57:00] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.88, 1.36, 1.69 [01:58:16] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.95, 10.54, 9.56 [02:01:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.52, 9.23, 9.20 [02:07:06] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.03, 6.97, 8.16 [02:09:18] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:12:13] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.016 second response time [02:19:32] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 3.00, 3.27, 7.61 [02:25:24] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 2.51, 2.91, 6.10 [02:34:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:44:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:45:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.13, 10.11, 8.96 [02:48:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.28, 9.24, 8.33 [02:51:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.70, 9.20, 9.06 [02:54:05] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.28, 8.17, 8.21 [02:55:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:00:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:04:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:06:19] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.65, 9.24, 8.93 [03:09:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.80, 9.46, 9.06 [03:09:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.79, 8.69, 7.83 [03:12:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.19, 4.03, 3.39 [03:15:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.47, 8.82, 8.06 [03:16:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.21, 2.39, 1.79 [03:18:13] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.01, 2.13, 1.42 [03:18:13] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.73, 9.20, 8.88 [03:18:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.21, 8.56, 8.12 [03:19:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.46, 8.57, 7.74 [03:19:24] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.27, 1.87, 1.69 [03:19:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:21:12] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.63, 9.44, 9.04 [03:21:13] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.55, 1.38, 1.25 [03:21:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.37, 7.69, 7.87 [03:22:13] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.49, 8.36, 7.82 [03:22:26] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.37, 1.51, 1.57 [03:24:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.67, 3.74, 3.64 [03:27:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.71, 3.63 [03:27:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:30:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.24, 2.67, 3.25 [03:32:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:33:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.05, 8.55, 8.39 [03:35:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.26, 8.14, 7.86 [03:36:28] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.71, 8.81, 8.48 [03:37:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.17, 8.86, 8.30 [03:38:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.03, 7.86, 7.80 [03:39:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:40:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.38, 8.23, 8.16 [03:42:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.16, 10.19, 9.55 [03:45:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.89, 9.51, 9.41 [03:46:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.96, 3.78, 3.32 [03:48:13] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.20, 10.05, 9.64 [03:52:32] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 7.21, 7.30, 6.24 [03:55:31] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 9.16, 8.07, 6.72 [03:55:32] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.56, 3.14, 3.20 [04:03:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.52, 9.50, 9.65 [04:07:43] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 7.45, 7.90, 7.38 [04:08:59] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.84, 3.54, 3.26 [04:10:44] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 10.08, 8.79, 7.81 [04:11:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.59, 3.14, 3.16 [04:24:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:28:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:38:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:40:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.71, 9.25, 9.97 [04:42:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.52, 9.28, 8.97 [04:42:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:43:15] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.31, 9.59, 9.96 [04:45:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.28, 8.93, 8.91 [04:46:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.24, 9.54, 9.93 [04:49:25] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 3.92, 5.46, 7.73 [04:50:34] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/resetUserTokens.php (END - exit=0) [04:50:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:52:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:53:57] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.39, 7.54, 8.31 [04:55:14] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 3.13, 4.20, 6.45 [04:58:03] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.31, 7.31, 8.36 [05:00:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.04, 3.80, 3.15 [05:03:53] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.57, 3.17 [05:06:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.07, 3.74, 3.29 [05:09:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.64, 3.36, 3.23 [05:38:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:43:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:01:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:06:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:09:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:14:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:51:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:56:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:12:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:17:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:24:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:25:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.10, 8.80, 7.54 [08:27:57] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.17, 7.17, 7.13 [08:29:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:40:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:50:53] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.18, 8.39, 7.05 [08:53:46] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.62, 8.47, 7.28 [08:56:40] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.05, 7.80, 7.23 [09:00:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:20:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:25:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:32:14] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/50afb08c3089...551b609549a4 [09:32:15] [02miraheze/ssl] 07Reception123 03551b609 - bitcoin21.ch -> www.bitcoin21.ch redirect [09:46:49] paladox: if we did https://phabricator.wikimedia.org/T231827#7425173 manually on affected wikis wouldn't that resolve the issue for us temporarily? [09:46:50] [url] ⚓ T231827 Data too long for column 'si_title' | phabricator.wikimedia.org [09:47:29] Yes [09:49:50] thanks! [10:07:01] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 0.59, 2.07, 1.58 [10:09:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:09:56] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.60, 1.38, 1.39 [10:11:59] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.51, 3.67, 2.33 [10:14:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:17:38] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.63, 1.69, 1.84 [10:20:29] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.55, 1.19, 1.62 [11:18:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.65, 5.35, 3.72 [11:19:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.89, 4.56, 3.17 [11:20:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.29, 3.54, 2.82 [11:22:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.72, 3.39, 2.96 [11:23:41] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.04, 2.86, 2.68 [11:24:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.60, 3.58, 3.42 [11:27:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.25, 2.97, 3.22 [11:30:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:35:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:46:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:51:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:55:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:00:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:25:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.57, 9.18, 8.21 [12:27:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:31:24] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.90, 7.46, 7.82 [12:37:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:41:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:43:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.61, 8.86, 8.41 [12:46:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.76, 7.99, 8.16 [12:51:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:57:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.54, 8.25, 7.93 [12:58:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:00:37] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.84, 7.49, 7.72 [13:03:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:14:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:19:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:28:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.69, 9.20, 8.60 [13:31:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.55, 8.21, 8.34 [13:40:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:41:47] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.58, 8.48, 8.41 [13:42:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.54, 3.33, 2.88 [13:44:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.86, 7.79, 8.17 [13:45:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:47:59] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.23, 3.66, 3.15 [13:51:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:51:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.25, 8.64, 8.41 [13:54:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.43, 3.22 [13:57:04] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.12, 3.27, 3.20 [13:58:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.46, 8.19, 8.31 [14:01:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:05:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.64, 8.41, 8.32 [14:07:56] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.57, 7.20, 7.83 [14:21:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:21:59] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.40, 8.89, 7.71 [14:22:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.11, 8.61, 8.11 [14:24:57] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.93, 8.19, 7.64 [14:25:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.61, 7.96, 7.93 [14:26:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:30:10] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.49, 8.39, 7.33 [14:33:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.33, 8.56, 8.17 [14:35:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.24, 10.25, 8.88 [14:36:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.53, 9.09, 8.42 [14:38:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.45, 9.96, 9.05 [14:39:20] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.95, 7.67, 7.58 [14:39:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.67, 9.17, 8.56 [14:41:00] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.93, 10.15, 9.26 [14:44:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.95, 9.38, 9.08 [14:46:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:48:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.36, 10.83, 9.47 [14:50:00] PROBLEM - mon111 Current Load on mon111 is WARNING: WARNING - load average: 6.00, 5.00, 4.44 [14:50:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.23, 10.77, 9.78 [14:53:03] RECOVERY - mon111 Current Load on mon111 is OK: OK - load average: 4.28, 4.69, 4.43 [14:53:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.68, 7.40, 8.12 [15:04:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.44, 9.06, 9.55 [15:05:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.05, 8.77, 8.18 [15:06:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:07:42] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.50, 10.12, 9.90 [15:08:56] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.67, 3.60, 2.73 [15:09:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.43, 9.64, 9.04 [15:10:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.51, 4.29, 3.28 [15:10:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.28, 3.92, 3.18 [15:10:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:10:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.68, 9.53, 9.71 [15:11:47] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.86, 8.09, 8.01 [15:13:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.73, 3.26 [15:13:50] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.59, 10.00, 9.82 [15:15:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:15:54] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.84, 7.71, 8.30 [15:16:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.41, 3.18, 3.10 [15:16:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.74, 8.45, 9.24 [15:18:16] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.64, 1.31, 1.93 [15:21:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:24:20] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.91, 1.04, 1.60 [15:25:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.18, 9.09, 9.89 [15:25:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.17, 3.62, 3.49 [15:28:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.37, 3.15, 3.33 [15:34:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.08, 10.83, 10.21 [15:36:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.12, 3.52, 3.47 [15:38:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.14, 4.40, 3.50 [15:41:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.74, 7.19, 8.20 [15:41:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.67, 3.36 [15:42:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.56, 2.60, 3.15 [15:46:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.72, 9.01, 8.42 [15:48:28] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.08, 8.77, 8.48 [15:50:53] PROBLEM - mon111 Current Load on mon111 is WARNING: WARNING - load average: 5.74, 5.24, 4.64 [15:50:57] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.95, 3.20, 3.28 [15:51:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.77, 9.16, 8.74 [15:53:04] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.05, 8.46, 8.42 [15:55:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.04, 3.65, 3.37 [15:57:42] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.23, 8.20, 8.44 [15:58:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.59, 3.41 [16:00:16] PROBLEM - mon111 Current Load on mon111 is CRITICAL: CRITICAL - load average: 6.17, 5.32, 4.89 [16:02:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.29, 3.21, 3.27 [16:03:41] PROBLEM - mon111 Current Load on mon111 is WARNING: WARNING - load average: 5.34, 4.98, 4.81 [16:04:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.15, 3.51, 3.49 [16:06:55] RECOVERY - mon111 Current Load on mon111 is OK: OK - load average: 4.33, 4.89, 4.83 [16:07:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.85, 3.01, 3.28 [16:07:51] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.15, 8.69, 7.94 [16:08:09] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 0.84, 2.13, 1.70 [16:09:59] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.71, 3.62, 3.44 [16:11:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.84, 1.52, 1.54 [16:14:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.27, 9.41, 8.92 [16:15:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.02, 9.44, 8.62 [16:16:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.17, 3.01, 3.29 [16:17:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.63, 8.07, 7.99 [16:20:46] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.91, 7.85, 8.35 [16:22:11] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.58, 7.93, 8.19 [16:29:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.83, 8.70, 8.51 [16:34:25] PROBLEM - mon111 Current Load on mon111 is WARNING: WARNING - load average: 4.44, 5.25, 5.07 [16:39:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.46, 7.91, 8.24 [16:40:55] RECOVERY - mon111 Current Load on mon111 is OK: OK - load average: 4.75, 5.01, 5.03 [16:43:11] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.59, 4.05, 3.67 [16:49:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.64, 3.57, 3.66 [16:52:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.29, 3.85, 3.72 [16:55:59] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.34, 3.31, 3.55 [16:56:57] PROBLEM - mon111 Current Load on mon111 is WARNING: WARNING - load average: 5.38, 5.46, 5.14 [16:59:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.94, 3.00, 3.37 [16:59:55] RECOVERY - mon111 Current Load on mon111 is OK: OK - load average: 3.77, 4.89, 4.99 [17:00:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.23, 8.10, 9.77 [17:03:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.10, 8.71, 9.70 [17:06:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:06:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.45, 9.09, 9.67 [17:09:21] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.24, 2.39, 1.69 [17:12:28] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.51, 1.51, 1.47 [17:17:50] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.31, 1.98, 1.64 [17:19:12] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.70, 9.98, 9.57 [17:20:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:20:58] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.25, 1.66, 1.58 [17:25:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.16, 9.55, 9.57 [17:28:29] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.32, 9.43, 9.49 [17:31:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.97, 9.57, 8.68 [17:31:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.67, 8.85, 9.29 [17:33:41] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.48, 8.68, 7.91 [17:33:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.40, 10.20, 8.79 [17:34:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.88, 9.96, 8.99 [17:34:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.46, 9.99, 9.61 [17:37:00] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.70, 7.82, 7.69 [17:37:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.74, 9.23, 8.68 [17:37:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.74, 9.61, 9.05 [17:39:48] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.85, 9.17, 8.17 [17:43:02] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.47, 8.09, 7.91 [17:43:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.10, 7.34, 8.11 [17:44:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.67, 8.99, 8.45 [17:47:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.93, 8.27, 8.28 [17:58:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.54, 8.61, 8.23 [17:59:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.10, 9.58, 9.24 [18:01:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.61, 8.01, 8.07 [18:02:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.05, 8.71, 8.95 [18:07:43] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 15.92, 10.62, 6.66 [18:12:00] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.20, 9.58, 9.05 [18:18:52] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.24, 2.94 [18:20:05] PROBLEM - mon111 Current Load on mon111 is WARNING: WARNING - load average: 4.51, 5.11, 4.72 [18:23:16] RECOVERY - mon111 Current Load on mon111 is OK: OK - load average: 3.97, 4.57, 4.58 [18:25:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.50, 4.24, 3.53 [18:29:25] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.46, 3.87, 3.18 [18:29:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.67, 4.49, 3.87 [18:30:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.57, 8.57, 9.91 [18:32:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.82, 3.63 [18:32:39] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.59, 3.21, 3.03 [18:32:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.96, 3.53, 3.60 [18:33:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.88, 9.72, 10.05 [18:35:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.52, 9.28, 9.89 [18:35:36] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.51, 2.94, 3.34 [18:36:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.50, 2.41, 3.13 [18:37:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.68, 9.33, 9.82 [18:38:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.49, 10.03, 10.05 [18:40:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.94, 10.38, 10.08 [18:50:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.55, 8.90, 9.73 [18:59:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 9.37, 10.29, 10.12 [19:06:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.23, 9.78, 9.94 [19:20:54] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.008 second response time [19:25:00] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.56, 9.13, 9.01 [19:25:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.73, 8.84, 7.99 [19:28:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.17, 8.60, 8.83 [19:28:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.50, 8.31, 7.93 [19:30:09] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.300 second response time [19:31:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.75, 9.44, 9.11 [19:34:25] PROBLEM - heavyironmodding.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'heavyironmodding.org' expires in 15 day(s) (Sun 20 Feb 2022 19:18:29 GMT +0000). [19:39:50] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/551b609549a4...ff0f3b8cd39e [19:39:51] [02miraheze/ssl] 07MirahezeSSLBot 03ff0f3b8 - Bot: Update SSL cert for heavyironmodding.org [19:40:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.70, 9.21, 9.51 [19:47:34] PROBLEM - parapedia.pl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'parapedia.pl' expires in 15 day(s) (Sun 20 Feb 2022 19:16:28 GMT +0000). [19:50:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:54:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ff0f3b8cd39e...b7b13e93f8e9 [19:54:49] [02miraheze/ssl] 07MirahezeSSLBot 03b7b13e9 - Bot: Update SSL cert for parapedia.pl [19:56:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:03:03] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.010 second response time [20:04:08] RECOVERY - heavyironmodding.org - LetsEncrypt on sslhost is OK: OK - Certificate 'heavyironmodding.org' will expire on Thu 05 May 2022 18:39:43 GMT +0000. [20:06:04] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.73, 11.02, 10.02 [20:06:11] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.031 second response time [20:09:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.40, 9.84, 9.77 [20:12:49] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.90, 8.81, 9.79 [20:16:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.62, 9.71, 8.58 [20:19:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.22, 10.47, 10.14 [20:19:37] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.71, 7.94, 8.07 [20:22:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.36, 9.04, 9.31 [20:28:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.56, 9.41, 9.38 [20:34:58] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.04, 9.38, 9.27 [20:38:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.22, 9.43, 9.33 [20:40:21] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:35] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 1.109 second response time [20:48:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.05, 8.58, 8.87 [20:48:41] RECOVERY - parapedia.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'parapedia.pl' will expire on Thu 05 May 2022 18:54:42 GMT +0000. [20:51:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.58, 8.54, 8.79 [21:02:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.88, 7.96, 7.70 [21:04:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.27, 9.30, 9.11 [21:08:36] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.23, 10.03, 8.68 [21:10:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.83, 9.66, 9.41 [21:11:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.08, 9.29, 8.65 [21:21:29] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.51, 7.85, 8.32 [21:36:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.62, 9.09, 8.79 [21:39:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.65, 9.08, 8.84 [21:43:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 9.87, 10.10, 9.33 [21:46:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.67, 8.43, 8.82 [21:46:13] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 0.82, 2.13, 1.80 [21:49:17] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.93, 7.63, 8.42 [21:49:24] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.61, 1.44, 1.59 [22:11:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:16:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:16:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.87, 4.01, 3.39 [22:19:52] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.83, 2.96, 3.09 [22:21:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:25:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.27, 8.80, 9.79 [22:25:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:30:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:36:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:49:43] PROBLEM - cp31 Disk Space on cp31 is WARNING: DISK WARNING - free space: / 4277 MB (10% inode=96%); [22:49:54] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 3.57, 6.46, 7.92 [22:56:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:06] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:12:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:15:54] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [23:16:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:19:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.17, 10.69, 9.12 [23:22:07] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1321s [23:22:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:31:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:31:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.46, 9.88, 9.56 [23:36:36] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:43:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.77, 10.25, 9.67 [23:46:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.01, 9.07, 9.33 [23:48:19]