[00:02:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.60, 3.70, 3.10 [00:03:15] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 8363 MB (21% inode=96%); [00:03:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:04:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.52, 3.45, 2.77 [00:04:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.34, 1.88, 1.60 [00:05:54] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.05, 3.28, 3.04 [00:06:01] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 334 failures. Last run 5 minutes ago with 334 failures. Failed resources (up to 3 shown): File[wiki.landev.vn_private],File[lcn.zfc.id.lv],File[lcn.zfc.id.lv_private],File[inourownwords.online] [00:06:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.50, 8.80, 8.66 [00:07:09] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.39, 2.91, 2.68 [00:07:09] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.99, 1.40, 1.46 [00:07:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.99, 8.77, 8.37 [00:07:58] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.55, 7.83, 8.41 [00:08:14] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.74, 7.97, 8.48 [00:09:16] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.50, 7.51, 8.34 [00:12:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.69, 10.30, 9.32 [00:13:43] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.13, 9.68, 8.85 [00:16:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.16, 8.50, 8.77 [00:16:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.95, 5.23, 4.00 [00:16:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.40, 9.25, 8.80 [00:16:59] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:18:28] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.55, 8.65, 8.60 [00:18:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.67, 9.27, 8.51 [00:20:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.45, 9.78, 9.09 [00:20:13] RECOVERY - cloud11 SMART on cloud11 is OK: OK: [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean [00:21:33] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.25, 8.28, 8.46 [00:21:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.93, 8.79, 8.44 [00:22:02] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.31, 8.60, 8.61 [00:22:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.98, 3.98, 3.85 [00:23:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.84, 9.71, 9.18 [00:24:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.90, 8.22, 8.27 [00:28:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.22, 9.32, 9.05 [00:28:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.79, 8.41, 8.53 [00:28:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.44, 2.59, 3.28 [00:31:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.33, 8.47, 8.74 [00:33:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:33:44] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.30, 8.06, 8.39 [00:33:57] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:34:21] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.59, 7.36, 8.09 [00:34:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.85, 9.51, 9.15 [00:37:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.76, 9.23, 9.07 [00:39:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:39:49] Hi, i want to request deletion of my miraheze.org account and associated Phabricator account. [00:40:08] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.08, 9.11, 8.84 [00:40:53] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.54, 9.73, 9.27 [00:43:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.45, 9.03, 8.79 [00:43:38] Guest56: Please make a request on the Stewards' noticeboard. SRE can handle the Phabricator part though. [00:43:48] Thank you. [00:43:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.75, 8.78, 8.98 [00:44:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:45:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.50, 9.07, 8.96 [00:46:16] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.71, 9.73, 9.10 [00:48:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.66, 9.94, 9.29 [00:49:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.85, 9.59, 9.17 [00:51:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:51:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.96, 9.66, 9.32 [00:52:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.35, 9.34, 9.19 [00:55:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.86, 8.91, 9.01 [01:03:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.33, 8.80, 8.52 [01:04:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.56, 8.84, 8.81 [01:05:56] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.36, 8.39, 8.46 [01:06:49] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.15, 7.67, 8.45 [01:10:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.44, 3.28 [01:11:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:13:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.07, 2.92, 3.12 [01:13:41] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.49, 9.29, 8.88 [01:14:04] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.09, 8.85, 8.07 [01:15:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:16:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.07, 9.19, 9.26 [01:17:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.81, 8.75, 8.17 [01:18:49] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.18, 9.73, 9.44 [01:19:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.54, 7.75, 7.89 [01:22:33] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.47, 8.81, 8.71 [01:25:30] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.77, 8.98, 8.82 [01:27:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.79, 9.21, 9.51 [01:30:47] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.26, 9.69, 9.65 [01:31:14] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.00, 8.19, 8.49 [01:33:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.45, 9.37, 9.52 [01:34:37] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.94, 2.19, 1.51 [01:35:41] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.93, 1.87, 1.67 [01:38:38] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.44, 1.35, 1.51 [01:40:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:48:38] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.85, 9.74, 9.32 [01:48:58] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.32, 7.55, 8.23 [01:51:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:54:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.12, 9.94, 9.59 [01:55:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.46, 8.79, 8.56 [01:56:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:58:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.83, 8.37, 8.43 [02:01:59] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.30, 2.78 [02:05:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.23, 2.77, 2.67 [02:08:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:09:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.75, 7.41, 8.34 [02:15:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.14, 8.78, 8.17 [02:18:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.45, 7.90, 7.92 [02:19:44] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.63, 10.82, 9.28 [02:20:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.14, 8.75, 8.54 [02:25:27] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.00, 8.55, 8.27 [02:25:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.48, 9.46, 9.36 [02:26:43] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.88, 8.14, 8.47 [02:28:28] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.50, 7.76, 8.02 [02:33:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:37:51] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.91, 7.46, 8.29 [02:40:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:44:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.77, 8.86, 8.59 [02:45:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:47:52] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.63, 9.24, 8.76 [02:49:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:54:06] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.23, 8.42, 8.61 [02:59:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:00:04] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 4.96, 7.15, 8.09 [03:08:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:10:31] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.04, 1.49, 1.88 [03:13:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:16:20] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.05, 1.20, 1.62 [03:19:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:21:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.24, 8.71, 8.14 [03:24:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:24:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.30, 8.17, 8.00 [03:35:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.20, 8.47, 7.86 [03:35:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:38:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.27, 7.58, 7.63 [03:40:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:47:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.97, 8.88, 7.84 [03:50:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.07, 9.75, 8.67 [03:50:48] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.44, 8.32, 7.81 [03:51:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:53:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.88, 9.30, 8.68 [03:56:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:57:35] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.08, 4.37, 2.88 [03:59:17] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.38, 7.67, 8.24 [04:06:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.54, 1.41, 2.00 [04:07:35] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.20, 2.66, 2.02 [04:12:15] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.75, 0.93, 1.58 [04:14:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:15:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.15, 8.40, 7.90 [04:24:05] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.80, 7.40, 7.64 [04:24:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:46:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:54:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.39, 9.29, 8.36 [05:00:03] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.44, 7.75, 8.06 [05:01:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:03:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.08, 1.50, 1.95 [05:05:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:08:24] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 5.83, 7.05, 5.08 [05:08:56] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.19, 1.24, 1.68 [05:15:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:17:07] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 28.02, 16.45, 9.60 [05:27:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:30:29] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.66, 2.51, 1.87 [05:40:26] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 3.85, 6.14, 7.59 [05:43:23] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 15.68, 10.25, 8.88 [05:47:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:52:21] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 4.19, 6.48, 7.65 [05:53:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:55:47] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 3.21, 4.78, 6.72 [05:59:54] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:00:23] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:00:54] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:01:04] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:01:09] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:01:46] PROBLEM - mwtask111 JobRunner Service on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:01:47] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:02:48] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:03:02] PROBLEM - mwtask111 ferm_active on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:03:24] PROBLEM - mwtask111 php-fpm on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:03:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:03:42] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:04:00] PROBLEM - mwtask111 NTP time on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:04:29] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:05:06] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:06:22] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [06:06:34] RECOVERY - mwtask111 ferm_active on mwtask111 is OK: OK ferm input default policy is set [06:06:55] RECOVERY - mwtask111 php-fpm on mwtask111 is OK: PROCS OK: 2 processes with command name 'php-fpm7.4' [06:07:11] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 0.790 second response time [06:07:31] RECOVERY - mwtask111 NTP time on mwtask111 is OK: NTP OK: Offset 0.004880666733 secs [06:07:50] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.511 second response time [06:07:51] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 0.631 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [06:08:11] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [06:08:16] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.400 second response time [06:08:21] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 45 minutes ago with 0 failures [06:08:22] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [06:08:46] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.070 second response time [06:08:50] RECOVERY - mwtask111 JobRunner Service on mwtask111 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [06:09:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.80, 8.25, 7.91 [06:10:34] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 22233 bytes in 4.251 second response time [06:13:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.40, 8.16, 7.45 [06:16:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.14, 7.79, 7.45 [06:19:01] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.31, 9.62, 8.57 [06:22:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.61, 8.63, 8.41 [06:24:56] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.03, 7.58, 8.04 [06:25:06] PROBLEM - wiki.mistleague.io - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mistleague.io' expires in 15 day(s) (Mon 28 Feb 2022 06:01:04 GMT +0000). [06:32:31] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/aad9641c0f9f...02f21931bb4a [06:32:32] [02miraheze/ssl] 07MirahezeSSLBot 0302f2193 - Bot: Update SSL cert for wiki.mistleague.io [06:44:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:49:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:49:43] !log INSERT INTO content_models (model_id, model_name) VALUES ( 1, 'wikitext'); INSERT INTO content_models (model_id, model_name) VALUES ( 2, 'css'); INSERT INTO content_models (model_id, model_name) VALUES ( 3, 'javascript'); INSERT INTO content_models (model_id, model_name) VALUES ( 4, 'Scribunto'); INSERT INTO content_models (model_id, model_name) VALUES ( 5, 'json'); (wikicryptoswiki) [06:49:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:52:42] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 6.08, 6.97, 6.47 [06:55:20] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/15ae6d0555ba...66b4eea5208c [06:55:21] [02miraheze/dns] 07Reception123 0366b4eea - add proficientprofit.click zone [06:55:33] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 5.15, 6.16, 6.24 [06:56:00] RECOVERY - wiki.mistleague.io - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mistleague.io' will expire on Fri 13 May 2022 05:32:23 GMT +0000. [06:56:15] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/66b4eea5208c...dbb3de7ba8da [06:56:16] [02miraheze/dns] 07Reception123 03dbb3de7 - add it to famepedia.org zone [07:08:24] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/dbb3de7ba8da...7e26fd25ddeb [07:08:25] [02miraheze/dns] 07Reception123 037e26fd2 - add sammaditthi.cyou zone [07:12:03] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±2] 13https://github.com/miraheze/ssl/compare/02f21931bb4a...c4b1ecfafa92 [07:12:05] [02miraheze/ssl] 07Reception123 03fe6f46b - add tech.teojingyao.com cert [07:12:06] [02miraheze/ssl] 07Reception123 03c4b1ecf - add wiki.chippyapps.com cert [07:20:24] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 2.16, 8.44, 8.55 [07:23:41] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 3.45, 6.51, 7.79 [07:26:34] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 1.11, 4.13, 6.65 [08:00:35] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.16, 1.33, 1.97 [08:06:18] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.06, 0.60, 1.46 [08:17:19] [02miraheze/ssl] 07Reception123 pushed 033 commits to 03master [+3/-0/±4] 13https://github.com/miraheze/ssl/compare/c4b1ecfafa92...d9e9cff91054 [08:17:21] [02miraheze/ssl] 07Reception123 039337075 - add www.proficientprofit.click cert [08:17:22] [02miraheze/ssl] 07Reception123 03acb8f65 - add it.famepedia.org cert [08:17:24] [02miraheze/ssl] 07Reception123 03d9e9cff - add sammaditthi.cyou cert [10:17:07] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 7 packages available for upgrade (0 critical updates). [10:17:31] RECOVERY - mw101 APT on mw101 is OK: APT OK: 8 packages available for upgrade (0 critical updates). [10:17:32] RECOVERY - test101 APT on test101 is OK: APT OK: 8 packages available for upgrade (0 critical updates). [10:17:33] RECOVERY - es111 APT on es111 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [10:17:59] RECOVERY - mw111 APT on mw111 is OK: APT OK: 8 packages available for upgrade (0 critical updates). [10:18:08] RECOVERY - mw102 APT on mw102 is OK: APT OK: 8 packages available for upgrade (0 critical updates). [10:19:11] RECOVERY - mw121 APT on mw121 is OK: APT OK: 8 packages available for upgrade (0 critical updates). [10:19:12] RECOVERY - mw122 APT on mw122 is OK: APT OK: 8 packages available for upgrade (0 critical updates). [10:19:14] RECOVERY - mw112 APT on mw112 is OK: APT OK: 8 packages available for upgrade (0 critical updates). [10:22:27] RECOVERY - puppet111 APT on puppet111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [10:27:51] RECOVERY - cloud12 APT on cloud12 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [10:28:08] RECOVERY - cloud10 APT on cloud10 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [10:28:19] RECOVERY - cloud11 APT on cloud11 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [10:29:49] Reception123: are you able to deal with the 10 criticals in icinga for ssl? [10:30:55] RECOVERY - es101 APT on es101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [10:31:25] JohnLewis: ah it's the wildcards, yeah I'll get to those in ~30 mins [10:31:35] RECOVERY - matomo101 APT on matomo101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [10:31:36] great [10:31:49] not sure why wiki.yapsavun.xyz is critical though as the bot should deal with that [10:32:22] RECOVERY - es121 APT on es121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [10:34:16] RECOVERY - graylog121 APT on graylog121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [10:46:55] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1ccb0da0953f...44e7fc72b6da [10:46:56] [02miraheze/puppet] 07JohnFLewis 0344e7fc7 - phabricator: set allowed-uris to hostname for stunnel monitoring [10:52:11] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/7e26fd25ddeb...c76ddf455c08 [10:52:12] [02miraheze/dns] 07Reception123 03c76ddf4 - add acme challenge to orain.org [10:53:30] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/c76ddf455c08...99d074722a49 [10:53:31] [02miraheze/dns] 07Reception123 0399d0747 - add acme challenge for orain.porg [10:54:14] Reception123: porg? That's a new domain TLD :P [10:54:33] JohnLewis: lol, at least the typo is only in the commit message :D [10:55:07] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d9e9cff91054...fab7c8f72b94 [10:55:09] [02miraheze/ssl] 07Reception123 03fab7c8f - renew *.orain.org cert [10:56:45] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/99d074722a49...fa4dec7f4b70 [10:56:47] [02miraheze/dns] 07Reception123 03fa4dec7 - add acme challenge for miraheze.com [10:57:30] Reception123: do you want to see a refreshing graph as well? [10:57:50] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fab7c8f72b94...3c27759d5e68 [10:57:51] [02miraheze/ssl] 07Reception123 033c27759 - renew miraheze.com cert [10:58:34] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/fa4dec7f4b70...915bb6560263 [10:58:35] [02miraheze/dns] 07Reception123 03915bb65 - add acme challenge for miraheze.wiki [10:58:43] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=mon111.miraheze.org&var-port=9100&viewPanel=281&from=now-7d&to=now-1m [10:59:03] JohnLewis: oh wow, that's fun [10:59:41] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/915bb6560263...4b99fbea16b4 [10:59:42] [02miraheze/dns] 07Reception123 034b99fbe - add acme challenge to miraheze.wiki [11:00:41] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3c27759d5e68...d2ee43888254 [11:00:43] [02miraheze/ssl] 07Reception123 03d2ee438 - renew miraheze.wiki [11:01:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:06:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:06:36] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/44e7fc72b6da...5b6de7ce59a4 [11:06:37] [02miraheze/puppet] 07JohnFLewis 035b6de7c - varnish: monitor a 500 error on phab121 only [11:09:30] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.318 second response time [11:17:35] RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Fri 13 May 2022 09:57:30 GMT +0000. [11:20:12] RECOVERY - cp31 Stunnel HTTP for phab121 on cp31 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.292 second response time [11:22:47] RECOVERY - cp30 Stunnel HTTP for phab121 on cp30 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.309 second response time [11:24:59] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.098 second response time [11:26:20] RECOVERY - poserdazfreebies.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Fri 13 May 2022 09:54:55 GMT +0000. [11:29:54] RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Fri 13 May 2022 09:54:55 GMT +0000. [11:38:04] RECOVERY - orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Fri 13 May 2022 09:54:55 GMT +0000. [11:38:54] RECOVERY - isv.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Fri 13 May 2022 09:54:55 GMT +0000. [11:54:24] RECOVERY - miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Fri 13 May 2022 10:00:27 GMT +0000. [12:22:16] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 0.93, 2.12, 1.70 [12:25:15] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.98, 1.54, 1.55 [12:37:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:42:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:50:02] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.50, 8.55, 7.59 [12:50:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:55:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:55:37] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.79, 2.04, 1.64 [12:55:54] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.91, 7.62, 7.55 [12:58:29] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.79, 1.49, 1.49 [13:22:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:27:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:33:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:38:14] PROBLEM - mw122 APT on mw122 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (2 critical updates). [13:38:33] PROBLEM - bast101 APT on bast101 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:38:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:38:52] PROBLEM - mw111 APT on mw111 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (2 critical updates). [13:43:02] PROBLEM - db111 APT on db111 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (1 critical updates). [13:43:36] PROBLEM - cloud12 APT on cloud12 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:43:54] PROBLEM - phab121 APT on phab121 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:45:35] PROBLEM - cloud10 APT on cloud10 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:45:55] PROBLEM - ldap111 APT on ldap111 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:45:56] PROBLEM - mw121 APT on mw121 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (2 critical updates). [13:46:06] PROBLEM - bast121 APT on bast121 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:46:16] PROBLEM - mail121 APT on mail121 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:46:28] PROBLEM - mw102 APT on mw102 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (2 critical updates). [13:47:11] PROBLEM - es121 APT on es121 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [13:49:20] PROBLEM - es101 APT on es101 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [13:49:29] PROBLEM - gluster101 APT on gluster101 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (1 critical updates). [13:50:28] PROBLEM - jobchron121 APT on jobchron121 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:51:21] RECOVERY - cloud10 APT on cloud10 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:51:51] RECOVERY - ldap111 APT on ldap111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:51:52] RECOVERY - mw121 APT on mw121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:52:05] RECOVERY - bast121 APT on bast121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:52:12] RECOVERY - mail121 APT on mail121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:52:19] RECOVERY - es101 APT on es101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:52:23] RECOVERY - gluster101 APT on gluster101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:52:25] RECOVERY - mw102 APT on mw102 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:52:31] RECOVERY - cloud12 APT on cloud12 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:52:56] RECOVERY - phab121 APT on phab121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:53:05] RECOVERY - es121 APT on es121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:53:06] RECOVERY - mw122 APT on mw122 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:53:13] RECOVERY - bast101 APT on bast101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:53:25] RECOVERY - jobchron121 APT on jobchron121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:53:46] RECOVERY - mw111 APT on mw111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:55:14] PROBLEM - db121 APT on db121 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (1 critical updates). [13:56:12] PROBLEM - db101 APT on db101 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (1 critical updates). [13:56:33] PROBLEM - es111 APT on es111 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [13:57:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:58:14] PROBLEM - cp20 APT on cp20 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:58:56] PROBLEM - cp21 APT on cp21 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:59:07] PROBLEM - puppet111 APT on puppet111 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:59:53] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:01:15] PROBLEM - graylog121 APT on graylog121 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (1 critical updates). [14:01:31] PROBLEM - gluster111 APT on gluster111 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:01:32] PROBLEM - mon111 APT on mon111 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (1 critical updates). [14:02:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:03:23] PROBLEM - gluster121 APT on gluster121 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:03:58] PROBLEM - mw101 APT on mw101 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [14:04:56] PROBLEM - mem121 APT on mem121 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:14:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:24:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:27:05] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.96, 8.33, 7.80 [14:29:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.86, 7.44, 7.53 [14:36:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:41:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:43:23] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 0.93, 3.35, 2.71 [14:50:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:51:59] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.27, 1.32, 1.91 [14:54:55] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.15, 1.52, 1.87 [14:55:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:57:50] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.18, 1.38, 1.76 [15:01:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:05:25] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03prometheus [+16/-15/±16] 13https://github.com/miraheze/puppet/commit/371040ab2247 [15:05:26] [02miraheze/puppet] 07JohnFLewis 03371040a - prometheus: rewrite for prometheus101 [15:05:28] [02puppet] 07JohnFLewis created branch 03prometheus - 13https://github.com/miraheze/puppet [15:06:34] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.56, 1.11, 1.54 [15:06:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:11:39] PROBLEM - mon111 Puppet on mon111 is WARNING: WARNING: Puppet is currently disabled, message: John, last run 13 minutes ago with 0 failures [15:12:16] PROBLEM - prometheus101 Puppet on prometheus101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:13:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/6f79d81cb156 [15:13:15] [02miraheze/puppet] 07paladox 036f79d81 - mon111: increase MaxConcurrentChecks to 12 [15:13:16] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [15:13:18] [02puppet] 07paladox opened pull request 03#2421: mon111: increase MaxConcurrentChecks to 12 - 13https://github.com/miraheze/puppet/pull/2421 [15:13:35] [02puppet] 07paladox closed pull request 03#2421: mon111: increase MaxConcurrentChecks to 12 - 13https://github.com/miraheze/puppet/pull/2421 [15:13:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5b6de7ce59a4...0918c2fd1d3d [15:13:38] [02miraheze/puppet] 07paladox 030918c2f - mon111: increase MaxConcurrentChecks to 12 (#2421) [15:13:40] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [15:13:41] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [15:13:49] PROBLEM - db111 Puppet on db111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:15:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:20:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:35:32] RECOVERY - prometheus101 Puppet on prometheus101 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [15:40:06] RECOVERY - db111 Puppet on db111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:43:21] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03prometheus [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/371040ab2247...aff6f96a5afa [15:43:23] [02miraheze/puppet] 07JohnFLewis 03aff6f96 - fixes [15:47:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:48:52] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.65, 11.44, 9.20 [15:49:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.47, 9.23, 8.30 [15:52:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.17, 9.72, 8.97 [15:52:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:52:44] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.91, 8.37, 8.09 [15:53:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.19, 3.47, 3.20 [15:54:32] PROBLEM - prometheus101 Puppet on prometheus101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[prometheus-reload] [15:59:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.80, 2.67, 2.95 [15:59:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:00:00] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:00:32] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.50, 8.89, 8.72 [16:01:04] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:03:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.75, 9.40, 8.96 [16:05:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.82, 8.90, 8.29 [16:06:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.96, 9.72, 9.14 [16:08:15] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 17.33, 12.12, 9.60 [16:08:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.25, 9.73, 8.47 [16:11:50] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:11:57] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:11:57] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.26, 7.15, 7.66 [16:14:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:14:36] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 8.13, 11.72, 9.94 [16:14:54] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.024 second response time [16:15:04] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.176 second response time [16:15:54] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.93, 9.80, 9.88 [16:18:41] RECOVERY - prometheus101 Puppet on prometheus101 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [16:19:35]