[00:01:52] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 7086 MB (18% inode=96%); [00:04:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.23, 8.45, 7.86 [00:05:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.34, 10.05, 9.07 [00:06:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.40, 10.36, 9.41 [00:06:22] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 536 failures. Last run 5 minutes ago with 536 failures. Failed resources (up to 3 shown): File[wiki.scvo.org_private],File[www.thesimswiki.com],File[www.thesimswiki.com_private],File[munwiki.info] [00:08:23] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.69, 9.22, 8.92 [00:12:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.48, 8.89, 7.58 [00:15:00] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.12, 8.61, 7.62 [00:15:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.24, 9.57, 9.65 [00:17:29] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.49, 8.01, 8.07 [00:17:59] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.93, 8.29, 7.69 [00:19:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:21:28] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.91, 7.62, 7.67 [00:31:36] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [00:32:58] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.48, 7.82, 8.44 [00:33:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:33:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.45, 9.44, 9.24 [00:36:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.18, 9.43, 9.33 [00:43:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.30, 8.71, 8.58 [00:44:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.68, 9.73, 8.64 [00:45:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.01, 9.52, 9.39 [00:47:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.06, 9.39, 8.71 [00:48:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.42, 9.72, 9.52 [00:49:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.43, 3.17 [00:51:39] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.22, 10.01, 9.66 [00:53:55] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.73, 8.21, 8.43 [00:55:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.17, 3.18, 3.14 [01:01:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.24, 9.68, 9.10 [01:04:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.47, 9.04, 8.95 [01:09:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.16, 8.52, 8.31 [01:09:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.15, 9.02, 9.86 [01:12:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.13, 9.80, 9.99 [01:13:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:13:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.87, 8.20, 7.76 [01:15:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.83, 8.90, 9.58 [01:16:06] PROBLEM - es101 Current Load on es101 is CRITICAL: CRITICAL - load average: 4.56, 3.37, 2.30 [01:16:43] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.23, 7.75, 7.67 [01:17:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:18:17] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.87, 7.87, 8.15 [01:19:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.28, 10.81, 10.23 [01:19:16] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 2.07, 2.89, 2.32 [01:19:54] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.78, 10.18, 9.37 [01:24:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.14, 9.20, 9.84 [01:25:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.39, 9.26, 9.33 [01:27:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:40:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.15, 3.73, 3.11 [01:40:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:43:04] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.07, 3.34, 3.06 [01:43:49] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.00, 8.19, 8.38 [01:50:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:03:21] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.57, 7.66, 8.28 [02:10:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.65, 10.37, 9.35 [02:11:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:22:13] PROBLEM - mon111 Current Load on mon111 is WARNING: WARNING - load average: 5.41, 4.72, 4.34 [02:23:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.82, 9.33, 8.38 [02:25:11] RECOVERY - mon111 Current Load on mon111 is OK: OK - load average: 3.49, 4.36, 4.28 [02:26:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.16, 8.46, 8.22 [02:31:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:31:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.61, 9.07, 9.75 [02:40:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.50, 7.07, 8.41 [02:51:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:55:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.64, 8.77, 8.56 [02:56:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:01:13] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.94, 8.11, 8.39 [03:12:11] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.72, 8.30, 7.42 [03:14:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.94, 9.02, 8.65 [03:15:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.76, 3.45, 3.04 [03:15:13] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.14, 8.39, 7.63 [03:17:09] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.77, 8.16, 8.35 [03:18:09] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.88, 2.98, 2.96 [03:38:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.63, 8.20, 7.97 [03:39:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.46, 9.15, 8.80 [03:42:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:44:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.22, 8.78, 7.83 [03:44:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.16, 7.80, 7.88 [03:47:07] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.60, 8.16, 7.75 [03:47:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:48:11] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.12, 8.59, 8.60 [03:51:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.94, 9.36, 8.91 [03:53:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:58:55] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.18, 1.77, 1.49 [04:01:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.56, 8.45, 8.10 [04:01:50] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.13, 2.93 [04:04:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.88, 7.55, 7.83 [04:04:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.05, 2.80, 2.85 [04:05:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.03, 1.43, 1.47 [04:06:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.32, 9.84, 9.29 [04:09:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.41, 9.42, 9.20 [04:13:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:18:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:23:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:23:49] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.55, 7.79, 8.46 [04:43:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.68, 7.91, 6.98 [04:46:48] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.02, 7.86, 7.12 [04:47:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.72, 9.26, 8.74 [04:50:32] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.33, 8.01, 8.35 [04:54:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:59:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:51:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:56:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:09:15] PROBLEM - cloud10 SMART on cloud10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:12:16] RECOVERY - cloud10 SMART on cloud10 is OK: OK: [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean [06:13:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:18:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:21:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:24:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.62, 9.20, 8.52 [06:27:30] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.56, 8.10, 8.21 [06:30:48] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/5e3efc42ad5e...a8230d5ea7b2 [06:30:50] [02miraheze/dns] 07Reception123 03a8230d5 - add bitcoin21.ch zone [06:31:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:34:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.50, 9.53, 8.67 [06:35:20] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 1.49, 2.76, 2.24 [06:36:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:36:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.04, 8.72, 8.28 [06:40:13] PROBLEM - ns1 ferm_active on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:40:39] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:41:17] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:41:18] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:41:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:41:40] PROBLEM - ns1 SSH on ns1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.60, 9.66, 9.15 [06:42:06] PROBLEM - ns1 conntrack_table_size on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:42:22] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.88, 1.64, 1.93 [06:42:56] PROBLEM - ns1 Disk Space on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:43:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.00, 8.14, 8.21 [06:44:13] PROBLEM - ns1 NTP time on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:44:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:45:08] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/a8230d5ea7b2...b37cd60c6d2c [06:45:09] [02miraheze/dns] 07Reception123 03b37cd60 - Delete m33ks.ml [06:45:12] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/b37cd60c6d2c...d23675527d41 [06:45:13] [02miraheze/dns] 07Reception123 03d236755 - Delete m33ks.com [06:46:23] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/d23675527d41...a4ddf0f1803e [06:46:25] [02miraheze/dns] 07Reception123 03a4ddf0f - readd m33ks.ml zone (refresh) [06:46:38] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±2] 13https://github.com/miraheze/ssl/compare/bfacb62fd59a...305d9d3a70fa [06:46:40] [02miraheze/ssl] 07Reception123 037add664 - add wiki.thesohnlandgov.info cert [06:46:41] [02miraheze/ssl] 07Reception123 03305d9d3 - add www.bitcoin21.ch cert [06:49:10] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.40, 1.23, 1.61 [06:49:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:50:16] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/305d9d3a70fa...1480fe5f10a5 [06:50:17] [02miraheze/ssl] 07Reception123 031480fe5 - add m33ks.ml cert [06:51:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.81, 8.85, 8.67 [06:52:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.88, 7.58, 8.29 [06:54:59] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.04, 8.23, 8.44 [06:55:32] !log DELETED and DROPPED schizoidnightmareswiki (reset req) [06:55:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:10:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:20:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:44:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:49:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:05:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:10:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:48:49] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.03233200312 secs [08:57:18] PROBLEM - ns1 NTP time on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:15:12] PROBLEM - famedata.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'famedata.org' expires in 15 day(s) (Sat 19 Feb 2022 10:01:26 GMT +0000). [10:16:56] PROBLEM - www.famedata.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'famedata.org' expires in 15 day(s) (Sat 19 Feb 2022 10:01:26 GMT +0000). [10:17:19] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1480fe5f10a5...d13dbb01e109 [10:17:21] [02miraheze/ssl] 07MirahezeSSLBot 03d13dbb0 - Bot: Update SSL cert for famedata.org [10:18:11] PROBLEM - www.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.petrawiki.org' expires in 15 day(s) (Sat 19 Feb 2022 10:07:28 GMT +0000). [10:30:31] PROBLEM - vault.agentisai.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vault.agentisai.com' expires in 15 day(s) (Sat 19 Feb 2022 10:17:35 GMT +0000). [10:32:02] PROBLEM - wiki.eversd.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.eversd.com' expires in 15 day(s) (Sat 19 Feb 2022 10:20:08 GMT +0000). [10:32:08] PROBLEM - hr.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hr.petrawiki.org' expires in 15 day(s) (Sat 19 Feb 2022 10:14:58 GMT +0000). [10:33:30] PROBLEM - translate.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'translate.petrawiki.org' expires in 15 day(s) (Sat 19 Feb 2022 10:12:30 GMT +0000). [10:35:50] PROBLEM - petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.petrawiki.org' expires in 15 day(s) (Sat 19 Feb 2022 10:07:28 GMT +0000). [10:36:19] PROBLEM - en.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.petrawiki.org' expires in 15 day(s) (Sat 19 Feb 2022 10:07:28 GMT +0000). [10:37:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d13dbb01e109...afa07c09e691 [10:37:43] [02miraheze/ssl] 07MirahezeSSLBot 03afa07c0 - Bot: Update SSL cert for vault.agentisai.com [10:37:46] PROBLEM - wiki.yapsavun.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.yapsavun.xyz' expires in 15 day(s) (Sat 19 Feb 2022 10:26:20 GMT +0000). [10:44:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:45:26] PROBLEM - polandballwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.polandballwiki.com' expires in 15 day(s) (Sat 19 Feb 2022 10:24:22 GMT +0000). [10:45:33] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/afa07c09e691...a7239f3c978e [10:45:34] [02miraheze/ssl] 07MirahezeSSLBot 03a7239f3 - Bot: Update SSL cert for hr.petrawiki.org [10:45:46] PROBLEM - reviews.thejoshmeister.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'reviews.thejoshmeister.com' expires in 15 day(s) (Sat 19 Feb 2022 10:32:42 GMT +0000). [10:45:46] PROBLEM - files.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'files.petrawiki.org' expires in 15 day(s) (Sat 19 Feb 2022 10:10:50 GMT +0000). [10:45:59] RECOVERY - famedata.org - LetsEncrypt on sslhost is OK: OK - Certificate 'famedata.org' will expire on Wed 04 May 2022 09:17:13 GMT +0000. [10:47:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a7239f3c978e...604b35dcfc6b [10:47:58] [02miraheze/ssl] 07MirahezeSSLBot 03604b35d - Bot: Update SSL cert for en.petrawiki.org [10:48:09] RECOVERY - www.famedata.org - LetsEncrypt on sslhost is OK: OK - Certificate 'famedata.org' will expire on Wed 04 May 2022 09:17:13 GMT +0000. [10:48:40] PROBLEM - www.polandballwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.polandballwiki.com' expires in 15 day(s) (Sat 19 Feb 2022 10:24:22 GMT +0000). [10:49:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:50:46] PROBLEM - pt.polandballwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pt.polandballwiki.com' expires in 15 day(s) (Sat 19 Feb 2022 10:22:16 GMT +0000). [10:50:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/604b35dcfc6b...e1566b662e68 [10:50:54] [02miraheze/ssl] 07MirahezeSSLBot 03e1566b6 - Bot: Update SSL cert for translate.petrawiki.org [10:52:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e1566b662e68...9c286aedf9d7 [10:52:12] [02miraheze/ssl] 07MirahezeSSLBot 039c286ae - Bot: Update SSL cert for wiki.eversd.com [10:57:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9c286aedf9d7...336b12f221ca [10:57:49] [02miraheze/ssl] 07MirahezeSSLBot 03336b12f - Bot: Update SSL cert for pt.polandballwiki.com [10:58:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.60, 8.14, 7.49 [10:58:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/336b12f221ca...2b864da3aaec [10:58:37] [02miraheze/ssl] 07MirahezeSSLBot 032b864da - Bot: Update SSL cert for files.petrawiki.org [11:00:37] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2b864da3aaec...5b3b1bffb23b [11:00:39] [02miraheze/ssl] 07MirahezeSSLBot 035b3b1bf - Bot: Update SSL cert for reviews.thejoshmeister.com [11:01:12] RECOVERY - vault.agentisai.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vault.agentisai.com' will expire on Wed 04 May 2022 09:37:36 GMT +0000. [11:01:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:01:56] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.33, 7.62, 7.39 [11:03:31] RECOVERY - hr.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'hr.petrawiki.org' will expire on Wed 04 May 2022 09:45:27 GMT +0000. [11:06:21] RECOVERY - en.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.petrawiki.org' will expire on Wed 04 May 2022 09:47:49 GMT +0000. [11:07:13] RECOVERY - petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.petrawiki.org' will expire on Wed 04 May 2022 09:47:49 GMT +0000. [11:07:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5b3b1bffb23b...50afb08c3089 [11:07:48] [02miraheze/ssl] 07MirahezeSSLBot 0350afb08 - Bot: Update SSL cert for www.polandballwiki.com [11:14:16] PROBLEM - christipedia.nl - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/p [11:14:16] ist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404743909835815 seconds [11:18:50] RECOVERY - www.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.petrawiki.org' will expire on Wed 04 May 2022 09:47:49 GMT +0000. [11:20:23] PROBLEM - miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.wiki' expires in 15 day(s) (Sat 19 Feb 2022 11:08:45 GMT +0000). [11:21:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:30:17] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=icbfbreedingwiki --username-prefix=icbfbreeding --no-updates --report 1 /home/reception/icbfbreedingwiki.xml (START) [11:30:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:30:54] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=icbfbreedingwiki --username-prefix=icbfbreeding --no-updates --report 1 /home/reception/icbfbreedingwiki.xml (END - exit=0) [11:30:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:34:19] RECOVERY - translate.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'translate.petrawiki.org' will expire on Wed 04 May 2022 09:50:46 GMT +0000. [11:34:55] RECOVERY - wiki.eversd.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.eversd.com' will expire on Wed 04 May 2022 09:52:04 GMT +0000. [11:35:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:39:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.19, 10.22, 8.28 [11:40:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.46, 8.53, 7.47 [11:42:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.53, 9.85, 8.53 [11:43:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.39, 7.88, 7.44 [11:44:29] RECOVERY - christipedia.nl - reverse DNS on sslhost is OK: SSL OK - christipedia.nl reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK [11:45:46] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.07, 9.39, 8.52 [11:46:51] RECOVERY - polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.polandballwiki.com' will expire on Wed 04 May 2022 10:07:41 GMT +0000. [11:46:54] RECOVERY - files.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'files.petrawiki.org' will expire on Wed 04 May 2022 09:58:31 GMT +0000. [11:48:15] RECOVERY - reviews.thejoshmeister.com - LetsEncrypt on sslhost is OK: OK - Certificate 'reviews.thejoshmeister.com' will expire on Wed 04 May 2022 10:00:32 GMT +0000. [11:49:55] RECOVERY - www.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.polandballwiki.com' will expire on Wed 04 May 2022 10:07:41 GMT +0000. [11:51:01] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.05, 2.06, 1.77 [11:52:00] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 9.73, 10.12, 8.80 [11:52:41] RECOVERY - pt.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.polandballwiki.com' will expire on Wed 04 May 2022 09:57:43 GMT +0000. [11:54:19] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.91, 1.98, 1.80 [11:55:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.79, 8.59, 8.44 [11:55:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.52, 9.66, 9.19 [11:57:39] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.73, 1.46, 1.64 [11:58:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.12, 8.24, 8.30 [12:00:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:05:49] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.25, 7.78, 8.42 [12:06:17] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.008 second response time [12:08:46] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/a3ad22cd15e7...e209c44cc23d [12:08:47] [02miraheze/ManageWiki] 07translatewiki 03e209c44 - Localisation updates from https://translatewiki.net. [12:08:49] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/MirahezeMagic/compare/7111a2f8da18...fc878f0c71a0 [12:08:50] [02miraheze/MirahezeMagic] 07translatewiki 03fc878f0 - Localisation updates from https://translatewiki.net. [12:09:34] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.015 second response time [12:14:11] miraheze/MirahezeMagic - translatewiki the build has errored. [12:18:07] miraheze/ManageWiki - translatewiki the build passed. [12:19:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:20:41] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=icbfbreedingwiki (START) [12:20:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:20:50] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=icbfbreedingwiki (END - exit=0) [12:20:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:30:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:40:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:46:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:51:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:57:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:02:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:20:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b6ccaea02ef9...f65ca1bd5eac [13:20:07] [02miraheze/puppet] 07paladox 03f65ca1b - Remove paladox ssh key [13:30:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.21, 9.40, 8.45 [13:31:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:32:27] RECOVERY - ns1 ferm_active on ns1 is OK: OK ferm input default policy is set [13:36:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:40:57] PROBLEM - ns1 ferm_active on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:43:50] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.29, 9.46, 8.18 [13:44:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.06, 7.81, 8.09 [13:46:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:47:29] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.03, 8.41, 8.04 [13:51:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.97, 8.93, 8.33 [13:52:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.56, 8.55, 8.48 [13:57:15] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.008 second response time [14:00:42] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 3.021 second response time [14:02:17] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.99, 7.68, 8.02 [14:02:43] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.77, 8.10, 8.39 [14:04:53] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.55, 2.08, 1.70 [14:05:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.68, 9.68, 8.78 [14:08:27] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.72, 1.55, 1.57 [14:11:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.32, 9.32, 8.95 [14:14:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.30, 8.92, 8.87 [14:15:37] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.14, 8.15, 8.43 [14:16:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:18:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.10, 9.87, 9.26 [14:21:23] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.39, 9.09, 8.15 [14:22:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:24:10] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.05, 2.58, 2.04 [14:27:53] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.77, 1.65, 1.79 [14:29:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.28, 8.81, 9.15 [14:32:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.88, 9.24, 8.83 [14:32:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:35:13] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.58, 1.11, 1.49 [14:39:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:43:25] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.94, 8.00, 8.47 [14:43:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.39, 10.30, 9.65 [14:47:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.89, 8.66, 7.85 [14:50:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.06, 8.49, 8.33 [14:52:17] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.78, 10.98, 9.73 [14:54:30] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.21, 8.12, 7.93 [14:59:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.49, 9.61, 9.56 [15:05:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.18, 10.82, 9.69 [15:07:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.48, 8.73, 8.31 [15:11:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.46, 9.64, 8.77 [15:16:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.36, 9.77, 9.90 [15:18:57] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.06, 10.39, 9.55 [15:23:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.18, 9.60, 9.28 [15:23:08] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.26, 9.21, 8.58 [15:24:36] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.53, 10.27, 10.02 [15:30:37] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.40, 9.64, 9.79 [15:30:50] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.40, 9.39, 8.89 [15:34:34] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.15, 10.21, 9.99 [15:34:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.71, 9.28, 8.95 [15:38:33] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.34, 10.35, 9.53 [15:40:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.98, 8.50, 8.34 [15:42:44] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.06, 8.00, 8.50 [15:44:06] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.11, 8.27, 8.30 [15:46:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.96, 9.92, 9.60 [15:51:42] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.90, 9.39, 9.86 [15:52:11] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.98, 8.42, 8.52 [15:55:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.13, 9.47, 9.76 [15:55:52] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.85, 8.17, 8.45 [15:59:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.91, 9.11, 9.55 [16:01:11] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.28, 7.51, 8.44 [16:02:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.04, 9.08, 9.41 [16:04:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.11, 8.80, 9.65 [16:09:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.56, 9.27, 9.46 [16:10:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.37, 10.53, 9.44 [16:14:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.06, 9.75, 9.41 [16:15:46] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.006155312061 secs [16:19:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.37, 9.80, 9.39 [16:21:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.80, 10.32, 9.58 [16:22:28] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.92, 1.81, 1.57 [16:23:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.67, 9.87, 9.57 [16:24:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.18, 9.98, 9.62 [16:25:28] PROBLEM - ns1 NTP time on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:26:21] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.38, 1.07, 1.32 [16:27:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.00, 10.35, 9.82 [16:28:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.58, 9.54, 9.53 [16:29:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.37, 9.31, 9.56 [16:30:28] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:52] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:32:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.47, 10.04, 9.72 [16:34:26] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.168 second response time [16:35:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.73, 9.11, 9.58 [16:36:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 8.91, 10.14, 9.93 [16:36:32] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.25, 8.86, 8.52 [16:37:26] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.91, 3.50, 2.42 [16:38:54] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.516 second response time [16:39:19] [02puppet] 07Universal-Omega opened pull request 03#2413: base::puppet: remove temporary Buster; use Bullseye - 13https://github.com/miraheze/puppet/pull/2413 [16:40:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.55, 9.75, 9.84 [16:41:34] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 2.65, 2.90, 2.42 [16:42:53] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.56, 9.45, 9.48 [16:43:19] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:43:55] RECOVERY - ns1 SSH on ns1 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [16:43:56] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet last ran 10 hours ago [16:43:57] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.58, 9.85, 9.92 [16:45:24] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:45:40] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.02134656906 secs [16:45:42] RECOVERY - ns1 Disk Space on ns1 is OK: DISK OK - free space: / 3730 MB (52% inode=82%); [16:46:03] RECOVERY - ns1 conntrack_table_size on ns1 is OK: OK: nf_conntrack is 0 % full [16:46:51] RECOVERY - ns1 ferm_active on ns1 is OK: OK ferm input default policy is set [16:47:02] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.558 second response time [16:47:31] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:47:33] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.321 second response time [16:47:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.46, 10.00, 9.73 [16:47:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.84, 10.41, 10.15 [16:50:47] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.48, 8.75, 8.48 [16:51:25] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.87, 8.32, 8.45 [16:54:34] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.00, 9.32, 9.83 [16:57:17] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.24, 7.77, 8.16 [16:57:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.71, 9.65, 9.89 [16:59:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:02:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.79, 9.22, 9.91 [17:03:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.77, 8.57, 9.69 [17:03:30] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.53, 6.96, 8.43 [17:05:53] [02puppet] 07RhinosF1 commented on pull request 03#2413: base::puppet: remove temporary Buster; use Bullseye - 13https://github.com/miraheze/puppet/pull/2413#issuecomment-1029204844 [17:06:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.02, 9.88, 9.99 [17:09:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.21, 8.76, 8.85 [17:14:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.01, 9.19, 8.75 [17:14:51] [02puppet] 07Universal-Omega commented on pull request 03#2413: base::puppet: remove temporary Buster; use Bullseye - 13https://github.com/miraheze/puppet/pull/2413#issuecomment-1029213479 [17:15:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:16:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.60, 9.74, 9.30 [17:18:11] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.19, 11.21, 10.22 [17:18:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.40, 3.08 [17:18:23] CosmicAlpha: based on? [17:19:05] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.13, 9.78, 9.38 [17:20:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.13, 9.59, 9.07 [17:21:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.23, 2.52, 2.77 [17:22:11] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.61, 9.50, 9.37 [17:23:47] RhinosF1: http://apt.puppetlabs.com/pool/bullseye/puppet7/p/ — https://github.com/miraheze/puppet/pull/2188 says it didn't support for server or DB is why, but that seems to indicate it now does I think. [17:23:47] [url] Index of /pool/bullseye/puppet7/p | apt.puppetlabs.com [17:23:48] [url] base::puppet: Set release to buster by paladox · Pull Request #2188 · miraheze/puppet · GitHub | github.com [17:25:08] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.49, 10.11, 9.63 [17:25:37] CosmicAlpha: lgtm [17:28:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.48, 9.24, 9.41 [17:28:35] [02puppet] 07Universal-Omega commented on pull request 03#2413: base::puppet: remove temporary Buster; use Bullseye - 13https://github.com/miraheze/puppet/pull/2413#issuecomment-1029226800 [17:29:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.23, 9.75, 9.28 [17:30:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.10, 9.56, 9.87 [17:32:01] PROBLEM - ns1 Current Load on ns1 is WARNING: WARNING - load average: 0.16, 0.16, 1.77 [17:32:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.00, 9.25, 9.21 [17:34:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.08, 10.18, 9.70 [17:35:15] RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.02, 0.10, 1.43 [17:37:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.62, 9.11, 9.37 [17:40:07] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.91, 9.50, 9.59 [17:40:56] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.34, 9.25, 9.35 [17:43:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.52, 9.72, 9.68 [17:44:08]