[00:00:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.63, 7.92, 8.29 [00:03:58] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 7596 MB (19% inode=96%); [00:06:26] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 511 failures. Last run 5 minutes ago with 511 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [00:07:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:08:53] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 4.777 second response time [00:14:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:19:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:29:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.71, 9.03, 8.78 [00:33:57] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:35:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.03, 9.33, 8.98 [00:35:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:40:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:46:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:48:54] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:51:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:54:00] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.10, 7.72, 8.29 [00:58:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:03:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:08:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:14:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.06, 9.67, 8.79 [01:20:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 2.91, 2.56 [01:23:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.47, 4.02, 3.08 [01:24:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.46, 9.12, 7.91 [01:26:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.84, 3.21 [01:27:54] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.46, 8.21, 7.80 [01:29:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.30, 3.35, 3.11 [01:39:12] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.41, 9.85, 9.92 [01:44:28] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.55, 2.12, 1.70 [01:45:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.98, 3.62, 3.17 [01:47:26] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.72, 1.49, 1.53 [01:48:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.47, 3.30, 3.14 [01:48:36] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:51:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.42, 9.55, 9.24 [01:52:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:54:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.40, 9.02, 9.09 [01:57:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.11, 9.84, 9.38 [02:00:47] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.49, 9.93, 9.54 [02:01:02] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.49, 2.30, 2.02 [02:04:11] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.15, 1.65, 1.80 [02:07:14] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.60, 1.20, 1.60 [02:12:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:16:00] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.75, 7.14, 8.28 [02:27:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:29:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.95, 9.18, 8.76 [02:30:57] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 1.984 second response time [02:35:38] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.75, 9.60, 9.04 [02:36:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.32, 9.19, 8.39 [02:38:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.15, 9.01, 8.89 [02:39:01] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.88, 8.46, 8.23 [02:42:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:45:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:48:09] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.16, 7.74, 8.35 [02:58:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.73, 9.33, 8.82 [03:00:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:01:51] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.61, 7.72, 8.31 [03:01:56] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:10:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:15:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.71, 3.59, 3.20 [03:18:56] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.49, 4.22, 3.52 [03:20:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:24:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:25:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.94, 3.66 [03:29:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:33:12] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 2.399 second response time [03:34:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.03, 4.11, 3.80 [03:36:44] PROBLEM - es101 Current Load on es101 is CRITICAL: CRITICAL - load average: 4.44, 4.11, 3.13 [03:37:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.78, 3.44, 3.59 [03:38:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:39:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.68, 8.67, 8.35 [03:40:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.49, 3.57 [03:42:27] PROBLEM - es101 Disk Space on es101 is WARNING: DISK WARNING - free space: / 27010 MB (10% inode=99%); [03:43:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:45:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.83, 8.25, 8.38 [03:46:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 3.90, 3.74 [03:47:23] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:47:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:50:34] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 5.984 second response time [03:52:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:53:23] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.64, 4.59, 4.04 [03:56:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:00:37] PROBLEM - cloud10 SMART on cloud10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:01:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:02:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.89, 3.69, 3.17 [04:03:54] RECOVERY - cloud10 SMART on cloud10 is OK: OK: [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean [04:04:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:05:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.65, 2.95, 2.96 [04:08:07] RECOVERY - es101 Disk Space on es101 is OK: DISK OK - free space: / 32312 MB (12% inode=99%); [04:08:44] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 2.13, 3.01, 3.87 [04:09:29] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:09:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.76, 3.45, 3.86 [04:09:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:12:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.41, 3.56, 3.80 [04:13:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:15:52] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.90, 2.90, 3.51 [04:15:53] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 6.313 second response time [04:18:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:21:21] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 2.26, 2.79, 3.31 [04:22:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.02, 2.86, 3.33 [04:25:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:26:42] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.014 second response time [04:30:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:34:25] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.69, 3.27 [04:35:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:36:00] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 2.062 second response time [04:37:30] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.20, 3.01, 3.08 [04:40:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:44:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:49:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:52:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.51, 3.58, 3.29 [04:55:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:56:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.50, 3.32, 3.27 [04:56:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 3.90, 3.40 [05:00:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:03:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.13, 3.20, 3.30 [05:03:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:06:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.97, 1.73, 1.53 [05:07:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.27, 9.02, 8.43 [05:08:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:09:08] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.77, 1.22, 1.35 [05:11:02] PROBLEM - es121 Current Load on es121 is CRITICAL: CRITICAL - load average: 4.15, 3.78, 3.39 [05:17:13] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.97, 7.64, 8.06 [05:23:25] PROBLEM - es121 Current Load on es121 is WARNING: WARNING - load average: 3.62, 3.94, 3.78 [05:27:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:29:12] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.008 second response time [05:32:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:38:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:39:09] PROBLEM - cloud12 SMART on cloud12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:42:28] RECOVERY - cloud12 SMART on cloud12 is OK: OK: [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean [05:44:40] PROBLEM - es121 Current Load on es121 is CRITICAL: CRITICAL - load average: 4.40, 3.91, 3.69 [05:47:41] PROBLEM - es121 Current Load on es121 is WARNING: WARNING - load average: 2.93, 3.52, 3.58 [05:50:44] RECOVERY - es121 Current Load on es121 is OK: OK - load average: 2.61, 3.05, 3.38 [05:50:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.55, 9.24, 8.48 [05:53:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:53:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.13, 8.37, 8.28 [06:07:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.60, 8.70, 8.29 [06:13:09] PROBLEM - cloud12 SMART on cloud12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:14:04] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.37, 7.77, 8.05 [06:16:26] RECOVERY - cloud12 SMART on cloud12 is OK: OK: [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean [06:23:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.28, 8.48, 7.70 [06:23:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:26:31] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.11, 7.66, 7.55 [06:27:38] PROBLEM - cloud12 SMART on cloud12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:28:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:30:50] RECOVERY - cloud12 SMART on cloud12 is OK: OK: [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean [06:43:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:48:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:48:39] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.019 second response time [06:56:30] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:59:34] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 5.126 second response time [07:02:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:07:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:10:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.66, 8.78, 7.90 [07:18:06] PROBLEM - es111 Disk Space on es111 is WARNING: DISK WARNING - free space: / 26524 MB (9% inode=99%); [07:22:43] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.69, 8.32, 8.25 [07:23:01] PROBLEM - es111 Current Load on es111 is CRITICAL: CRITICAL - load average: 4.83, 4.11, 3.12 [07:24:08] RECOVERY - es111 Disk Space on es111 is OK: DISK OK - free space: / 30121 MB (11% inode=99%); [07:26:32] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [07:29:01] PROBLEM - es111 Current Load on es111 is WARNING: WARNING - load average: 3.61, 3.92, 3.39 [07:30:05] [02puppet] 07Universal-Omega opened pull request 03#2414: Remove SSH key for Universal Omega - 13https://github.com/miraheze/puppet/pull/2414 [07:30:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:30:42] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.007 second response time [07:30:56] PROBLEM - es111 Disk Space on es111 is WARNING: DISK WARNING - free space: / 25357 MB (9% inode=99%); [07:31:43] PROBLEM - wikipanel.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikipanel.org' expires in 15 day(s) (Tue 22 Feb 2022 07:24:03 GMT +0000). [07:33:49] RECOVERY - es111 Disk Space on es111 is OK: DISK OK - free space: / 35113 MB (13% inode=99%); [07:34:56] RECOVERY - es111 Current Load on es111 is OK: OK - load average: 2.08, 3.02, 3.22 [07:35:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.04, 2.98, 2.40 [07:35:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:35:43] PROBLEM - wiki.controlistes.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.controlistes.fr' expires in 15 day(s) (Tue 22 Feb 2022 07:25:49 GMT +0000). [07:36:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e504e699ce55...bd00e88b03bc [07:36:22] [02miraheze/ssl] 07MirahezeSSLBot 03bd00e88 - Bot: Update SSL cert for wikipanel.org [07:37:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bd00e88b03bc...b6451fb96a31 [07:37:37] [02miraheze/ssl] 07MirahezeSSLBot 03b6451fb - Bot: Update SSL cert for wiki.controlistes.fr [07:38:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.92, 3.17, 2.56 [07:38:23] PROBLEM - ngsswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ngsswiki.com' expires in 15 day(s) (Tue 22 Feb 2022 07:20:48 GMT +0000). [07:39:50] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 3.203 second response time [07:40:04] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b6451fb96a31...a5bb1e7ce5c6 [07:40:06] [02miraheze/ssl] 07MirahezeSSLBot 03a5bb1e7 - Bot: Update SSL cert for ngsswiki.com [07:41:17] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.74, 2.98, 2.59 [07:44:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:46:35] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.009 second response time [07:49:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:50:01] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [07:52:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:55:10] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 5.02, 4.49, 7.67 [07:57:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:02:20] RECOVERY - wikipanel.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipanel.org' will expire on Sat 07 May 2022 06:36:14 GMT +0000. [08:03:56] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 5.10, 4.98, 6.53 [08:04:02] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.032 second response time [08:05:38] RECOVERY - wiki.controlistes.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.controlistes.fr' will expire on Sat 07 May 2022 06:37:30 GMT +0000. [08:06:43] [02puppet] 07Reception123 closed pull request 03#2414: Remove SSH key for Universal Omega - 13https://github.com/miraheze/puppet/pull/2414 [08:06:45] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f65ca1bd5eac...c6d6e87e77df [08:06:46] [02miraheze/puppet] 07Universal-Omega 03c6d6e87 - Remove SSH key for Universal Omega (#2414) [08:08:26] RECOVERY - ngsswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'ngsswiki.com' will expire on Sat 07 May 2022 06:39:57 GMT +0000. [08:14:05] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.009 second response time [08:34:58] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 9.576 second response time [08:42:33] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:46:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:51:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:57:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:02:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:22:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.15, 4.12, 3.18 [09:23:41] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.87, 2.29, 1.55 [09:24:45] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [09:25:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.02, 3.60, 3.14 [09:26:35] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.57, 1.54, 1.39 [09:28:32] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.86, 2.91, 2.96 [09:41:16] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.040 second response time [09:47:53] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.014 second response time [09:51:00] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:52:56] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.024 second response time [10:59:38] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:14:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:17:26] Quick query to observers, is anyone able to access my CentralAuth page? https://meta.miraheze.org/wiki/Special:CentralAuth/Raidarr [11:19:28] [url] Global account information for Raidarr - Miraheze Meta | meta.miraheze.org [11:19:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:20:23] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.136 second response time [11:26:33] @raidarr: yes, it loads for me [11:26:40] Yep, I can access it. [11:26:59] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.017 second response time [11:28:51] Hm, breaks for me. Good to know, will troubleshoot on my end. [11:29:56] Cache issue it seems like. [11:47:15] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.027 second response time [11:54:00] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.010 second response time [11:57:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:05:39] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.483 second response time [12:07:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:12:52] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:57] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 13.00, 10.54, 6.51 [12:15:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:20:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:27:38] PROBLEM - wiki.dunmoore.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.dunmoore.net' expires in 15 day(s) (Tue 22 Feb 2022 12:20:38 GMT +0000). [12:28:23] PROBLEM - www.winenjoy.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.winenjoy.net' expires in 15 day(s) (Tue 22 Feb 2022 12:16:42 GMT +0000). [12:29:52] PROBLEM - www.recyclerie-uzege.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.recyclerie-uzege.org' expires in 15 day(s) (Tue 22 Feb 2022 12:17:40 GMT +0000). [12:31:25] PROBLEM - wiki.simorgh.me - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.simorgh.me' expires in 15 day(s) (Tue 22 Feb 2022 12:19:43 GMT +0000). [12:33:32] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.020 second response time [12:33:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a5bb1e7ce5c6...5ff74fa0b8ac [12:33:57] [02miraheze/ssl] 07MirahezeSSLBot 035ff74fa - Bot: Update SSL cert for wiki.dunmoore.net [12:39:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.46, 8.74, 7.14 [12:40:10] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.012 second response time [12:40:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5ff74fa0b8ac...dd10eaa0a521 [12:40:37] [02miraheze/ssl] 07MirahezeSSLBot 03dd10eaa - Bot: Update SSL cert for www.recyclerie-uzege.org [12:42:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.36, 9.87, 7.87 [12:43:12] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.039 second response time [12:43:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:43:53] PROBLEM - pt.graalmilitary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pt.graalmilitary.com' expires in 15 day(s) (Tue 22 Feb 2022 12:22:58 GMT +0000). [12:44:28] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 3.07, 4.63, 7.76 [12:45:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.86, 9.69, 8.03 [12:46:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/dd10eaa0a521...e26885a7f9a3 [12:46:42] [02miraheze/ssl] 07MirahezeSSLBot 03e26885a - Bot: Update SSL cert for wiki.simorgh.me [12:46:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.57, 9.32, 7.62 [12:47:27] PROBLEM - pwsc.polandballwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pwsc.polandballwiki.com' expires in 15 day(s) (Tue 22 Feb 2022 12:22:18 GMT +0000). [12:47:52] PROBLEM - wiki.umbranet.mandelbrotgames.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.umbranet.mandelbrotgames.com' expires in 15 day(s) (Tue 22 Feb 2022 12:18:37 GMT +0000). [12:48:09] PROBLEM - staff.polandballwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'staff.polandballwiki.com' expires in 15 day(s) (Tue 22 Feb 2022 12:21:32 GMT +0000). [12:48:51] PROBLEM - wiki.ct777.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ct777.cf' expires in 15 day(s) (Tue 22 Feb 2022 12:24:49 GMT +0000). [12:48:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.63, 9.80, 8.55 [12:49:59] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e26885a7f9a3...83f3063fc335 [12:50:01] [02miraheze/ssl] 07MirahezeSSLBot 0383f3063 - Bot: Update SSL cert for www.winenjoy.net [12:50:09] PROBLEM - popira.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'popira.info' expires in 15 day(s) (Tue 22 Feb 2022 12:23:55 GMT +0000). [12:50:14] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:50:14] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 2.02, 3.11, 6.11 [12:51:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.11, 8.28, 7.89 [12:53:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:54:27] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.99, 2.07, 1.52 [12:55:42] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.39, 9.38, 8.32 [12:55:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/83f3063fc335...e09b3655722b [12:55:53] [02miraheze/ssl] 07MirahezeSSLBot 03e09b365 - Bot: Update SSL cert for pwsc.polandballwiki.com [12:57:27] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.61, 1.97, 1.59 [12:57:53] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e09b3655722b...333e0ec6241c [12:57:55] [02miraheze/ssl] 07MirahezeSSLBot 03333e0ec - Bot: Update SSL cert for wiki.umbranet.mandelbrotgames.com [12:57:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.24, 9.59, 8.87 [12:57:59] RECOVERY - wiki.dunmoore.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dunmoore.net' will expire on Sat 07 May 2022 11:33:49 GMT +0000. [12:58:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.88, 9.57, 8.57 [12:59:44] RECOVERY - www.recyclerie-uzege.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.recyclerie-uzege.org' will expire on Sat 07 May 2022 11:40:29 GMT +0000. [13:00:26] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.86, 1.53, 1.49 [13:00:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:00:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.98, 8.93, 8.75 [13:02:09] RECOVERY - wiki.simorgh.me - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.simorgh.me' will expire on Sat 07 May 2022 11:46:35 GMT +0000. [13:02:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/333e0ec6241c...ad61bd30b9e7 [13:02:42] [02miraheze/ssl] 07MirahezeSSLBot 03ad61bd3 - Bot: Update SSL cert for staff.polandballwiki.com [13:04:28] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.78, 7.83, 8.15 [13:05:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ad61bd30b9e7...d09f0b020d5e [13:05:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:05:36] [02miraheze/ssl] 07MirahezeSSLBot 03d09f0b0 - Bot: Update SSL cert for popira.info [13:06:09] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d09f0b020d5e...e52f4f595d67 [13:06:11] [02miraheze/ssl] 07MirahezeSSLBot 03e52f4f5 - Bot: Update SSL cert for pt.graalmilitary.com [13:06:46] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.19, 7.15, 8.00 [13:12:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e52f4f595d67...b678fed3b2af [13:12:28] [02miraheze/ssl] 07MirahezeSSLBot 03b678fed - Bot: Update SSL cert for wiki.ct777.cf [13:17:08] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 2.276 second response time [13:22:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:23:40] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.007 second response time [13:27:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:29:28] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.030 second response time [13:30:19] RECOVERY - www.winenjoy.net - LetsEncrypt on sslhost is OK: OK - Certificate 'www.winenjoy.net' will expire on Sat 07 May 2022 11:49:53 GMT +0000. [13:35:14] !log MariaDB [gratisdatawiki]> REPLACE INTO /*_*/wb_id_counters VALUE((SELECT COALESCE(MAX(CAST(SUBSTRING(`page_title`, 2) AS UNSIGNED)), 0) FROM `page` WHERE `page_namespace` = 120), 'wikibase-item'); [13:35:16] !log MariaDB [gratisdatawiki]> REPLACE INTO /*_*/wb_id_counters VALUE((SELECT COALESCE(MAX(CAST(SUBSTRING(`page_title`, 2) AS UNSIGNED)), 0) FROM `page` WHERE `page_namespace` = 122), 'wikibase-property'); [13:35:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:35:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:36:24] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 9.92, 8.69, 6.73 [13:36:53] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/a1616f434524 [13:36:54] [02miraheze/mw-config] 07Reception123 03a1616f4 - allowEntityImport set to false for gratisdatawiki per T8741 [13:36:56] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config [13:36:57] [02mw-config] 07Reception123 opened pull request 03#4470: allowEntityImport set to false for gratisdatawiki per T8741 - 13https://github.com/miraheze/mw-config/pull/4470 [13:38:00] miraheze/mw-config - Reception123 the build passed. [13:38:21] [02mw-config] 07Reception123 closed pull request 03#4470: allowEntityImport set to false for gratisdatawiki per T8741 - 13https://github.com/miraheze/mw-config/pull/4470 [13:38:22] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config [13:38:24] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [13:38:25] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/36c9f51ff561...7219c757b031 [13:38:27] [02miraheze/mw-config] 07Reception123 037219c75 - allowEntityImport set to false for gratisdatawiki per T8741 (#4470) [13:39:16] miraheze/mw-config - Reception123 the build passed. [13:40:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:43:03] RECOVERY - pt.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.graalmilitary.com' will expire on Sat 07 May 2022 12:06:04 GMT +0000. [13:45:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:46:44] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:56] RECOVERY - pwsc.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pwsc.polandballwiki.com' will expire on Sat 07 May 2022 11:55:45 GMT +0000. [13:48:00] !log [@test101] starting deploy of {'config': True} to skip [13:48:01] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [13:48:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:48:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:48:25] RECOVERY - staff.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'staff.polandballwiki.com' will expire on Sat 07 May 2022 12:02:35 GMT +0000. [13:48:39] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c6d6e87e77df...f0391e66b972 [13:48:41] [02miraheze/puppet] 07JohnFLewis 03f0391e6 - T7889: Change CSP source for reviwiki [13:48:41] RECOVERY - wiki.umbranet.mandelbrotgames.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.umbranet.mandelbrotgames.com' will expire on Sat 07 May 2022 11:57:48 GMT +0000. [13:49:29] RECOVERY - wiki.ct777.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ct777.cf' will expire on Sat 07 May 2022 12:12:20 GMT +0000. [13:50:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.64, 3.33 [13:51:33] RECOVERY - popira.info - LetsEncrypt on sslhost is OK: OK - Certificate 'popira.info' will expire on Sat 07 May 2022 12:05:29 GMT +0000. [13:52:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:52:54] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 6.100 second response time [13:53:04] !log [@mwtask111] starting deploy of {'config': True} to all [13:53:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:53:20] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 16s [13:53:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:53:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.59, 2.97, 3.11 [13:53:59] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f0391e66b972...9d31d2082086 [13:54:01] [02miraheze/puppet] 07JohnFLewis 039d31d20 - T8008: Add fbcdn.net to img-src only [13:54:36] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [13:56:10] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9d31d2082086...f02ff2d837d2 [13:56:11] [02miraheze/puppet] 07JohnFLewis 03f02ff2d - T8143: add bing maps to iframe-src [13:59:56] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.008 second response time [14:05:56] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 4.218 second response time [14:07:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:08:19] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.77, 9.21, 8.10 [14:11:29] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.21, 8.30, 7.96 [14:11:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:19:06] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [14:21:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:23:36] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.014 second response time [14:26:41] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.016 second response time [14:26:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.86, 4.01, 3.43 [14:30:22] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.85, 4.82, 2.64 [14:32:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.67, 8.93, 8.52 [14:35:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.50, 9.99, 9.02 [14:36:16] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.99, 3.78 [14:38:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.17, 9.14, 8.86 [14:38:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 4.00, 3.51 [14:39:23] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 4.07, 3.86 [14:39:35] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.60, 1.43, 1.83 [14:42:13] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:42:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:42:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.99, 3.82, 3.77 [14:42:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.57, 1.67, 1.82 [14:44:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.12, 7.87, 8.44 [14:45:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 3.86, 3.78 [14:48:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.52, 3.13, 3.51 [14:48:48] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.21, 1.74, 1.88 [14:51:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.10, 2.92, 3.38 [14:52:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.10, 8.57, 8.64 [14:52:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.41, 9.01, 8.27 [14:57:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.29, 3.76 [14:57:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:58:06] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.80, 1.26, 1.65 [14:59:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.59, 8.38, 8.26 [15:00:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 4.06, 4.02 [15:01:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.30, 9.02, 8.90 [15:02:34] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.54, 3.53, 3.19 [15:03:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.97, 3.99 [15:04:10] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.028 second response time [15:05:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.14, 3.66, 3.29 [15:06:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.61, 9.66, 8.89 [15:08:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:08:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.88, 3.30, 3.22 [15:10:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 4.37, 4.15 [15:16:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.67, 4.43, 3.80 [15:18:13] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:19:39] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.46, 9.57, 9.18 [15:19:50] PROBLEM - cp31 Disk Space on cp31 is WARNING: DISK WARNING - free space: / 4254 MB (10% inode=96%); [15:21:35] RECOVERY - cloud11 SMART on cloud11 is OK: OK: [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean [15:22:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.18, 9.73, 9.35 [15:26:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.71, 3.30, 3.60 [15:26:28] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.12, 1.94, 1.61 [15:27:20] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.008 second response time [15:28:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:29:17] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.42, 10.45, 9.70 [15:31:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:32:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.95, 3.93, 3.93 [15:32:55] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.51, 1.70, 1.64 [15:33:50]