[00:02:57] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:05:20] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [00:05:43] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.33, 3.10 [00:07:38] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.879 second response time [00:10:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.57, 3.54, 3.21 [00:12:01] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.22, 2.97, 3.04 [00:12:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [00:12:59] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [00:13:09] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:17] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:14:43] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:14:59] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.603 second response time [00:15:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:15:04] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.335 second response time [00:15:16] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.009 second response time [00:20:36] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:21:02] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:22:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:23:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:32:56] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:34:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.13, 2.95 [00:35:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:36:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.05, 2.74, 2.83 [00:47:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:49:02] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.38, 7.06, 5.54 [00:49:31] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:50:18] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:50:30] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:51:06] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:51:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.72, 4.03, 3.48 [00:52:22] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.369 second response time [00:52:33] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 7.277 second response time [00:53:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:53:23] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:54:58] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.51, 7.27, 6.16 [00:57:20] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.60, 7.07, 5.77 [00:57:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.95, 3.65 [00:57:55] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.57, 6.66, 5.53 [00:58:55] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.27, 7.56, 6.52 [01:00:13] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [01:00:54] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.90, 7.50, 6.61 [01:01:53] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.15, 7.56, 6.16 [01:02:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:02:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.15, 18.82, 23.60 [01:02:52] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.34, 7.62, 6.74 [01:04:51] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.00, 7.11, 6.67 [01:05:51] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.78, 7.62, 6.54 [01:06:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.99, 4.69, 5.82 [01:07:06] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.07, 6.65, 6.24 [01:11:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.90, 4.44, 3.95 [01:14:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.76, 16.16, 19.57 [01:17:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.85, 3.92, 3.84 [01:18:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.96, 5.87, 5.61 [01:19:39] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.21, 6.34, 6.61 [01:19:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.83, 4.02, 3.87 [01:20:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.79, 19.80, 19.86 [01:20:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.14, 5.65, 5.57 [01:22:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:22:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.69, 18.88, 19.48 [01:22:47] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.96, 5.66, 6.48 [01:23:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.65, 3.80, 3.85 [01:25:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.24, 4.15, 3.99 [01:30:54] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.98, 4.19, 4.88 [01:35:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.01, 3.67, 3.89 [01:37:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.86, 3.94 [01:39:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.36, 3.64, 3.84 [01:49:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.63, 3.64 [01:52:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:55:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.86, 3.79, 3.78 [02:01:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.88, 3.80 [02:03:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.14, 3.66, 3.73 [02:05:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.97, 4.00, 3.84 [02:11:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.61, 3.57, 3.76 [02:12:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:19:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.32, 3.83, 3.80 [02:25:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.39, 3.62, 3.83 [02:30:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.81, 1.37, 1.16 [02:32:40] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.40, 1.73, 1.32 [02:33:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.78, 3.82 [02:34:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.15, 1.52, 1.29 [02:35:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:37:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.62, 3.51, 3.72 [02:40:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:43:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:48:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:55:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:57:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.41, 3.54, 3.56 [02:59:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.59, 3.58 [03:00:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:05:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.63, 3.05, 3.34 [03:09:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.17, 3.79, 3.60 [03:09:58] [02mw-config] 07Universal-Omega synchronize pull request 03#4296: Drop test3wiki from config + configure beta - 13https://git.io/JyeJO [03:10:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JyeEZ [03:10:01] [02miraheze/mw-config] 07Universal-Omega 03ce4fd7f - Update LocalSettings.php [03:11:01] miraheze/mw-config - Universal-Omega the build passed. [03:13:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.18, 3.75, 3.66 [03:15:18] [02mw-config] 07Universal-Omega synchronize pull request 03#4293: Configure beta realm for SiteConfiguration tag - 13https://git.io/JDbe3 [03:15:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JyeEp [03:15:21] [02miraheze/mw-config] 07Universal-Omega 037b1af4f - Update LocalSettings.php [03:16:29] miraheze/mw-config - Universal-Omega the build passed. [03:16:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:18:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jyeui [03:18:08] [02miraheze/mw-config] 07Universal-Omega 038121ece - Update LocalSettings.php [03:18:09] [02mw-config] 07Universal-Omega synchronize pull request 03#4296: Drop test3wiki from config + configure beta - 13https://git.io/JyeJO [03:18:15] [02mw-config] 07Universal-Omega edited pull request 03#4296: Drop test3wiki from config + configure betaheze - 13https://git.io/JyeJO [03:18:28] [02mw-config] 07Universal-Omega edited pull request 03#4293: Configure betaheze realm for SiteConfiguration tag - 13https://git.io/JDbe3 [03:19:11] miraheze/mw-config - Universal-Omega the build passed. [03:19:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.56, 4.05, 3.76 [03:21:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:21:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.98, 3.77 [03:35:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.99, 3.13, 3.35 [03:49:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.11, 3.42, 3.42 [03:55:18] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.69, 3.03, 3.32 [04:13:35] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.02, 1.47, 1.10 [04:15:32] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.53, 1.40, 1.11 [04:31:41] PROBLEM - wiki.chouverse.cludisciples.cyou - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.chouverse.cludisciples.cyou' expires in 15 day(s) (Fri 07 Jan 2022 04:27:49 GMT +0000). [04:31:59] PROBLEM - mabi.myomi.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mabi.myomi.net' expires in 15 day(s) (Fri 07 Jan 2022 04:29:15 GMT +0000). [04:32:00] PROBLEM - wiki.misslessfighters.club - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.misslessfighters.club' expires in 15 day(s) (Fri 07 Jan 2022 04:27:02 GMT +0000). [04:34:41] PROBLEM - thepolicyhub.org.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'thepolicyhub.org.uk' expires in 15 day(s) (Fri 07 Jan 2022 04:31:02 GMT +0000). [04:35:03] PROBLEM - wiki.homiedb.party - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.homiedb.party' expires in 15 day(s) (Fri 07 Jan 2022 04:28:35 GMT +0000). [04:37:29] PROBLEM - wiki.consid.vn - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.consid.vn' expires in 15 day(s) (Fri 07 Jan 2022 04:30:19 GMT +0000). [04:37:37] PROBLEM - zamlung.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'zamlung.tk' expires in 15 day(s) (Fri 07 Jan 2022 04:31:55 GMT +0000). [04:57:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.62, 3.41, 3.06 [04:59:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.96, 2.99, 2.96 [05:03:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.36, 3.10 [05:05:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.32, 3.16, 3.05 [05:12:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.55, 2.71, 1.75 [05:16:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.96, 1.65, 1.53 [06:04:32] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.97, 5.41, 3.39 [06:06:32] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.49, 6.35, 3.97 [06:09:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:12:32] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.36, 7.55, 5.36 [06:14:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:22:32] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.14, 6.42, 5.87 [06:26:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:31:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:41:33] RECOVERY - mw13 APT on mw13 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:45:01] RECOVERY - mw9 APT on mw9 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:45:31] RECOVERY - mw12 APT on mw12 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:47:46] RECOVERY - mw11 APT on mw11 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:52:16] RECOVERY - mw10 APT on mw10 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:54:32] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.69, 6.64, 5.91 [06:56:32] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.80, 7.15, 6.17 [06:58:32] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.74, 6.48, 6.05 [06:59:23] RECOVERY - bacula2 Bacula Static on bacula2 is OK: OK: Diff, 3684105 files, 223.5GB, 2021-12-22 06:58:00 (1.4 minutes ago) [07:05:35] RECOVERY - bacula2 Bacula Private Git on bacula2 is OK: OK: Full, 6809 files, 57.38MB, 2021-12-22 07:05:00 (35.0 seconds ago) [07:58:14] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=primalfeararkwiki --username-prefix wikia:arksurvivalevolved --report 1 --no-updates /home/reception/0001-ark-templates-pfrequired.xml (START) [07:58:16] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=primalfeararkwiki --username-prefix wikia:arksurvivalevolved --report 1 --no-updates /home/reception/0001-ark-templates-pfrequired.xml (END - exit=256) [07:58:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:58:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:58:38] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=primalfeararkwiki --username-prefix wikia:arksurvivalevolved --report 1 --no-updates /home/reception/primalfear/0001-ark-templates-pfrequired.xml (START) [07:58:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:59:04] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=primalfeararkwiki --username-prefix wikia:arksurvivalevolved --report 1 --no-updates /home/reception/primalfear/0002-ark-modules-pf.xml (START) [07:59:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:59:20] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=primalfeararkwiki --username-prefix wikia:arksurvivalevolved --report 1 --no-updates /home/reception/primalfear/0003-ark-pf-content.xml (START) [07:59:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:59:28] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=primalfeararkwiki --username-prefix wikia:arksurvivalevolved --report 1 --no-updates /home/reception/primalfear/0002-ark-modules-pf.xml (END - exit=0) [07:59:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:00:52] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=primalfeararkwiki --username-prefix wikia:arksurvivalevolved --report 1 --no-updates /home/reception/primalfear/0001-ark-templates-pfrequired.xml (END - exit=0) [08:00:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:04:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:14:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:19:58] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=primalfeararkwiki --username-prefix wikia:arksurvivalevolved --report 1 --no-updates /home/reception/primalfear/0003-ark-pf-content.xml (END - exit=0) [08:20:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:20:15] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=primalfeararkwiki (START) [08:20:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:37:30] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=primalfeararkwiki (END - exit=0) [08:37:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:50:52] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.35, 6.86, 6.30 [08:52:48] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.82, 6.45, 6.22 [10:57:12] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JyvHo [10:57:13] [02miraheze/dns] 07Reception123 03d0a2aee - add hoseki.wiki zone [11:05:26] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±2] 13https://git.io/Jyv7q [11:05:28] [02miraheze/ssl] 07Reception123 033b95037 - add wiki.leapit.no cert [11:05:29] [02miraheze/ssl] 07Reception123 03a6d943d - add hoseki.wiki cert [11:05:34] PROBLEM - ns1 SSH on ns1 is CRITICAL: Server answer [11:06:24] err [11:06:26] Reception123: [11:07:21] RhinosF1: I've never seen that error before [11:07:32] but I indeed can't SSH to ns1 [11:07:33] Reception123: is ssh on ns1 working [11:07:34] RECOVERY - ns1 SSH on ns1 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [11:07:38] it's back [11:07:51] yeah, fixed itself now [11:08:15] Reception123: I think going ahead with https://github.com/miraheze/puppet/pull/2170 might be best [11:08:15] [url] php: lower pm.max_requests to 1,000 by RhinosF1 · Pull Request #2170 · miraheze/puppet · GitHub | github.com [11:08:29] i can do the bullseye VMs if you salt the rest [11:09:01] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=alicesoftwiki --username-prefix wikia:alicesoft --report 1 --no-updates /home/reception/alicesoft_pages_full.xml (START) [11:09:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:11:41] RhinosF1: sure [11:11:56] [02puppet] 07Reception123 closed pull request 03#2170: php: lower pm.max_requests to 1,000 - 13https://git.io/JDj7S [11:11:58] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jyv5B [11:11:59] [02miraheze/puppet] 07RhinosF1 03226efc6 - php: lower pm.max_requests to 1,000 (#2170) [11:12:19] Reception123: lmk when pulled [11:12:23] RhinosF1: done [11:12:49] Reception123: run puppet and restart php pls on mw*, i'll do test/mw8/mwask1 [11:13:13] done puppet, will restart after [11:13:22] RhinosF1: also a large file import is coming (5GB zipped) [11:13:33] so if it's larger than like 10GB unzipped I'll have to pause it for now [11:13:54] ok [11:14:16] !log restart php on bullseye mw VMs [11:14:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:15:17] !log restart php on mw* (buster) [11:15:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:19:23] .in 10mins move graph to 1 [11:19:23] RhinosF1: Okay, will remind at 2021-12-22 - 11:29:23GMT [11:21:35] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.22, 2.65 [11:23:30] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.33, 2.85, 2.58 [11:29:23] RhinosF1: move graph to 1 [11:33:41] .in 1hour15mins move to 3 [11:33:41] RhinosF1: Okay, will remind at 2021-12-22 - 12:48:41GMT [11:44:51] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'm.miraheze.org' expires in 15 day(s) (Fri 07 Jan 2022 11:42:07 GMT +0000). [12:48:41] RhinosF1: move to 3 [12:53:03] .in 90mins up to 6 [12:53:03] RhinosF1: Okay, will remind at 2021-12-22 - 14:23:03GMT [13:20:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.71, 3.74, 3.17 [13:22:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.51, 3.16 [13:24:25] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.66, 3.23, 3.10 [13:24:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:29:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:36:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.50, 3.27 [13:38:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 3.93, 3.47 [13:39:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.60, 3.40 [13:41:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.63, 3.11, 3.24 [13:47:45] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.49, 6.19, 5.15 [13:48:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.01, 6.13, 5.17 [13:49:45] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.46, 6.21, 5.28 [13:50:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.05, 6.21, 5.32 [14:04:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:07:00] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.59, 3.24 [14:08:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.85, 3.38 [14:10:51] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.61, 3.34, 3.25 [14:18:35] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=ysmwiki --update (END - exit=65280) [14:18:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:18:46] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=ysmwikiwiki --update (END - exit=0) [14:18:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:19:47] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.37, 6.55, 5.79 [14:21:46] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.04, 6.35, 5.81 [14:23:06] RhinosF1: up to 6 [14:24:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.70, 3.38 [14:34:09] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=alicesoftwiki --username-prefix wikia:alicesoft --report 1 --no-updates /home/reception/alicesoft_pages_full.xml (END - exit=0) [14:34:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:34:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.95, 3.82, 3.71 [14:34:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:40:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.55, 3.60 [14:42:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.14, 3.40, 3.54 [14:42:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.09, 6.48, 5.90 [14:44:48] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.38, 6.30, 5.92 [14:45:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:48:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.17, 4.08, 3.77 [14:50:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:52:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.38, 3.51, 3.63 [14:52:45] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.98, 6.27, 5.94 [14:53:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:53:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.16, 6.58, 5.79 [14:54:42] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.31, 6.74, 6.13 [14:55:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.25, 6.23, 5.56 [14:55:50] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.90, 6.36, 5.81 [14:57:32] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.05, 5.91, 5.54 [14:58:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:58:36] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.64, 6.65, 6.27 [15:04:24] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.90, 6.89, 6.49 [15:04:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.33, 4.12, 3.79 [15:04:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:06:21] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.99, 6.22, 6.29 [15:06:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.22, 3.33, 3.54 [15:09:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:12:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.65, 3.61 [15:14:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.57, 3.59 [15:18:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.15, 3.18, 3.40 [15:18:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:22:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.79, 3.83, 3.60 [15:26:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.85, 3.50, 3.52 [15:30:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.94, 3.21, 3.40 [15:33:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:40:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.96, 3.60 [15:41:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:44:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 4.00, 3.70 [15:46:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:50:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.62, 3.58 [15:52:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.53, 6.54, 5.84 [15:54:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.35, 5.99, 5.72 [15:56:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.32, 3.70, 3.64 [16:00:02] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.87, 6.35, 5.67 [16:00:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.88, 3.70 [16:02:01] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.93, 5.73, 5.52 [16:02:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.48, 3.57 [16:06:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.29, 3.10, 3.37 [16:17:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.39, 3.31 [16:19:13] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.13, 3.40, 3.33 [16:23:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.85, 4.00, 3.58 [16:25:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.03, 3.62, 3.49 [16:30:47] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.86, 4.30, 3.77 [16:32:43] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.09, 3.79, 3.65 [16:36:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.76, 3.65 [16:38:30] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.76, 3.37, 3.52 [16:42:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.75, 2.76, 3.25 [16:52:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.55, 3.30, 3.26 [16:53:41] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.34, 6.31, 5.03 [16:54:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.50, 6.71, 5.73 [16:54:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.18, 2.80, 3.07 [16:55:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:55:41] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.28, 5.92, 5.04 [17:02:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.32, 3.59, 3.26 [17:04:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.58, 6.67, 6.25 [17:14:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.73, 3.81, 3.70 [17:18:37] PROBLEM - wiki.minkyu.kim - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:33] PROBLEM - wiki.homiedb.party - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:41] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.15, 6.57, 5.75 [17:20:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:21:41] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.83, 5.89, 5.60 [17:21:42] PROBLEM - wiki.homiedb.party - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/pyth [17:21:42] ist-packages/dns/resolver.py", line 898, in query raise NoNameservers(request=request, errors=errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wiki.homiedb.party. IN CNAME: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS o [17:21:42] ion timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [17:25:42] PROBLEM - wiki.cyberfurs.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:28:25] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.32, 3.00, 3.37 [17:28:56] RECOVERY - wiki.homiedb.party - reverse DNS on sslhost is OK: SSL OK - wiki.homiedb.party reverse DNS resolves to cp21.miraheze.org - CNAME OK [17:32:27] RECOVERY - wiki.minkyu.kim - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minkyu.kim' will expire on Fri 14 Jan 2022 04:26:43 GMT +0000. [17:33:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:38:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.88, 3.59, 3.37 [17:38:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:40:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.74, 3.18, 3.24 [17:41:37] PROBLEM - wiki.minkyu.kim - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:45:41] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.66, 6.24, 5.77 [17:47:41] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.22, 5.35, 5.49 [17:48:31] RECOVERY - wiki.minkyu.kim - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minkyu.kim' will expire on Fri 14 Jan 2022 04:26:43 GMT +0000. [17:52:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:52:50] RECOVERY - wiki.cyberfurs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cyberfurs.org' will expire on Fri 14 Jan 2022 06:38:38 GMT +0000. [17:54:50] PROBLEM - wiki.homiedb.party - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.homiedb.party' expires in 15 day(s) (Fri 07 Jan 2022 04:28:35 GMT +0000). [17:56:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.49, 3.31 [18:00:46] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.91, 3.90, 3.50 [18:04:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.95, 3.65 [18:07:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:08:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.93, 4.13, 3.76 [18:10:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.86, 3.95, 3.73 [18:21:49] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.30, 6.93, 6.30 [18:22:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.42, 3.68, 3.63 [18:23:46] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.06, 6.55, 6.23 [18:24:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.00, 3.35, 3.51 [18:24:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:29:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:30:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.56, 2.90, 3.28 [18:37:10] !log added raidarr's email to cvt@ [18:37:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:43:37] PROBLEM - wiki.homiedb.party - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:46:15] PROBLEM - wiki.minkyu.kim - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:47:00] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 3.53, 3.31 [18:48:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.93, 3.62, 3.36 [18:50:51] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.67, 3.99, 3.53 [18:53:06] RECOVERY - wiki.minkyu.kim - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minkyu.kim' will expire on Fri 14 Jan 2022 04:26:43 GMT +0000. [18:56:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.68, 3.67, 3.60 [18:59:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:02:25] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.88, 2.84, 3.29 [19:02:50] PROBLEM - graylog2 APT on graylog2 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (4 critical updates). [19:04:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:04:43] PROBLEM - wiki.homiedb.party - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.homiedb.party' expires in 15 day(s) (Fri 07 Jan 2022 04:28:35 GMT +0000). [19:07:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:12:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:24:44] PROBLEM - puppet3 APT on puppet3 is CRITICAL: APT CRITICAL: 8 packages available for upgrade (4 critical updates). [19:30:52] PROBLEM - mail2 APT on mail2 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (4 critical updates). [19:36:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.89, 3.74, 3.43 [19:42:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.50, 3.27, 3.36 [19:52:39] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 22.37, 13.61, 6.53 [19:53:17] I got a few DB errors but it seems to have resolved [19:53:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:54:35] Agent: db errors? [19:55:32] RhinosF1: on Meta for a few moments https://usercontent.irccloud-cdn.com/file/MRZjlU8N/Screenshot_20211222-135202%7E2.png [19:55:39] Ok [19:57:02] Agent: yeah we know what that is [19:57:15] Now we solved mediawiki being stupid and OOM'ing [19:57:17] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=db11.miraheze.org&var-port=9100&viewPanel=78&from=now-30m&to=now-1m [19:57:25] JohnLewis, paladox: ^ [19:58:32] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 3.19, 6.33, 5.42 [19:58:58] Noted, but not a lot we can do until new Infra. That will have increase memory allocations [19:59:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.74, 3.44 [19:59:13] If it becomes more frequent, then we can look at alternatives [20:00:24] JohnLewis: yeah I know [20:00:25] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.58, 6.68, 5.83 [20:00:29] It's been stable for a while [20:00:32] Hey Voidwalker [20:00:43] hey [20:01:00] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.60, 3.90, 3.53 [20:01:27] Voidwalker: will you fix void bot [20:01:38] Oh you did [20:01:42] I missed it today [20:01:46] yeah, was gonna say [20:02:17] Also happy Christmas [20:02:22] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.24, 7.71, 6.31 [20:02:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.19, 3.86, 3.57 [20:03:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:04:19] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.26, 7.01, 6.22 [20:06:16] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.40, 5.99, 5.94 [20:06:47] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.70, 3.57 [20:08:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.48, 3.61, 3.55 [20:17:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:24:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.41, 2.08, 1.35 [20:26:55] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.59, 1.51, 1.22 [20:28:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.55, 2.90, 3.29 [20:32:25]