[00:00:00] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: connect to address 51.195.220.68 port 5666: Connection refusedconnect to host 51.195.220.68 port 5666: Connection refused [00:00:07] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: connect to address 51.195.220.68 port 5666: Connection refusedconnect to host 51.195.220.68 port 5666: Connection refused [00:00:11] PROBLEM - mail2 Puppet on mail2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:00:19] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: connect to address 51.195.220.68 port 5666: Connection refusedconnect to host 51.195.220.68 port 5666: Connection refused [00:00:25] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: connect to address 51.195.220.68 port 5666: Connection refusedconnect to host 51.195.220.68 port 5666: Connection refused [00:00:30] PROBLEM - cp20 APT on cp20 is CRITICAL: connect to address 51.195.220.68 port 5666: Connection refusedconnect to host 51.195.220.68 port 5666: Connection refused [00:00:48] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: connect to address 51.195.220.68 port 5666: Connection refusedconnect to host 51.195.220.68 port 5666: Connection refused [00:00:54] PROBLEM - cp20 NTP time on cp20 is CRITICAL: connect to address 51.195.220.68 port 5666: Connection refusedconnect to host 51.195.220.68 port 5666: Connection refused [00:01:15] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:10] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.16, 3.69, 2.68 [00:03:17] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.84, 1.74, 1.44 [00:05:09] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.70, 3.22, 2.63 [00:05:15] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.56, 1.70, 1.46 [00:08:45] PROBLEM - ldap2 Puppet on ldap2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:18:01] PROBLEM - cp20 Varnish Backends on cp20 is WARNING: No backends detected. If this is an error, see readme.txt [00:18:03] RECOVERY - cp20 Stunnel Http for mon2 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35370 bytes in 0.032 second response time [00:18:12] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.023 second response time [00:18:31] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.584 second response time [00:18:31] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 1.591 second response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::929a [00:18:33] RECOVERY - cp20 APT on cp20 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:18:52] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 21% [00:18:58] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset -9.733438492e-05 secs [00:19:04] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.01, 1.55, 1.40 [00:19:11] PROBLEM - cp20 ferm_active on cp20 is UNKNOWN: NRPE: Unable to read output [00:19:23] PROBLEM - cp20 Puppet on cp20 is UNKNOWN: NRPE: Unable to read output [00:19:26] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.41, 0.65, 0.35 [00:19:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.81, 3.59, 3.30 [00:19:28] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 21077 MB (53% inode=97%); [00:19:32] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3124 bytes in 0.024 second response time [00:19:36] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.014 second response time [00:19:49] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [00:19:55] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.815 second response time [00:19:56] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw8 mw11 [00:20:15] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:21:02] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.28, 1.43, 1.37 [00:21:08] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [00:21:19] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:21:52] PROBLEM - cp20 Varnish Backends on cp20 is WARNING: No backends detected. If this is an error, see readme.txt [00:23:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 3.95, 3.54 [00:23:32] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyLqV [00:23:34] [02miraheze/dns] 07paladox 03cf6634d - Revert "depool gb" [00:23:49] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 7 backends are down. mw8 mw9 mw10 mw11 mw12 mw13 mediawiki [00:23:55] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:58] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 98% [00:24:06] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: connect to address localhost and port 8093: Connection refusedHTTP CRITICAL - Unable to open TCP socket [00:24:06] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: connect to address localhost and port 8088: Connection refusedHTTP CRITICAL - Unable to open TCP socket [00:24:08] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 5680 bytes in 0.007 second response time [00:24:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:24:12] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 7 backends are down. mw8 mw9 mw10 mw11 mw12 mw13 mediawiki [00:24:48] PROBLEM - cp21 Stunnel Http for mon2 on cp21 is CRITICAL: connect to address localhost and port 8201: Connection refusedHTTP CRITICAL - Unable to open TCP socket [00:24:54] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: connect to address localhost and port 8085: Connection refusedHTTP CRITICAL - Unable to open TCP socket [00:25:06] RECOVERY - cloud5 Puppet on cloud5 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [00:25:14] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: connect to address localhost and port 8087: Connection refusedHTTP CRITICAL - Unable to open TCP socket [00:25:16] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: connect to address 198.244.148.90 port 5666: Connection refusedconnect to host 198.244.148.90 port 5666: Connection refused [00:25:17] RECOVERY - gluster4 Puppet on gluster4 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [00:25:24] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [00:25:41] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [00:25:49] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: connect to address localhost and port 8086: Connection refusedHTTP CRITICAL - Unable to open TCP socket [00:25:59] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.535 second response time [00:26:13] PROBLEM - cp21 Varnish Backends on cp21 is WARNING: No backends detected. If this is an error, see readme.txt [00:26:48] RECOVERY - cp21 Stunnel Http for mon2 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35377 bytes in 0.142 second response time [00:26:49] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.019 second response time [00:27:13] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.315 second response time [00:27:17] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.843 second response time [00:28:07] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.833 second response time [00:28:12] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw9 [00:28:12] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3125 bytes in 4.347 second response time [00:28:21] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [00:28:59] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.47, 1.88, 1.57 [00:29:15] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [00:29:38] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:29:52] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:29:58] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 12% [00:30:10] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.012 second response time [00:30:11] RECOVERY - mail2 Puppet on mail2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:30:12] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 9 backends are healthy [00:30:59] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.70, 1.48, 1.46 [00:31:01] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.01, 3.22, 3.36 [00:31:37] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.189 second response time [00:31:51] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.598 second response time [00:31:59] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.447 second response time [00:32:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:34:25] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:36:44] RECOVERY - ldap2 Puppet on ldap2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:38:52] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:40:10] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:21] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:41:38] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.010 second response time [00:44:12] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw10 mw11 [00:44:22] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 3 backends are down. mw11 mw12 mw13 [00:45:49] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 9 backends are healthy [00:45:59] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.006 second response time [00:46:12] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 9 backends are healthy [00:46:17] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.203 second response time [00:46:20] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [00:46:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:46:52] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:50:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:54:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:56:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.48, 2.36, 1.55 [00:57:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.37, 3.03 [00:58:55] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.28, 1.97, 1.50 [01:00:55] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.74, 1.50, 1.38 [01:03:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.57, 3.66, 3.24 [01:04:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:05:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.73, 3.67, 3.30 [01:08:51] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.76, 3.40 [01:10:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.18, 17.04, 23.25 [01:12:38] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.49, 19.83, 23.52 [01:12:42] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.49, 3.26, 3.28 [01:14:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:14:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.66, 21.76, 23.82 [01:16:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.61, 3.45 [01:18:30] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.04, 4.03, 3.61 [01:19:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:19:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.79, 4.73, 5.87 [01:20:38] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.33, 23.52, 23.56 [01:21:51] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.61, 5.67, 6.06 [01:23:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:23:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.61, 5.13, 5.84 [01:24:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.34, 21.70, 22.95 [01:28:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:33:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.77, 4.35, 5.03 [01:35:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:37:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.63, 5.25, 5.29 [01:39:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.19, 4.30, 4.95 [01:40:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:40:38] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.69, 17.54, 19.91 [01:50:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.76, 2.38, 1.57 [01:52:55] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.15, 1.91, 1.50 [01:54:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.62, 3.44, 3.88 [01:54:55] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.11, 1.59, 1.43 [01:59:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:04:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:06:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.63, 3.03, 3.39 [02:12:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.48, 3.35, 3.41 [02:15:03] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 50169 MB (11% inode=98%); [02:16:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.73, 3.55 [02:18:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.53, 3.49 [02:20:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.60, 3.89, 3.63 [02:22:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.66, 3.58 [02:26:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.64, 3.56 [02:30:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.58, 3.57 [02:32:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.66, 3.59 [02:36:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.73, 3.63 [02:48:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.43, 3.47 [02:56:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.77, 3.44, 3.52 [03:02:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.88, 4.02, 3.71 [03:04:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.99, 3.96, 3.72 [03:08:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.32, 3.85, 3.70 [03:14:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.87, 3.77, 3.80 [03:16:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.52, 4.00, 3.87 [03:22:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.88, 3.86 [03:24:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 4.09, 3.95 [03:26:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.17, 3.40, 3.72 [03:34:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.22, 3.97, 3.85 [03:36:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.79, 3.80 [03:44:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.01, 2.62, 3.25 [04:52:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.32, 1.64, 1.15 [04:54:55] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.29, 1.48, 1.15 [05:08:43] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.24, 3.03 [05:10:38] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.51, 2.88, 2.91 [05:26:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.47, 2.72, 1.61 [05:30:55] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.41, 1.81, 1.47 [05:32:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.35, 3.10 [05:32:55] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.58, 1.35, 1.34 [05:34:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.46, 3.90, 3.32 [05:36:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.72, 3.33 [05:38:55] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.40, 1.72, 1.50 [05:40:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.18, 3.17, 3.20 [05:40:55] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.58, 1.28, 1.36 [06:02:40] SRE [06:02:52] ```MediaWiki internal error. [06:02:52] Original exception: [37fa258b67e35eed472d4e89] 2021-12-24 06:02:10: Fatal exception of type "Wikimedia\Rdbms\DBQueryDisconnectedError" [06:02:52] Exception caught inside exception handler. [06:02:53] Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information.``` [06:02:54] on `csydeswiki` [06:03:10] CosmicAlpha: can you run a trace? [06:26:47] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.47, 3.18 [06:28:43] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.36, 3.25, 3.12 [06:41:05] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26185 MB (5% inode=98%); [06:44:16] dmehus: CosmicAlpha is away for a bit [06:45:14] Agent, okay [06:45:29] he'll probably see it later; if not, remind him when he's back [07:05:05] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 31726 MB (7% inode=98%); [07:09:05] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 25799 MB (5% inode=98%); [07:15:05] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 34457 MB (7% inode=98%); [08:38:11] dmehus: are you still around? [09:20:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.15, 18.85, 14.24 [09:22:38] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.10, 17.00, 14.13 [10:22:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 6.06, 2.66, 1.58 [10:24:55] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.37, 1.93, 1.44 [10:26:55] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.62, 1.42, 1.30 [10:27:41] PROBLEM - www.burnout.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.22356736059200:error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:../ssl/record/rec_layer_s3.c:1544:SSL alert number 80 [10:33:03] PROBLEM - www.burnout.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.burnout.wiki could not be found [11:23:33] RECOVERY - www.burnout.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.burnout.wiki' will expire on Sat 24 Dec 2022 23:59:59 GMT +0000. [14:30:28] [ANNOUNCEMENT] Happy Holidays from the MirahezeBots team! [15:31:41] !log delete user account on phabricator to allow them to re-create it as they entered the wrong email [15:32:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:36:09] Could someone disable WikiJS Phab acc? [15:40:02] paladox: ^ [15:40:35] Why? [15:40:59] Request of owner [15:41:32] ok [15:42:01] !log disable an account on phabricator [15:42:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:57:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:02:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:04:04] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 3.72, 3.27 [16:05:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.57, 3.26 [16:13:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.39, 3.25, 3.31 [16:22:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:23:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.23, 3.20 [16:25:16] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.55, 3.31 [16:27:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.10, 3.36, 3.27 [16:32:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:33:43] PROBLEM - wiki.bikinibottomdiscord.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.bikinibottomdiscord.org' expires in 7 day(s) (Sat 01 Jan 2022 16:29:36 GMT +0000). [16:34:08] PROBLEM - wiki.beergeeks.co.il - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.beergeeks.co.il' expires in 7 day(s) (Sat 01 Jan 2022 16:32:07 GMT +0000). [16:36:34] PROBLEM - zh.internetpedia.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'zh.internetpedia.tk' expires in 7 day(s) (Sat 01 Jan 2022 16:35:15 GMT +0000). [16:36:56] PROBLEM - threedomwiki.pcast.site - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'threedomwiki.pcast.site' expires in 7 day(s) (Sat 01 Jan 2022 16:33:17 GMT +0000). [16:36:59] PROBLEM - puardian.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'puardian.xyz' expires in 7 day(s) (Sat 01 Jan 2022 16:31:11 GMT +0000). [16:37:22] PROBLEM - rct.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'rct.wiki' expires in 7 day(s) (Sat 01 Jan 2022 16:34:17 GMT +0000). [16:38:41] PROBLEM - ecole.science - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'ecole.science' expires in 7 day(s) (Sat 01 Jan 2022 16:36:15 GMT +0000). [16:52:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:57:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:18:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.28, 3.62, 3.11 [17:20:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.72, 3.30, 3.06 [17:21:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:26:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:30:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.59, 3.26 [17:32:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.82, 3.81, 3.37 [17:34:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.70, 3.38 [17:36:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:40:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.35, 2.98, 3.17 [17:41:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:44:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:46:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.49, 3.36 [17:48:25] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.78, 3.24, 3.28 [17:49:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:09:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.84, 3.39, 3.16 [18:13:44] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.30, 3.35, 3.20 [18:21:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.28, 3.97, 3.47 [18:23:23] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.65, 3.41 [18:24:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:29:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.68, 3.11, 3.31 [18:29:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:53:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.51, 3.36 [18:54:07] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.23, 6.42, 5.21 [18:55:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 3.83, 3.50 [18:56:03] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.07, 6.24, 5.29 [18:59:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:04:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:04:47] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.58, 3.88, 3.79 [19:09:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:20:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.68, 3.67, 3.56 [19:22:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.55, 3.52 [19:24:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.48, 3.12, 3.36 [19:29:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:30:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.85, 3.82, 3.58 [19:33:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:38:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.89, 3.17, 3.35 [19:38:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:46:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:56:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:04:16] !log [void@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=ysmwikiwiki (START) [20:04:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:49] !log [void@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=ysmwikiwiki (END - exit=256) [20:04:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:23:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:28:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:45:03] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48952 MB (10% inode=98%); [20:54:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:55:03] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 49930 MB (11% inode=98%); [21:04:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:55:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.45, 3.10 [21:57:37] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.83, 3.34, 3.11 [22:16:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:19:45] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.11, 6.41, 5.21 [22:21:45] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.90, 6.26, 5.31 [22:22:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.90, 3.44, 3.04 [22:24:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.68, 3.20, 3.00 [22:26:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:29:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JyYYU [22:29:51] [02miraheze/puppet] 07paladox 03d5712e9 - varnish: Remove vmtouch [22:29:52] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [22:29:54] [02puppet] 07paladox opened pull request 03#2172: varnish: Remove vmtouch - 13https://git.io/JyYYT [22:30:00] [02puppet] 07paladox edited pull request 03#2172: varnish: Remove vmtouch (part 1) - 13https://git.io/JyYYT [22:30:18] [02puppet] 07paladox edited pull request 03#2172: varnish: Remove vmtouch (part 1) - 13https://git.io/JyYYT [22:30:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/JyYYq [22:30:40] [02miraheze/puppet] 07paladox 038159ffa - Delete generateVmtouch.py [22:30:41] [02puppet] 07paladox synchronize pull request 03#2172: varnish: Remove vmtouch (part 1) - 13https://git.io/JyYYT [22:30:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/JyYYm [22:30:52] [02miraheze/puppet] 07paladox 030ab2e29 - Delete vmtouch.systemd.erb [22:30:53] [02puppet] 07paladox synchronize pull request 03#2172: varnish: Remove vmtouch (part 1) - 13https://git.io/JyYYT [22:31:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-2/±1] 13https://git.io/JyYYG [22:31:21] [02miraheze/puppet] 07paladox 03629b9d0 - varnish: Remove vmtouch (part 1) (#2172) [22:31:22] [02puppet] 07paladox closed pull request 03#2172: varnish: Remove vmtouch (part 1) - 13https://git.io/JyYYT [22:31:24] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [22:31:25] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [22:32:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyYYR [22:32:55] [02miraheze/puppet] 07paladox 031f2326f - Update init.pp [22:33:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyYYE [22:33:49] [02miraheze/puppet] 07paladox 03edf8d9b - Update init.pp [22:36:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.31, 3.04 [22:38:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.14, 2.84, 2.90 [23:00:03] !log [@mw11] starting deploy of {'l10nupdate': True} to ovlon [23:00:04] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:28] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 1823s [23:30:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:30:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:31:04] !log [@mw11] finished deploy of {'l10nupdate': True} to ovlon - SUCCESS in 1861s [23:31:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:35:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki