[00:00:12] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [00:01:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 3.52, 3.33 [00:01:41] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.96, 7.81, 7.81 [00:02:19] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.79, 21.51, 20.32 [00:02:19] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:02:21] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:02:40] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:03:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.00, 7.34, 7.49 [00:03:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.97, 3.53 [00:03:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [00:03:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.50, 7.53, 7.70 [00:04:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.13, 21.61, 20.54 [00:04:24] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.94, 7.38, 6.99 [00:04:38] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.358 second response time [00:05:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:05:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:05:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.54, 7.86, 7.80 [00:06:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.35, 19.75, 19.97 [00:06:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.76, 8.70, 7.52 [00:07:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.94, 3.64 [00:07:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.08, 8.06, 7.70 [00:08:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.93, 7.85, 7.35 [00:09:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.39, 7.79, 7.65 [00:09:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.73, 7.90, 7.89 [00:10:19] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.96, 23.89, 21.55 [00:12:18] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.47, 7.47, 7.26 [00:12:31] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:12:32] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:12:39] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:12:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:12:46] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:13:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.34, 7.81, 7.64 [00:13:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [00:13:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.42, 7.88, 7.84 [00:13:52] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:14:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.71, 7.62, 7.35 [00:14:28] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.384 second response time [00:14:29] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.345 second response time [00:14:36] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.018 second response time [00:14:46] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 0.376 second response time [00:14:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.98, 4.14, 3.74 [00:16:15] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.97, 8.43, 7.70 [00:16:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.26, 3.85, 3.69 [00:16:58] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:17:19] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:17:39] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:17:53] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:18:37] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:19:36] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:19:41] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.03, 7.92, 7.92 [00:19:43] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.88, 5.47, 4.37 [00:19:46] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:07] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:12] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.15, 7.73, 7.62 [00:20:27] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:38] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.812 second response time [00:20:40] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:58] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:21:03] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 0.202 second response time [00:21:26] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.010 second response time [00:21:41] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.32, 5.84, 4.64 [00:21:47] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.888 second response time [00:21:52] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.321 second response time [00:22:53] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:55] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.08, 7.27, 7.78 [00:23:20] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:23] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:58] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.700 second response time [00:24:19] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.496 second response time [00:24:41] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 5.571 second response time [00:24:47] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.409 second response time [00:25:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:25:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:25:41] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.369 second response time [00:26:08] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.15, 5.78, 6.72 [00:27:05] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.915 second response time [00:27:07] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.626 second response time [00:29:19] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.036 second response time [00:29:24] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.122 second response time [00:29:41] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 1.335 second response time [00:31:41] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.53, 7.23, 7.17 [00:32:02] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.91, 6.37, 6.57 [00:32:15] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:32:19] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:33:13] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:17] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:33:18] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:33:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:33:39] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.56, 7.43, 7.27 [00:34:00] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:34:01] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.05, 6.88, 6.70 [00:35:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.42, 8.26, 7.60 [00:35:44] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.663 second response time [00:36:02] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.991 second response time [00:36:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03nutcracker [+5/-0/±10] 13https://git.io/J93KF [00:36:22] [02miraheze/puppet] 07paladox 0357dda22 - Reintroduce nutcracker module [00:36:23] [02puppet] 07paladox created branch 03nutcracker - 13https://git.io/vbiAS [00:36:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.92, 4.04, 3.70 [00:36:33] [02puppet] 07paladox opened pull request 03#2284: Reintroduce nutcracker module - 13https://git.io/J93KN [00:37:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.88, 6.83, 7.98 [00:37:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.02, 6.94, 7.22 [00:37:17] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.899 second response time [00:37:20] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.071 second response time [00:37:22] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 1.972 second response time [00:37:25] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.52, 4.89, 5.20 [00:37:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:37:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:38:10] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J93KA [00:38:12] [02miraheze/mw-config] 07paladox 0328e3737 - Use nutcracker [00:38:13] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [00:38:15] [02mw-config] 07paladox opened pull request 03#4357: Use nutcracker - 13https://git.io/J93Kx [00:39:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.05, 6.79, 7.96 [00:39:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03nutcracker [+0/-0/±1] 13https://git.io/J93Kj [00:39:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.23, 6.92, 7.18 [00:39:15] [02miraheze/puppet] 07paladox 039165ae9 - Update nutcracker.pp [00:39:17] [02puppet] 07paladox synchronize pull request 03#2284: Reintroduce nutcracker module - 13https://git.io/J93KN [00:39:18] miraheze/mw-config - paladox the build passed. [00:39:24] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.55, 5.98, 5.55 [00:39:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.39, 8.00, 7.66 [00:39:44] [02puppet] 07paladox closed pull request 03#2284: Reintroduce nutcracker module - 13https://git.io/J93KN [00:39:45] [02miraheze/puppet] 07paladox deleted branch 03nutcracker [00:39:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+5/-0/±10] 13https://git.io/J936e [00:39:49] [02miraheze/puppet] 07paladox 034afa45e - Reintroduce nutcracker module (#2284) [00:39:50] [02puppet] 07paladox deleted branch 03nutcracker - 13https://git.io/vbiAS [00:40:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.96, 21.93, 23.90 [00:41:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.55, 7.59, 7.40 [00:41:22] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.78, 5.03, 5.25 [00:41:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J936I [00:41:33] [02miraheze/puppet] 07paladox 031b95e84 - Fix [00:41:41] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.60, 9.08, 8.11 [00:42:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J936q [00:42:59] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:43:00] [02miraheze/puppet] 07paladox 0385a9b8f - Fix [00:43:14] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:43:20] PROBLEM - mw13 Puppet on mw13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:43:29] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:43:32] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:43:50] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:44:17] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [00:44:19] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.83, 22.09, 23.45 [00:44:21] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:44:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:45:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.54, 7.11, 7.64 [00:45:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.87, 8.03, 7.85 [00:45:18] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.52, 4.75, 5.08 [00:45:26] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.463 second response time [00:45:51] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.32, 7.96, 7.62 [00:46:21] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [00:46:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.78, 3.76, 3.90 [00:46:54] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:46:59] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:47:13] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [00:47:17] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:47:17] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:47:19] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:47:50] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:48:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.09, 4.25, 4.06 [00:48:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J936C [00:48:39] [02miraheze/puppet] 07paladox 03581907e - Fix [00:49:00] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.679 second response time [00:49:19] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.468 second response time [00:49:20] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.120 second response time [00:49:43] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.98, 20.08, 18.93 [00:49:52] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:50:43] PROBLEM - mw112 Puppet on mw112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:51:36] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:51:38] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:51:40] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.74, 19.93, 19.01 [00:51:47] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.04, 8.69, 7.98 [00:51:51] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:52:03] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.77, 23.17, 23.57 [00:53:26] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:53:36] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:55:27] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.579 second response time [00:55:39] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.668 second response time [00:55:41] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.169 second response time [00:55:42] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:55:51] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:56:02] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 3.364 second response time [00:56:07] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.821 second response time [00:57:33] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.95, 20.51, 19.57 [00:57:40] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.111 second response time [00:57:48] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.338 second response time [00:58:14] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:58:22] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:58:24] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:58:35] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:58:43] RECOVERY - mw112 Puppet on mw112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:58:54] [02mw-config] 07paladox closed pull request 03#4357: Use nutcracker - 13https://git.io/J93Kx [00:58:56] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [00:58:57] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [00:58:59] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J936o [00:59:00] [02miraheze/mw-config] 07paladox 031bde1b9 - Use nutcracker (#4357) [00:59:39] !log [paladox@mwtask111] starting deploy of {'config': True} to scsvg [00:59:45] !log [paladox@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 6s [00:59:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:59:57] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:59:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:00:01] miraheze/mw-config - paladox the build passed. [01:00:13] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.809 second response time [01:00:19] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.649 second response time [01:00:21] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.318 second response time [01:00:34] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.336 second response time [01:00:40] !log [paladox@mw11] starting deploy of {'config': True} to all [01:00:50] !log [paladox@mw11] finished deploy of {'config': True} to all - SUCCESS in 10s [01:00:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:01:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:01:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:01:33] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.97, 20.31, 19.74 [01:01:49] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:02:18] !log [@test101] starting deploy of {'config': True} to skip [01:02:19] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:03:05] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J936X [01:03:06] [02miraheze/mw-config] 07paladox 03d7fb884 - Revert "Use nutcracker (#4357)" [01:03:09] !log [paladox@mw11] starting deploy of {'config': True} to all [01:03:14] !log [paladox@mw11] finished deploy of {'config': True} to all - SUCCESS in 4s [01:03:52] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:04:15] miraheze/mw-config - paladox the build passed. [01:04:57] !log [@test101] finished deploy of {'l10nupdate': True} to skip - FAIL: [256, 0, 0] in 7494s [01:05:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.91, 7.18, 7.99 [01:07:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.31, 6.56, 7.82 [01:07:39] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:10:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.26, 18.74, 20.36 [01:11:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.17, 7.47, 7.82 [01:11:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.80, 8.22, 8.13 [01:11:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:11:33] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.05, 21.09, 20.26 [01:11:37] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:11:58] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:12:07] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:13:00] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:14:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.71, 20.91, 20.94 [01:15:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.98, 7.52, 7.85 [01:15:49] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:17:00] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:17:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.11, 7.99, 7.99 [01:17:06] !log [@test101] starting deploy of {'config': True} to skip [01:17:07] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:17:16] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.005 second response time [01:17:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:17:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:17:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:17:48] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.692 second response time [01:18:13] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.299 second response time [01:18:15] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.529 second response time [01:19:00] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 1.665 second response time [01:19:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:22:06] !log [@mwtask111] starting deploy of {'config': True} to scsvg [01:22:12] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 6s [01:23:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:38] !log [@test3] starting deploy of {'config': True} to skip [01:24:39] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:24:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:25:04] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:25:09] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:25:22] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:25:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:25:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb [01:25:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:25:41] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:25:54] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:26:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:27:26] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 7.465 second response time [01:27:45] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.721 second response time [01:27:50] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.916 second response time [01:28:19] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.23, 21.13, 20.92 [01:29:02] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.399 second response time [01:29:10] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.958 second response time [01:29:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:29:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:30:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.63, 21.38, 21.01 [01:31:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.09, 6.90, 7.76 [01:31:16] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.24, 4.37, 4.01 [01:33:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.30, 6.91, 7.72 [01:33:16] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.76, 3.99, 3.91 [01:33:41] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J93iZ [01:33:43] [02miraheze/mw-config] 07paladox 030adcb72 - Use nutcracker on new mediawiki servers [01:33:44] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [01:33:46] [02mw-config] 07paladox opened pull request 03#4358: Use nutcracker on new mediawiki servers - 13https://git.io/J93in [01:33:54] [02mw-config] 07paladox closed pull request 03#4358: Use nutcracker on new mediawiki servers - 13https://git.io/J93in [01:33:56] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [01:33:57] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J93ic [01:33:59] [02miraheze/mw-config] 07paladox 03488894e - Use nutcracker on new mediawiki servers (#4358) [01:34:00] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [01:34:57] miraheze/mw-config - paladox the build passed. [01:34:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.93, 6.98, 7.95 [01:34:59] miraheze/mw-config - paladox the build passed. [01:35:00] !log [paladox@mwtask111] starting deploy of {'config': True} to scsvg [01:35:06] !log [paladox@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 6s [01:35:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:35:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:36:16] !log [paladox@test101] starting deploy of {'config': True} to skip [01:36:17] !log [paladox@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:36:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.25, 18.58, 20.08 [01:36:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:36:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:37:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.19, 6.43, 7.14 [01:37:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.19, 7.16, 7.58 [01:39:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.33, 6.50, 7.09 [01:39:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.91, 7.36, 7.63 [01:41:59] !log [@mw11] starting deploy of {'config': True} to ovlon [01:42:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:42:32] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 33s [01:43:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:44:17] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:45:24] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J93iP [01:45:26] [02miraheze/mw-config] 07paladox 0360ca059 - Set memcached timeout to 3s [01:45:27] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [01:45:29] [02mw-config] 07paladox opened pull request 03#4359: Set memcached timeout to 3s - 13https://git.io/J93iX [01:46:11] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:46:35] miraheze/mw-config - paladox the build passed. [01:47:32] !log [@test101] starting deploy of {'config': True} to skip [01:47:33] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:47:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:48:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:48:44] [02mw-config] 07paladox closed pull request 03#4359: Set memcached timeout to 3s - 13https://git.io/J93iX [01:48:46] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [01:48:47] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [01:49:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.98, 7.63, 7.32 [01:49:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.53, 7.58, 7.49 [01:50:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.67, 8.05, 7.57 [01:52:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.13, 20.55, 19.70 [01:52:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.74, 7.91, 7.58 [01:53:33] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.48, 19.46, 20.30 [01:54:42] !log [@test3] starting deploy of {'config': True} to skip [01:54:43] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:54:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.14, 8.30, 7.75 [01:55:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:55:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:56:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.41, 20.16, 19.76 [01:57:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.52, 7.81, 7.76 [01:57:33] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.01, 20.10, 20.41 [02:01:33] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.88, 19.61, 20.15 [02:02:21] [02puppet] 07paladox closed pull request 03#2282: site: split SCSVG and Ovlon MediaWiki servers - 13https://git.io/J9OzV [02:02:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J93Pl [02:02:24] [02miraheze/puppet] 07Universal-Omega 032c83127 - site: split SCSVG and Ovlon MediaWiki servers (#2282) [02:05:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.27, 7.94, 7.74 [02:07:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.09, 7.32, 7.52 [02:11:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.53, 7.53, 7.51 [02:15:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.30, 7.92, 7.76 [02:17:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.15, 8.35, 7.93 [02:17:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.99, 7.19, 7.96 [02:21:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.33, 8.21, 8.18 [02:21:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [02:22:23] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.60, 20.46, 19.41 [02:24:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.32, 19.89, 19.31 [02:27:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:33:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.50, 7.27, 7.94 [02:39:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.15, 7.88, 7.96 [02:43:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb [02:43:56] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:44:08] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:45:53] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.150 second response time [02:46:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.02, 21.32, 20.01 [02:47:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/J93XM [02:47:02] [02miraheze/mw-config] 07Universal-Omega 03e690bf5 - Install Patroller [02:47:03] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [02:47:07] [02mw-config] 07Universal-Omega opened pull request 03#4360: Install Patroller - 13https://git.io/J93XD [02:47:20] [02mw-config] 07Universal-Omega edited pull request 03#4360: Add Patroller to ManageWikiExtensions - 13https://git.io/J93XD [02:47:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:48:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:48:18] miraheze/mw-config - Universal-Omega the build passed. [02:48:19] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.07, 21.94, 20.36 [02:49:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.23, 7.24, 7.83 [02:50:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.75, 21.65, 20.44 [02:51:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.08, 7.96, 8.05 [02:52:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/J93X7 [02:52:15] [02miraheze/mw-config] 07Universal-Omega 03a800b1f - Add StandardDialogs to ManageWikiExtensions [02:52:17] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-5 - 13https://git.io/vbvb3 [02:52:21] [02mw-config] 07Universal-Omega opened pull request 03#4361: Add StandardDialogs to ManageWikiExtensions - 13https://git.io/J93X5 [02:53:21] miraheze/mw-config - Universal-Omega the build passed. [02:53:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.26, 7.29, 7.77 [02:53:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.30, 7.17, 7.78 [02:54:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.10, 19.35, 19.81 [02:55:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.19, 8.01, 7.98 [02:55:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.63, 7.96, 8.01 [02:55:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/J93Xj [02:55:58] [02miraheze/mw-config] 07Universal-Omega 0331c8447 - Update ManageWikiExtensions.php [02:55:59] [02mw-config] 07Universal-Omega synchronize pull request 03#4360: Add Patroller to ManageWikiExtensions - 13https://git.io/J93XD [02:57:02] miraheze/mw-config - Universal-Omega the build passed. [02:57:11] [02mw-config] 07Universal-Omega closed pull request 03#4361: Add StandardDialogs to ManageWikiExtensions - 13https://git.io/J93X5 [02:57:13] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J931v [02:57:14] [02miraheze/mw-config] 07Universal-Omega 038d3250d - Add StandardDialogs to ManageWikiExtensions (#4361) [02:57:16] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 [02:57:17] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 - 13https://git.io/vbvb3 [02:58:17] miraheze/mw-config - Universal-Omega the build passed. [02:58:36] [02mw-config] 07Universal-Omega closed pull request 03#4360: Add Patroller to ManageWikiExtensions - 13https://git.io/J93XD [02:58:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J931J [02:58:39] [02miraheze/mw-config] 07Universal-Omega 0310bf581 - Add Patroller to ManageWikiExtensions (#4360) [02:58:41] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 [02:58:42] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [02:59:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.65, 7.12, 7.72 [02:59:42] miraheze/mw-config - Universal-Omega the build passed. [02:59:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/J931T [02:59:48] [02miraheze/mw-config] 07Universal-Omega 0321df9fc - Update LocalSettings.php [02:59:50] [02mw-config] 07Universal-Omega synchronize pull request 03#4348: Add Bastion, ElasticSearch, and Graylog to `$wgIncidentReportingServices` - 13https://git.io/JS74g [02:59:58] [02mw-config] 07Universal-Omega edited pull request 03#4348: Add Bastion and ElasticSearch to `$wgIncidentReportingServices` - 13https://git.io/JS74g [03:00:07] [02mw-config] 07Universal-Omega edited pull request 03#4348: Add Bastion and ElasticSearch to `$wgIncidentReportingServices` - 13https://git.io/JS74g [03:00:50] miraheze/mw-config - Universal-Omega the build passed. [03:01:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 3.63, 6.29, 7.43 [03:01:16] [02mw-config] 07Universal-Omega closed pull request 03#4348: Add Bastion and ElasticSearch to `$wgIncidentReportingServices` - 13https://git.io/JS74g [03:01:18] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [03:01:19] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [03:01:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J931t [03:01:22] [02miraheze/mw-config] 07Universal-Omega 036ba3347 - Add Bastion and ElasticSearch to `$wgIncidentReportingServices` (#4348) [03:01:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.67, 7.03, 7.65 [03:02:17] miraheze/mw-config - Universal-Omega the build passed. [03:03:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.55, 6.98, 7.53 [03:05:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.52, 6.77, 7.41 [03:07:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.04, 7.77, 7.71 [03:07:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.95, 7.18, 7.31 [03:09:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.26, 7.92, 7.54 [03:11:31] !log [@mw11] starting deploy of {'config': True} to ovlon [03:11:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:11:47] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 16s [03:11:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:16:45] !log [universalomega@mw11] starting deploy of {'extensionlist': True} to all [03:16:57] !log [universalomega@test3] starting deploy of {'extensionlist': True} to skip [03:16:59] !log [universalomega@mw11] finished deploy of {'extensionlist': True} to all - SUCCESS in 14s [03:17:00] !log [universalomega@test3] finished deploy of {'extensionlist': True} to skip - SUCCESS in 1s [03:17:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:17:10] !log [universalomega@test101] starting deploy of {'extensionlist': True} to skip [03:17:11] !log [universalomega@test101] finished deploy of {'extensionlist': True} to skip - SUCCESS in 0s [03:17:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:17:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:17:24] !log [universalomega@mwtask111] starting deploy of {'extensionlist': True} to scsvg [03:17:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:17:31] !log [universalomega@mwtask111] finished deploy of {'extensionlist': True} to scsvg - SUCCESS in 7s [03:17:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:17:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:17:48] !log [@test101] starting deploy of {'config': True} to skip [03:17:49] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [03:17:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:18:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:18:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:18:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:21:40] !log [@mwtask111] starting deploy of {'config': True} to scsvg [03:21:45] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 5s [03:21:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:22:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:23:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4360-Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/J9312 [03:23:04] [02miraheze/mw-config] 07Universal-Omega 0394ff4aa - Revert "Add Patroller to ManageWikiExtensions (#4360)" [03:23:06] [02mw-config] 07Universal-Omega created branch 03revert-4360-Universal-Omega-patch-4 - 13https://git.io/vbvb3 [03:23:42] [02mw-config] 07Universal-Omega opened pull request 03#4362: Revert "Add Patroller to ManageWikiExtensions" - 13https://git.io/J931a [03:23:46] [02mw-config] 07Universal-Omega closed pull request 03#4362: Revert "Add Patroller to ManageWikiExtensions" - 13https://git.io/J931a [03:23:48] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-4360-Universal-Omega-patch-4 [03:23:49] [02mw-config] 07Universal-Omega deleted branch 03revert-4360-Universal-Omega-patch-4 - 13https://git.io/vbvb3 [03:23:51] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J931V [03:23:52] [02miraheze/mw-config] 07Universal-Omega 039c683c3 - Revert "Add Patroller to ManageWikiExtensions (#4360)" (#4362) [03:24:25] !log [@mwtask111] finished deploy of {'l10nupdate': True} to scsvg - FAIL: [256, 0, 0, 0, 0, 0, 0, 0, 0] in 15864s [03:24:29] !log [@test3] starting deploy of {'config': True} to skip [03:24:30] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [03:24:40] miraheze/mw-config - Universal-Omega the build passed. [03:24:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J931K [03:24:42] [02miraheze/mw-config] 07Universal-Omega 03d7742a4 - Remove Patroller from extension-list [03:24:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:25:07] miraheze/mw-config - Universal-Omega the build passed. [03:25:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:25:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.97, 7.53, 7.88 [03:25:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:25:53] miraheze/mw-config - Universal-Omega the build passed. [03:27:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.36, 7.20, 7.91 [03:31:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.30, 7.46, 7.97 [03:32:32] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/J93MU [03:32:33] [02miraheze/mw-config] 07Universal-Omega 03a5435b4 - Only.run LocalisationUpdate for core [03:32:35] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [03:33:01] [02mw-config] 07Universal-Omega opened pull request 03#4363: Only.run LocalisationUpdate for core - 13https://git.io/J93ML [03:33:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.93, 7.75, 7.99 [03:33:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.93, 8.43, 8.10 [03:34:04] miraheze/mw-config - Universal-Omega the build passed. [03:35:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.36, 7.20, 7.27 [03:35:40] [02mw-config] 07Universal-Omega edited pull request 03#4363: Only.run LocalisationUpdate for MediaWiki core - 13https://git.io/J93ML [03:36:12] [02mw-config] 07Universal-Omega edited pull request 03#4363: Only run LocalisationUpdate for MediaWiki core - 13https://git.io/J93ML [03:36:29] [02mw-config] 07Universal-Omega closed pull request 03#4363: Only run LocalisationUpdate for MediaWiki core - 13https://git.io/J93ML [03:36:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J93ME [03:36:32] [02miraheze/mw-config] 07Universal-Omega 03deadf60 - Only run LocalisationUpdate for MediaWiki core (#4363) [03:36:33] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [03:36:35] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [03:37:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.71, 7.79, 7.97 [03:37:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.05, 7.05, 7.23 [03:37:37] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to scsvg [03:37:37] miraheze/mw-config - Universal-Omega the build passed. [03:37:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:37:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:37:43] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to scsvg - SUCCESS in 6s [03:37:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:37:57] !log [universalomega@mwtask111] starting deploy of {'l10nupdate': True} to scsvg [03:38:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:39:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J93MK [03:39:36] [02miraheze/mw-config] 07Universal-Omega 0319ad034 - SCSVG only [03:39:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.90, 6.92, 7.67 [03:40:42] miraheze/mw-config - Universal-Omega the build passed. [03:41:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.77, 6.50, 7.76 [03:41:10] !log [@mw11] starting deploy of {'config': True} to ovlon [03:41:19] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 9s [03:41:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:41:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.18, 7.70, 7.35 [03:41:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:41:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.96, 7.96, 7.97 [03:43:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.71, 7.14, 7.18 [03:43:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.91, 7.30, 7.71 [03:44:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:45:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.91, 7.84, 7.44 [03:47:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.76, 8.68, 8.14 [03:47:17] !log [@test101] starting deploy of {'config': True} to skip [03:47:18] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [03:47:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:47:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:49:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.06, 7.06, 7.29 [03:51:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.85, 7.58, 7.53 [03:52:00] !log [@mwtask111] starting deploy of {'config': True} to scsvg [03:52:06] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 6s [03:52:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:52:33] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:52:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:53:15] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.51, 5.92, 6.80 [03:53:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.52, 6.30, 7.07 [03:54:18] !log [@test3] starting deploy of {'config': True} to skip [03:54:19] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [03:54:29] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.334 second response time [03:55:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:55:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.94, 7.62, 7.38 [03:55:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:56:30] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.28, 20.39, 19.57 [03:56:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.64, 3.97 [03:57:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.13, 6.08, 6.83 [03:57:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.60, 5.43, 6.61 [03:59:24] !log [universalomega@mwtask111] finished deploy of {'l10nupdate': True} to scsvg - SUCCESS in 1287s [03:59:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:01:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.32, 7.41, 7.21 [04:01:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.04, 7.48, 7.12 [04:02:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.13, 4.07, 3.99 [04:04:02] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.86, 7.42, 6.94 [04:04:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.16, 19.78, 19.76 [04:05:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.24, 7.18, 7.16 [04:05:59] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.26, 6.71, 6.75 [04:06:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.68, 3.88 [04:07:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.30, 7.36, 8.00 [04:07:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.41, 6.82, 7.20 [04:07:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.93, 5.98, 6.72 [04:09:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:10:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.60, 4.18, 4.05 [04:11:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.48, 7.61, 7.35 [04:11:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.73, 8.02, 7.32 [04:11:50] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.80, 7.99, 7.15 [04:12:43] 8:59 PM  !log [universalomega@mwtask111] finished deploy of {'l10nupdate': True} to scsvg - SUCCESS in 1287s [04:13:11] That's alot better than 15,000 seconds (4 hours!) [04:13:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.27, 7.07, 7.16 [04:13:38] I disabled the extension and skins updates. [04:13:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.09, 7.39, 7.16 [04:14:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.80, 3.68, 3.89 [04:15:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.28, 7.43, 7.28 [04:15:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:15:43] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.98, 7.95, 7.34 [04:16:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.96, 4.03, 3.98 [04:17:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.55, 5.66, 6.71 [04:17:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.65, 6.23, 8.00 [04:17:39] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.63, 8.93, 7.77 [04:17:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.23, 8.13, 7.50 [04:18:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.80, 3.90, 3.94 [04:19:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.52, 7.70, 8.31 [04:19:53] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [04:20:05] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:20:07] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:20:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.32, 19.77, 18.95 [04:21:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.00, 8.87, 7.79 [04:21:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [04:22:03] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.009 second response time [04:22:05] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.275 second response time [04:24:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.77, 1.46, 0.99 [04:24:30] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.24, 19.25, 18.93 [04:24:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.14, 4.59, 4.19 [04:25:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:25:34] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:26:14] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.60, 1.14, 0.93 [04:27:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.37, 7.54, 7.67 [04:27:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.02, 7.27, 7.65 [04:27:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.73, 7.88, 7.92 [04:29:41] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.35, 8.24, 8.05 [04:31:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 3.67, 6.58, 7.47 [04:32:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.02, 6.96, 8.00 [04:33:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.84, 6.28, 7.56 [04:35:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.80, 7.11, 7.70 [04:36:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.20, 8.21, 8.20 [04:37:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.26, 8.71, 7.83 [04:37:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.24, 5.19, 6.44 [04:37:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.79, 5.47, 6.75 [04:38:57] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.20, 7.45, 7.90 [04:40:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.41, 8.26, 8.14 [04:41:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.95, 6.96, 7.79 [04:42:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.89, 3.69, 3.99 [04:43:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.64, 7.56, 7.66 [04:43:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.39, 6.43, 6.53 [04:45:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.09, 5.97, 6.35 [04:47:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.78, 9.30, 8.28 [04:49:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.80, 7.90, 7.68 [04:49:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.94, 6.87, 6.60 [04:49:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.27, 7.62, 6.96 [04:50:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.31, 3.88, 3.82 [04:52:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.44, 20.10, 18.97 [04:53:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.80, 7.86, 7.21 [04:54:43] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.88, 19.84, 19.00 [04:55:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.32, 6.13, 6.45 [04:56:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.06, 3.88, 3.93 [04:57:41] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.06, 8.21, 7.46 [04:58:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 4.01, 3.96 [04:59:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.43, 8.35, 7.27 [05:00:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.83, 3.91 [05:02:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 4.07, 3.99 [05:05:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.83, 7.15, 7.19 [05:05:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.19, 7.54, 7.64 [05:06:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.69, 3.81, 3.94 [05:07:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.02, 7.49, 7.95 [05:07:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.73, 8.15, 7.86 [05:08:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.88, 4.28, 4.09 [05:09:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.69, 6.89, 7.97 [05:09:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.88, 8.35, 8.18 [05:09:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.06, 7.74, 7.37 [05:10:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.94, 3.99 [05:13:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.80, 7.24, 7.82 [05:13:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.94, 7.82, 7.86 [05:14:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.50, 4.19, 4.05 [05:15:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.28, 6.93, 7.65 [05:15:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.53, 8.37, 8.07 [05:19:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.66, 8.96, 8.28 [05:19:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.65, 7.79, 7.68 [05:22:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.09, 19.95, 18.89 [05:23:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.27, 7.81, 7.63 [05:25:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.41, 6.74, 7.28 [05:26:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.81, 19.83, 19.04 [05:27:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.77, 7.57, 7.93 [05:27:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.75, 7.45, 7.90 [05:29:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.27, 5.72, 6.72 [05:29:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.65, 7.76, 7.95 [05:33:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.24, 6.85, 7.60 [05:34:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.77, 7.21, 7.98 [05:35:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.14, 7.12, 7.92 [05:35:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.25, 7.95, 7.89 [05:36:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.92, 7.74, 8.04 [05:37:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.33, 7.65, 7.99 [05:37:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.70, 7.07, 7.02 [05:37:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.01, 7.48, 7.72 [05:41:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.77, 5.65, 6.73 [05:41:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.35, 7.92, 7.80 [05:41:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [05:43:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.68, 7.18, 7.54 [05:43:52] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:44:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.70, 3.99 [05:45:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.59, 7.69, 7.22 [05:45:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.09, 8.77, 8.09 [05:46:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 4.00, 4.07 [05:47:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.01, 8.01, 7.43 [05:47:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.95, 7.59, 7.25 [05:49:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.07, 7.34, 7.17 [05:50:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:51:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.86, 7.88, 7.41 [05:53:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.62, 7.68, 7.52 [05:54:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:55:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.92, 8.35, 7.68 [05:55:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.10, 7.62, 7.87 [05:57:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.04, 6.91, 7.82 [05:57:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.67, 7.80, 7.56 [05:59:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.11, 7.77, 7.62 [06:00:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.63, 7.02, 7.80 [06:01:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.99, 7.44, 7.55 [06:03:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.49, 7.66, 7.76 [06:03:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.57, 7.42, 7.61 [06:04:12] [02puppet] 07Universal-Omega opened pull request 03#2285: MediaWiki: add custom 502 error page for nginx - 13https://git.io/J93Qs [06:04:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:05:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.68, 7.59, 7.75 [06:05:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.09, 8.45, 7.83 [06:07:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.34, 8.22, 7.76 [06:07:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:10:29] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://git.io/J93Qc [06:10:31] [02miraheze/ErrorPages] 07Universal-Omega 03e477cfa - Add 502 error page [06:10:32] [02ErrorPages] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/JuBZG [06:10:34] [02ErrorPages] 07Universal-Omega opened pull request 03#3: Add 502 error page - 13https://git.io/J93QC [06:10:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.17, 7.81, 7.69 [06:11:33] miraheze/ErrorPages - Universal-Omega the build passed. [06:11:35] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J93Q4 [06:11:36] [02miraheze/ErrorPages] 07Universal-Omega 03e05da77 - Update en.json [06:11:38] [02ErrorPages] 07Universal-Omega synchronize pull request 03#3: Add 502 error page - 13https://git.io/J93QC [06:12:35] miraheze/ErrorPages - Universal-Omega the build passed. [06:12:49] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J93QB [06:12:51] [02miraheze/ErrorPages] 07Universal-Omega 0336adbd5 - Update qqq.json [06:12:52] [02ErrorPages] 07Universal-Omega synchronize pull request 03#3: Add 502 error page - 13https://git.io/J93QC [06:12:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.91, 7.85, 7.74 [06:13:48] miraheze/ErrorPages - Universal-Omega the build passed. [06:14:22] [02ErrorPages] 07Universal-Omega closed pull request 03#3: Add 502 error page - 13https://git.io/J93QC [06:14:24] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/J93Q0 [06:14:25] [02miraheze/ErrorPages] 07Universal-Omega 03274cead - Add 502 error page (#3) [06:14:27] [02ErrorPages] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/JuBZG [06:14:28] [02miraheze/ErrorPages] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [06:15:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.64, 6.72, 7.71 [06:15:21] miraheze/ErrorPages - Universal-Omega the build passed. [06:15:26] [02puppet] 07Universal-Omega edited pull request 03#2285: MediaWiki: add custom 502 error page for nginx - 13https://git.io/J93Qs [06:15:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [06:15:49] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:16:19] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:17:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.93, 5.84, 6.65 [06:17:09] [02puppet] 07Universal-Omega edited pull request 03#2285: MediaWiki: add custom 502 error page for nginx - 13https://git.io/J93Qs [06:17:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:17:40] [02puppet] 07Universal-Omega edited pull request 03#2285: MediaWiki: add custom 502 error page for nginx - 13https://git.io/J93Qs [06:17:44] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 1.126 second response time [06:18:16] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.322 second response time [06:20:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.07, 7.54, 7.45 [06:21:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.89, 7.04, 7.00 [06:22:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.68, 6.61, 7.14 [06:25:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.31, 7.45, 7.10 [06:25:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.99, 7.00, 7.28 [06:26:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.72, 7.23, 7.20 [06:30:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.06, 7.18, 7.26 [06:33:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 3.32, 6.80, 7.36 [06:33:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.50, 7.48, 7.67 [06:35:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.20, 7.77, 7.75 [06:37:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:38:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.06, 6.12, 6.72 [06:40:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:41:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.00, 7.52, 7.30 [06:41:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.78, 7.86, 7.90 [06:42:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.42, 7.48, 7.12 [06:43:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.62, 7.61, 7.37 [06:43:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.54, 8.73, 8.21 [06:44:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.25, 7.47, 7.16 [06:45:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.03, 7.78, 7.93 [06:45:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:49:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.51, 7.99, 7.59 [06:49:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:50:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.15, 8.06, 7.40 [06:51:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.85, 8.38, 8.04 [06:52:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.95, 7.78, 7.39 [06:54:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.56, 7.84, 7.44 [06:55:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.55, 7.51, 7.63 [06:56:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.27, 7.82, 7.50 [06:57:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.38, 7.68, 7.91 [06:59:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.99, 7.04, 7.87 [06:59:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:02:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.09, 7.37, 7.32 [07:05:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.43, 7.48, 7.33 [07:06:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.34, 7.88, 7.63 [07:07:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.88, 7.34, 7.29 [07:07:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.85, 7.49, 7.53 [07:07:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.67, 6.92, 7.96 [07:09:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.49, 6.91, 7.33 [07:09:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.02, 7.19, 7.98 [07:11:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.51, 7.40, 7.84 [07:11:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.45, 8.25, 7.76 [07:11:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.10, 7.73, 8.10 [07:13:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.04, 6.57, 7.50 [07:13:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.78, 7.26, 7.46 [07:13:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.48, 7.07, 7.82 [07:14:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.26, 7.42, 7.33 [07:15:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.76, 7.28, 7.19 [07:15:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.76, 7.98, 7.52 [07:15:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.74, 7.43, 7.70 [07:15:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.28, 8.28, 7.82 [07:15:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:16:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.52, 6.39, 6.97 [07:17:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.44, 6.20, 6.81 [07:17:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.60, 7.12, 7.27 [07:17:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.17, 7.09, 7.55 [07:17:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.25, 7.17, 7.48 [07:17:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.11, 7.85, 7.99 [07:18:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.28, 6.89, 7.07 [07:19:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.93, 6.79, 6.92 [07:19:08] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.58, 7.78, 7.47 [07:19:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.65, 7.99, 7.82 [07:20:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.33, 7.30, 7.22 [07:21:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.97, 6.52, 6.82 [07:21:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.52, 7.88, 7.56 [07:21:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.31, 7.62, 7.57 [07:23:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.41, 6.28, 6.68 [07:23:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.45, 6.92, 7.31 [07:25:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.50, 7.63, 7.85 [07:25:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.14, 7.11, 7.75 [07:26:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 13.11, 9.37, 7.98 [07:29:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.29, 7.53, 7.14 [07:29:41] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.58, 6.88, 7.42 [07:30:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.38, 7.63, 7.64 [07:31:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.84, 5.47, 6.50 [07:31:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.02, 7.30, 7.22 [07:33:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.58, 7.86, 7.29 [07:33:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.02, 7.11, 7.16 [07:33:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.88, 7.65, 7.64 [07:35:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.40, 7.36, 7.17 [07:35:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.12, 7.21, 7.32 [07:37:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.05, 6.98, 7.21 [07:37:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.94, 7.84, 7.64 [07:38:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.06, 7.23, 7.40 [07:39:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.39, 7.52, 7.57 [07:40:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:40:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.46, 6.80, 7.21 [07:41:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.53, 8.22, 7.81 [07:43:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.59, 6.18, 6.75 [07:43:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.09, 7.12, 7.15 [07:43:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.51, 6.20, 6.76 [07:46:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:46:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.96, 7.52, 7.29 [07:47:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.84, 7.92, 7.26 [07:47:08] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.23, 8.30, 7.16 [07:47:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.17, 7.99, 7.35 [07:49:14] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:49:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.31, 7.52, 7.27 [07:49:59] [02puppet] 07Reception123 closed pull request 03#2285: MediaWiki: add custom 502 error page for nginx - 13https://git.io/J93Qs [07:50:00] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J93Fn [07:50:02] [02miraheze/puppet] 07Universal-Omega 0327b723a - MediaWiki: add custom 502 error page for nginx (#2285) [07:50:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.45, 19.62, 18.22 [07:51:09] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.550 second response time [07:52:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.81, 18.82, 18.09 [07:55:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.42, 7.75, 7.47 [07:55:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.87, 8.17, 7.54 [07:58:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.08, 7.65, 7.69 [08:01:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.85, 7.14, 7.76 [08:04:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.70, 3.71, 3.99 [08:04:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.88, 5.73, 6.73 [08:05:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.89, 6.00, 6.72 [08:05:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.98, 7.39, 7.71 [08:08:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.60, 3.85 [08:10:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.55, 3.81 [08:11:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.41, 6.68, 7.77 [08:13:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.77, 7.22, 6.99 [08:13:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.74, 5.86, 6.80 [08:13:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.67, 6.45, 7.72 [08:17:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.42, 5.74, 6.66 [08:19:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.67, 8.11, 7.40 [08:19:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.21, 5.19, 6.78 [08:21:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.90, 7.53, 7.20 [08:21:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.50, 7.95, 7.59 [08:21:52] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.77, 8.33, 7.28 [08:23:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.41, 7.43, 7.43 [08:23:41] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.07, 6.22, 6.86 [08:23:49] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.78, 7.81, 7.22 [08:24:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.91, 4.12, 3.80 [08:25:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.79, 5.79, 6.63 [08:25:46] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.75, 8.51, 7.57 [08:26:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.88, 3.75 [08:27:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.99, 7.78, 7.54 [08:29:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.87, 7.38, 7.42 [08:29:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.77, 7.28, 7.39 [08:31:39] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.93, 7.47, 7.54 [08:31:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:35:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:36:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.22, 2.85, 3.29 [08:37:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.05, 6.13, 6.74 [08:37:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.47, 7.02, 6.55 [08:39:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.50, 5.63, 6.10 [08:40:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:43:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.63, 7.54, 7.24 [08:43:24] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.20, 8.38, 7.59 [08:45:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.31, 7.60, 7.31 [08:45:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.88, 7.41, 6.78 [08:45:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.57, 7.58, 6.92 [08:45:48] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.16, 7.18, 6.93 [08:46:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:47:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 7.76, 8.08, 7.55 [08:47:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.63, 6.71, 6.60 [08:49:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.79, 8.37, 7.44 [08:49:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.36, 7.42, 6.97 [08:52:24] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.21, 20.63, 18.79 [08:52:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.83, 3.54 [08:53:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.76, 8.83, 7.38 [08:53:44] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.299 second response time [08:53:49] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:53:57] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:54:01] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [08:54:02] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:54:12] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:54:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.71, 19.37, 18.55 [08:54:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [08:54:20] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:54:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.84, 3.58 [08:55:20] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw8 [08:55:26] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw8 mw13 [08:55:43] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.930 second response time [08:55:46] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 2.552 second response time [08:56:02] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.676 second response time [08:56:05] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 6.103 second response time [08:56:11] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 0.173 second response time [08:56:25] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.666 second response time [08:57:20] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.37, 7.77, 7.79 [08:57:20] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 18 backends are healthy [08:57:25] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [08:57:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:58:11] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:58:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.63, 3.37, 3.46 [08:59:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.22, 6.72, 7.04 [08:59:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.29, 7.06, 7.48 [09:00:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 3.57, 3.50 [09:01:10] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.22, 8.86, 8.09 [09:01:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.87, 8.64, 7.69 [09:01:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:01:41] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 13.65, 9.50, 8.30 [09:02:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.94, 3.63, 3.53 [09:05:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.82, 7.34, 7.72 [09:05:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 3.40, 5.91, 7.75 [09:05:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 3.41, 6.45, 7.08 [09:05:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.08, 7.76, 7.94 [09:06:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.61, 3.05, 3.32 [09:06:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 3.79, 6.43, 7.92 [09:07:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 3.99, 6.21, 7.77 [09:07:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.99, 5.66, 6.71 [09:10:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.57, 3.51, 3.45 [09:12:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.99, 3.06, 3.30 [09:12:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.22, 7.14, 7.65 [09:13:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.41, 7.25, 7.35 [09:13:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.41, 5.96, 6.99 [09:13:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.14, 7.19, 6.87 [09:13:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.60, 7.20, 7.26 [09:14:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.67, 6.86, 7.49 [09:15:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.25, 6.73, 7.15 [09:15:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.12, 6.22, 6.96 [09:16:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:17:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.22, 5.83, 6.67 [09:17:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.79, 5.64, 6.70 [09:17:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.96, 7.41, 7.30 [09:17:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.87, 7.13, 7.21 [09:18:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.37, 3.73, 3.56 [09:19:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.21, 7.29, 7.27 [09:20:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.93, 4.16, 3.74 [09:21:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.64, 8.73, 7.82 [09:21:19] PROBLEM - wikien.wildterra2.com - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdn [09:21:19] name = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1089, in query return self.resolve(qname, rdtype, rdclass, tcp, source, File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(sta [09:21:19] fetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406275510787964 seconds [09:21:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.45, 7.94, 7.45 [09:21:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:22:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.47, 3.87, 3.69 [09:23:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.49, 6.09, 6.78 [09:24:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.88, 3.70 [09:24:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.96, 7.25, 7.21 [09:25:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.73, 6.96, 7.35 [09:25:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.97, 6.06, 6.79 [09:25:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:26:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.91, 3.59, 3.62 [09:26:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.64, 6.92, 7.09 [09:30:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:30:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.48, 6.08, 6.77 [09:31:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.88, 5.78, 6.71 [09:32:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.82, 4.18, 3.81 [09:34:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.88, 3.74 [09:34:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.62, 6.96, 6.92 [09:35:00] RECOVERY - wikien.wildterra2.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/l [09:35:00] ios/plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - wikien.wildterra2.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [09:35:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:35:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.77, 6.53, 6.26 [09:36:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 4.02, 3.81 [09:36:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.68, 6.76, 6.85 [09:37:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.23, 6.00, 6.11 [09:38:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.80, 5.69, 6.45 [09:40:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.88, 3.75, 3.77 [09:40:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:43:48] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.34, 6.68, 6.32 [09:43:50] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.71, 8.11, 6.95 [09:44:03] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.37, 8.14, 7.07 [09:44:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 4.25, 3.96 [09:44:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.48, 6.56, 6.56 [09:45:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.27, 8.58, 7.08 [09:45:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.27, 6.01, 6.11 [09:45:45] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.27, 6.65, 6.56 [09:46:20] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.52, 6.81, 6.52 [09:47:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.21, 7.52, 6.87 [09:48:19] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.96, 7.47, 6.78 [09:48:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.72, 3.66, 3.80 [09:48:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.05, 7.22, 6.84 [09:49:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.77, 6.56, 6.59 [09:50:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.71, 7.03, 6.70 [09:50:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.81, 7.15, 6.87 [09:51:31] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.91, 7.63, 6.72 [09:51:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.56, 8.84, 7.36 [09:51:48] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.69, 7.23, 7.22 [09:52:15] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.75, 6.71, 6.61 [09:52:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.90, 19.28, 17.41 [09:52:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.62, 8.25, 7.31 [09:53:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:54:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.79, 18.23, 17.25 [09:54:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.51, 3.67 [09:54:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.96, 7.88, 7.28 [09:55:21] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.70, 6.80, 6.65 [09:55:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.86, 6.92, 6.72 [09:55:41] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.11, 8.41, 7.62 [09:56:11] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.20, 7.57, 6.95 [09:56:13] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26692 MB (5% inode=97%); [09:56:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.54, 3.67 [09:57:16] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.89, 7.48, 7.31 [09:58:10] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.54, 7.51, 7.00 [09:58:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.85, 3.78 [09:59:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.31, 6.50, 6.63 [10:00:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 3.68, 3.73 [10:02:07] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.79, 6.49, 6.73 [10:02:13] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 27674 MB (6% inode=97%); [10:02:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.90, 6.34, 6.78 [10:03:08] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.36, 7.35, 7.31 [10:03:26] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.66, 7.45, 7.72 [10:04:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 4.01, 3.85 [10:05:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.54, 7.57, 7.44 [10:06:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.70, 3.61, 3.73 [10:08:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:09:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.92, 6.49, 6.43 [10:11:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.58, 5.36, 6.59 [10:12:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.54, 3.36, 3.52 [10:13:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.84, 7.73, 7.42 [10:13:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.92, 8.54, 7.20 [10:13:56] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 12.73, 8.75, 7.27 [10:14:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.57, 3.58 [10:14:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.70, 8.26, 7.19 [10:15:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.57, 7.67, 7.44 [10:15:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.72, 6.83, 6.97 [10:17:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.02, 8.27, 7.69 [10:17:29] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.35, 7.04, 6.28 [10:17:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:17:53] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.63, 7.57, 7.12 [10:19:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.08, 7.88, 7.63 [10:19:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.13, 7.99, 7.36 [10:19:24] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.26, 7.39, 6.46 [10:20:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.32, 2.85, 3.27 [10:21:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.17, 8.48, 7.89 [10:21:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.48, 7.34, 7.20 [10:22:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.75, 7.32, 7.33 [10:23:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.00, 7.35, 7.54 [10:23:14] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.49, 6.98, 6.54 [10:23:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.05, 7.22, 7.35 [10:25:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.75, 7.61, 6.82 [10:25:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.63, 7.86, 7.43 [10:25:47] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.97, 7.76, 7.23 [10:26:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.37, 8.53, 7.81 [10:27:04] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.15, 7.03, 6.70 [10:27:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.99, 8.00, 7.73 [10:27:46] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.83, 7.28, 7.11 [10:29:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.66, 7.93, 7.74 [10:29:41] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.01, 7.74, 7.51 [10:29:44] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.31, 6.28, 6.77 [10:30:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.13, 7.59, 7.68 [10:31:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.63, 6.25, 6.50 [10:31:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.77, 6.95, 7.27 [10:31:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.84, 7.29, 7.34 [10:32:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.94, 7.92, 7.77 [10:34:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.27, 7.66, 7.70 [10:35:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.73, 8.31, 7.77 [10:36:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.16, 7.91, 7.78 [10:37:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.91, 7.61, 7.43 [10:38:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.98, 6.98, 7.45 [10:39:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.01, 7.17, 7.46 [10:39:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.00, 6.72, 7.13 [10:40:53] PROBLEM - phab121 APT on phab121 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [10:41:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.29, 7.44, 7.19 [10:41:46] PROBLEM - mwtask1 APT on mwtask1 is CRITICAL: APT CRITICAL: 22 packages available for upgrade (3 critical updates). [10:42:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.52, 7.77, 6.84 [10:42:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:43:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.12, 7.57, 7.54 [10:43:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.04, 8.22, 7.64 [10:43:23] PROBLEM - mw102 APT on mw102 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [10:43:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.91, 6.94, 7.05 [10:43:57] PROBLEM - mw121 APT on mw121 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [10:44:29] PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [10:44:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.31, 7.43, 6.83 [10:44:35] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 22 packages available for upgrade (3 critical updates). [10:44:56] PROBLEM - mail121 APT on mail121 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [10:45:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.39, 6.84, 7.21 [10:45:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.19, 6.11, 6.74 [10:46:55] PROBLEM - jobchron1 APT on jobchron1 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [10:47:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.74, 8.94, 7.51 [10:47:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.36, 7.67, 7.66 [10:49:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.94, 8.82, 8.07 [10:49:28] PROBLEM - jobchron121 APT on jobchron121 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [10:50:10] PROBLEM - test101 APT on test101 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [10:50:24] PROBLEM - mw112 APT on mw112 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [10:50:29] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.64, 6.10, 6.46 [10:50:31] PROBLEM - mw13 APT on mw13 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [10:50:32] PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [10:52:00] PROBLEM - mon2 APT on mon2 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (3 critical updates). [10:53:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.22, 7.73, 7.84 [10:53:07] PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [10:53:52] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (3 critical updates). [10:54:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.24, 8.35, 7.37 [10:54:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:55:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.70, 7.92, 7.80 [10:55:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.16, 8.05, 7.45 [10:56:26] PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 22 packages available for upgrade (3 critical updates). [10:57:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.25, 3.01 [10:57:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.13, 7.16, 7.22 [10:57:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.79, 7.82, 7.14 [10:58:23] PROBLEM - mon111 APT on mon111 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (3 critical updates). [10:58:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.55, 7.76, 7.41 [10:58:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.70, 8.74, 7.91 [10:59:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.44, 7.71, 7.68 [10:59:06] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.13, 7.30, 7.51 [10:59:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.32, 3.23, 3.03 [11:00:14] PROBLEM - mw12 APT on mw12 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:00:46] PROBLEM - mail2 APT on mail2 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:00:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.00, 7.71, 7.64 [11:01:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.74, 6.76, 7.35 [11:01:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.48, 6.50, 7.19 [11:01:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.66, 6.01, 6.70 [11:01:18] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.65, 6.29, 6.68 [11:02:37] PROBLEM - mw101 APT on mw101 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [11:03:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.94, 3.40, 3.12 [11:03:50] PROBLEM - phab2 APT on phab2 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [11:04:22] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.50, 5.73, 6.63 [11:04:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.29, 8.43, 7.88 [11:04:58] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.28, 3.24, 3.09 [11:05:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.23, 7.64, 7.45 [11:06:10] PROBLEM - mw111 APT on mw111 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [11:06:27] PROBLEM - mw122 APT on mw122 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [11:06:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.78, 7.73, 7.72 [11:07:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 3.24, 6.02, 6.89 [11:09:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.59, 5.34, 6.53 [11:09:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.05, 5.61, 6.61 [11:09:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:13:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.57, 6.59, 6.81 [11:14:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.86, 5.26, 6.52 [11:15:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.61, 6.18, 6.63 [11:23:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:24:54] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.33, 7.78, 6.91 [11:25:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.89, 6.82, 6.67 [11:26:51] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.33, 7.69, 6.99 [11:27:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.46, 6.61, 6.61 [11:28:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:28:47] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.32, 6.66, 6.69 [11:47:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:52:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:07:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.93, 6.59, 5.97 [12:09:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.57, 6.34, 5.97 [12:09:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:10:26] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wqbnwiki --username-prefix enwikiquote --report 1 /home/reception/Wikiquote-20220108091316.xml (START) [12:10:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:12:58] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wqbnwiki --username-prefix enwikiquote --report 1 /home/reception/Wikiquote-20220108091316.xml (END - exit=0) [12:13:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:14:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:15:53] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.33, 7.01, 6.20 [12:17:51] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.69, 7.33, 6.39 [12:19:50] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.81, 6.79, 6.29 [12:19:59] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.41, 3.02 [12:20:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:21:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.07, 7.48, 6.25 [12:21:45] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.72, 7.09, 6.50 [12:21:47] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.75, 8.34, 6.92 [12:21:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.09, 3.14, 2.99 [12:23:33] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.23, 19.97, 17.78 [12:23:40] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.42, 6.41, 6.31 [12:24:08] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.62, 7.94, 6.58 [12:25:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.93, 3.66, 3.23 [12:27:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 8.00, 7.98, 6.87 [12:27:44] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.79, 3.40, 3.19 [12:29:33] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.91, 19.43, 18.45 [12:29:57] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.78, 7.96, 7.10 [12:31:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.32, 5.87, 6.31 [12:31:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 3.74, 6.59, 7.10 [12:33:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.20, 5.53, 6.65 [12:33:50] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.43, 5.92, 6.46 [12:34:13] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26701 MB (5% inode=97%); [12:35:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:43:50] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.90, 7.22, 6.55 [12:44:07] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.10, 6.79, 5.93 [12:45:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.16, 7.17, 6.08 [12:46:05] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.62, 7.03, 6.12 [12:46:13] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 27619 MB (6% inode=97%); [12:46:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:47:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.93, 7.17, 6.19 [12:49:34] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.13, 8.22, 7.13 [12:50:00] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.97, 7.30, 6.40 [12:50:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.83, 3.87, 3.45 [12:51:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.91, 8.17, 6.81 [12:51:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:51:57] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.97, 7.38, 6.55 [12:52:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.60, 3.41 [12:53:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.57, 7.49, 6.72 [12:54:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.57, 7.00, 6.52 [12:54:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.35, 3.80, 3.49 [12:55:52] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.18, 6.53, 6.42 [12:56:25] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.08, 6.17, 6.27 [12:56:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.47, 3.02, 3.25 [12:57:11] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.48, 7.42, 7.29 [13:01:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.98, 6.33, 6.59 [13:01:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.87, 5.92, 6.76 [13:12:04] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.68, 6.22, 5.83 [13:14:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.65, 5.47, 5.60 [13:15:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:17:06] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.33, 6.89, 6.43 [13:20:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:21:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.20, 7.34, 6.71 [13:21:08] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.85, 7.42, 6.68 [13:21:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.39, 6.83, 6.32 [13:22:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.33, 21.17, 18.96 [13:23:05] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.30, 7.27, 6.74 [13:23:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.42, 3.13 [13:23:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:24:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.25, 19.29, 18.53 [13:24:46] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.07, 7.63, 6.55 [13:24:58] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.52, 6.90, 6.06 [13:25:03] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.10, 7.87, 7.01 [13:26:44] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.14, 6.45, 6.25 [13:26:57] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.80, 5.69, 5.72 [13:27:00] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.43, 7.10, 6.83 [13:27:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.41, 7.74, 7.20 [13:27:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.06, 6.49, 6.46 [13:28:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:28:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.42, 3.14, 3.12 [13:28:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.27, 6.34, 6.57 [13:29:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.49, 7.43, 7.50 [13:31:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.20, 6.25, 6.71 [13:37:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.57, 5.91, 6.72 [13:43:06] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.19, 7.52, 7.00 [13:45:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.14, 6.82, 6.83 [13:46:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:49:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.66, 7.42, 6.46 [13:49:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.97, 7.52, 7.09 [13:50:32] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.96, 6.97, 6.35 [13:51:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.71, 7.01, 6.43 [13:53:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.87, 5.99, 6.11 [13:53:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.00, 7.70, 7.33 [13:53:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.23, 7.06, 6.08 [13:54:27] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.24, 8.02, 6.93 [13:55:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.31, 8.40, 7.63 [13:55:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.04, 6.36, 5.96 [13:56:26] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.75, 7.55, 6.89 [13:56:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:57:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.52, 7.98, 7.57 [14:02:18] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.62, 6.21, 6.60 [14:03:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.27, 3.01 [14:05:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.88, 3.83, 3.25 [14:07:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.21, 5.56, 6.55 [14:09:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.84, 3.76, 3.35 [14:11:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.60, 4.02, 3.49 [14:13:24] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.25, 3.39, 3.33 [14:14:00] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.31, 6.57, 6.20 [14:15:57] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.96, 6.39, 6.19 [14:19:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.93, 3.78, 3.52 [14:21:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.16, 7.53, 6.98 [14:21:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.08, 4.32, 3.73 [14:21:47] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.84, 7.33, 6.59 [14:21:56] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.34, 6.77, 6.05 [14:22:06] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.12, 6.74, 6.02 [14:23:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.61, 7.30, 6.97 [14:23:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:23:45] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.86, 6.87, 6.52 [14:23:51] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.91, 6.22, 5.93 [14:24:02] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.35, 6.48, 6.01 [14:25:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.18, 3.72, 3.63 [14:25:42] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.11, 7.31, 6.72 [14:26:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 3.97, 3.74 [14:27:40] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.95, 7.39, 6.81 [14:28:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:28:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.91, 3.73 [14:31:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.68, 6.20, 6.68 [14:31:35] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.34, 6.55, 6.64 [14:34:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.64, 3.88, 3.69 [14:36:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.72, 3.66 [14:38:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.59, 4.17, 3.84 [14:44:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.60, 3.48, 3.68 [14:50:13] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26691 MB (5% inode=97%); [14:50:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.71, 4.11, 3.83 [14:51:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.21, 6.65, 5.78 [14:51:07] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.00, 7.75, 6.68 [14:51:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.17, 7.49, 6.59 [14:52:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.92, 3.80 [14:52:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.06, 2.72, 1.57 [14:53:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.24, 7.50, 6.71 [14:53:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:53:52] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.27, 6.51, 5.90 [14:54:09] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.80, 7.38, 6.40 [14:54:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.72, 4.34, 3.97 [14:54:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.46, 1.94, 1.42 [14:55:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.84, 6.28, 5.88 [14:55:01] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.16, 7.30, 6.77 [14:55:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.79, 6.80, 6.54 [14:56:06] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.32, 7.24, 6.47 [14:56:45] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.32, 1.39, 1.27 [14:57:48] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.12, 6.25, 5.96 [14:58:02] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.78, 6.53, 6.29 [15:01:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.79, 7.02, 6.28 [15:01:41] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.36, 6.90, 6.32 [15:03:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.79, 6.48, 6.17 [15:03:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:03:41] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.74, 6.46, 6.22 [15:04:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.60, 5.75, 6.47 [15:10:13] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 27588 MB (6% inode=97%); [15:10:40] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:11:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.83, 7.19, 6.77 [15:12:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.86, 4.00 [15:12:41] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.90, 8.02, 7.13 [15:13:41] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.16, 7.48, 6.60 [15:15:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:15:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.66, 7.60, 6.76 [15:17:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.01, 6.30, 6.60 [15:18:30] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.16, 7.93, 7.41 [15:19:50] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.04, 6.66, 6.44 [15:21:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.82, 6.95, 6.31 [15:21:06] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.02, 7.65, 7.08 [15:21:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.46, 6.36, 6.51 [15:23:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.46, 6.91, 6.36 [15:23:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:24:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 3.78, 3.81 [15:25:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.69, 7.16, 6.50 [15:25:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.16, 7.90, 7.38 [15:26:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.06, 7.91, 7.54 [15:27:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.11, 6.42, 6.31 [15:27:40] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.79, 7.31, 6.83 [15:27:41] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.37, 7.19, 6.86 [15:28:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.62, 3.93, 3.88 [15:29:08] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.25, 8.27, 7.59 [15:30:08] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.74, 7.67, 7.55 [15:30:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.29, 4.08, 3.94 [15:30:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/J9sKd [15:30:35] [02miraheze/puppet] 07paladox 031c87a44 - php: Allow disabling fpm [15:30:36] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [15:30:38] [02puppet] 07paladox opened pull request 03#2286: php: Allow disabling fpm - 13https://git.io/J9sKb [15:31:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.08, 6.84, 6.56 [15:31:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.82, 7.90, 7.55 [15:31:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-0/±0] 13https://git.io/J9sKx [15:31:20] [02miraheze/puppet] 07paladox 034491fb2 - Create php_version.pp [15:31:21] [02puppet] 07paladox synchronize pull request 03#2286: php: Allow disabling fpm - 13https://git.io/J9sKb [15:31:35] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.77, 7.72, 7.15 [15:32:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/J9sKj [15:32:19] [02miraheze/puppet] 07paladox 03e42a615 - Update init.pp [15:32:23] [02puppet] 07paladox synchronize pull request 03#2286: php: Allow disabling fpm - 13https://git.io/J9sKb [15:32:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.83, 3.86 [15:32:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/J9s6f [15:32:45] [02miraheze/puppet] 07paladox 033675820 - Update php_fpm.pp [15:32:46] [02puppet] 07paladox synchronize pull request 03#2286: php: Allow disabling fpm - 13https://git.io/J9sKb [15:33:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.00, 6.21, 6.36 [15:33:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.22, 7.87, 7.57 [15:33:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:33:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/J9s6I [15:33:50] [02miraheze/puppet] 07paladox 034c7bbc2 - Update php.pp [15:33:52] [02puppet] 07paladox synchronize pull request 03#2286: php: Allow disabling fpm - 13https://git.io/J9sKb [15:34:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/J9s6L [15:34:41] [02miraheze/puppet] 07paladox 03d65d8c0 - Update php.pp [15:34:42] [02puppet] 07paladox synchronize pull request 03#2286: php: Allow disabling fpm - 13https://git.io/J9sKb [15:35:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.55, 7.05, 7.31 [15:35:30] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.45, 5.58, 6.42 [15:35:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.79, 6.37, 6.65 [15:36:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/J9s6q [15:36:13] [02miraheze/puppet] 07paladox 034d32085 - Update jobchron1.yaml [15:36:14] [02puppet] 07paladox synchronize pull request 03#2286: php: Allow disabling fpm - 13https://git.io/J9sKb [15:36:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/J9s6Y [15:36:24] [02miraheze/puppet] 07paladox 03b03dc41 - Update jobchron121.yaml [15:36:26] [02puppet] 07paladox synchronize pull request 03#2286: php: Allow disabling fpm - 13https://git.io/J9sKb [15:37:53] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.73, 5.66, 6.60 [15:43:17] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.21, 6.74, 6.60 [15:43:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.06, 7.61, 7.06 [15:44:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.20, 4.11, 3.83 [15:45:14] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.73, 6.68, 6.59 [15:45:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:45:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.57, 7.36, 7.05 [15:48:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.91, 19.49, 17.81 [15:49:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.26, 7.70, 7.25 [15:50:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:51:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.63, 7.04, 6.63 [15:52:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.95, 19.48, 18.26 [15:52:32] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.25, 7.34, 6.89 [15:53:14] PROBLEM - housing.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'housing.wiki' expires in 15 day(s) (Mon 24 Jan 2022 15:46:28 GMT +0000). [15:53:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.58, 7.88, 7.55 [15:54:28] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.26, 6.64, 6.67 [15:55:08] PROBLEM - housing.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'housing.wiki' expires in 15 day(s) (Mon 24 Jan 2022 15:46:28 GMT +0000). [15:55:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:57:38] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.79, 7.35, 6.89 [15:59:33] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.37, 6.42, 6.60 [15:59:38] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9s6M [15:59:39] [02miraheze/ssl] 07MirahezeSSLBot 03aad265e - Bot: Update SSL cert for housing.wiki [16:00:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.10, 3.49, 3.85 [16:00:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:03:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.69, 5.56, 6.49 [16:05:41] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.32, 5.96, 6.65 [16:06:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 3.44, 3.67 [16:06:45] PROBLEM - publictestwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:39] PROBLEM - www.publictestwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:08:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.26, 3.57 [16:08:37] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.01, 6.44, 6.34 [16:08:58] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.27, 6.16, 5.82 [16:10:35] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.10, 6.14, 6.23 [16:10:56] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.89, 7.37, 6.28 [16:11:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:13:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.74, 7.42, 7.03 [16:14:27] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.07, 7.07, 6.60 [16:14:53] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.54, 7.78, 6.73 [16:16:24] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.09, 7.20, 6.71 [16:18:22] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.14, 6.73, 6.59 [16:19:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.31, 6.81, 6.38 [16:19:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.12, 7.05, 6.63 [16:20:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.63, 4.01, 3.81 [16:20:48] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.42, 6.74, 6.59 [16:21:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.38, 7.75, 7.25 [16:21:43] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.36, 7.77, 6.75 [16:21:48] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.24, 6.61, 6.51 [16:22:14] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.14, 7.29, 6.85 [16:22:54] RECOVERY - housing.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'housing.wiki' will expire on Fri 08 Apr 2022 14:59:32 GMT +0000. [16:23:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.97, 7.67, 6.83 [16:23:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.00, 7.55, 7.26 [16:24:43] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.04, 7.41, 6.95 [16:25:40] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.16, 7.61, 6.84 [16:26:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.89, 3.97, 3.92 [16:27:12] RECOVERY - housing.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'housing.wiki' will expire on Fri 08 Apr 2022 14:59:32 GMT +0000. [16:27:27] [02puppet] 07paladox closed pull request 03#2286: php: Allow disabling fpm - 13https://git.io/J9sKb [16:27:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±5] 13https://git.io/J9siu [16:27:30] [02miraheze/puppet] 07paladox 0315dee3d - php: Allow disabling fpm (#2286) [16:27:31] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [16:27:33] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [16:27:33] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.98, 6.69, 6.66 [16:27:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.63, 5.92, 6.66 [16:28:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.44, 7.30, 6.87 [16:28:40] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.96, 6.40, 6.62 [16:29:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9siz [16:29:22] [02miraheze/puppet] 07paladox 038ed6821 - Fix [16:30:30] PROBLEM - jobchron1 Puppet on jobchron1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:30:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9sia [16:30:53] [02miraheze/puppet] 07paladox 039d508fc - Update php_fpm.pp [16:31:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.80, 7.53, 7.11 [16:32:19] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:32:30] RECOVERY - jobchron1 Puppet on jobchron1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:32:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.95, 4.32, 4.06 [16:32:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.14, 6.80, 6.73 [16:32:58] [02puppet] 07Universal-Omega opened pull request 03#2287: MediaWiki: add 502 error page to custom domains also - 13https://git.io/J9si6 [16:33:59] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.91, 7.35, 7.55 [16:34:18] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.60, 6.79, 6.80 [16:34:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.59, 6.56, 6.64 [16:36:31] PROBLEM - publictestwiki.com - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/ [16:36:31] gios/plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL WARNING - rDNS OK but records conflict. {'NS': ['dns1.registrar-servers.com.', 'dns2.registrar-servers.c [16:36:31] 'CNAME': None} [16:39:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.90, 7.00, 6.69 [16:40:13] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.75, 7.68, 7.03 [16:40:28] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.79, 7.67, 7.03 [16:41:48] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.64, 7.35, 7.34 [16:43:46] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.83, 7.48, 7.39 [16:44:05] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.91, 7.53, 7.14 [16:44:19] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:45:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.28, 8.16, 7.43 [16:45:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.55, 6.20, 6.56 [16:45:43] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.41, 7.92, 7.54 [16:46:02] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.50, 7.75, 7.26 [16:46:45] [02puppet] 07Universal-Omega synchronize pull request 03#2259: MediaWiki: configure shellbox - 13https://git.io/JSDjg [16:46:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.29, 6.96, 6.84 [16:46:52] CosmicAlpha: 502 won't work because it'll be using php. It'll have to be in html. [16:47:58] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.69, 7.51, 7.24 [16:48:03] [02puppet] 07Universal-Omega synchronize pull request 03#2259: MediaWiki: configure shellbox - 13https://git.io/JSDjg [16:48:36] [02puppet] 07Universal-Omega synchronize pull request 03#2259: MediaWiki: configure shellbox - 13https://git.io/JSDjg [16:49:14] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:49:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 51.195.220.68/cpweb [16:49:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [16:49:54] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.14, 8.00, 7.44 [16:50:18] paladox: it does work, I tested on a known 502 page. [16:50:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.93, 19.94, 18.52 [16:50:38] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 7.93, 8.03, 7.32 [16:50:40] Yes it works with php but what if php doesn't respond? That's why there's a 502. [16:51:09] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.020 second response time [16:51:10] PROBLEM - www.publictestwiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.publictestwiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:51:11] i meant that if php didn't respond users won't see the 502 displayed for obvious reasons [16:51:46] Oh.... Good point. I'll convert to HTML then. [16:52:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 1.60, 3.19, 3.92 [16:53:19] that above error is correct - i cant reach testwiki at all [16:53:37] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.30, 7.75, 7.73 [16:53:39] domain expired [16:53:47] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.44, 7.39, 7.33 [16:54:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.78, 20.05, 18.93 [16:54:24] Reception123: @NDKilla: ^^ testwiki needs domain renewal [16:55:43] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.57, 8.07, 7.60 [16:56:31] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.90, 7.65, 6.81 [16:56:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.62, 4.04, 4.09 [16:57:31] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:57:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:57:59] RECOVERY - www.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Sun 27 Mar 2022 23:41:23 GMT +0000. [16:58:11] renewed [16:58:19] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.06, 7.71, 7.61 [16:58:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.58, 3.77, 3.98 [16:59:29] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.97, 7.70, 7.58 [17:00:14] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.65, 8.82, 8.05 [17:00:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.75, 4.25, 4.13 [17:01:42] [02puppet] 07Universal-Omega edited pull request 03#2259: MediaWiki: configure shellbox - 13https://git.io/JSDjg [17:03:14] RECOVERY - www.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Sun 27 Mar 2022 23:41:23 GMT +0000. [17:03:33] RECOVERY - publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Sun 27 Mar 2022 23:41:23 GMT +0000. [17:04:18] RECOVERY - publictestwiki.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [17:04:18] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - publictestwiki.com reverse DNS resolves to cp21.miraheze.org - CNAME OK [17:07:54] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.91, 7.93, 7.94 [17:08:17] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.53, 7.94, 7.60 [17:09:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:09:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb [17:10:15] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.27, 8.46, 7.83 [17:11:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:11:44] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.08, 7.86, 7.89 [17:14:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.39, 3.33, 3.91 [17:15:14] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:15:17] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:15:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [17:15:33] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:16:53] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:17:08] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:17:14] [02puppet] 07Universal-Omega synchronize pull request 03#2259: MediaWiki: configure shellbox - 13https://git.io/JSDjg [17:17:30] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.09, 7.81, 7.92 [17:19:58] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:20:23] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.99, 20.18, 19.29 [17:20:32] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:20:52] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:21:14] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:21:19] PROBLEM - www.publictestwiki.com - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/ [17:21:19] b/nagios/plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL WARNING - rDNS OK but records conflict. {'NS': ['dns1.registrar-servers.com.', 'dns2.registrar-serve [17:21:19] .'], 'CNAME': None} [17:21:21] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.85, 8.62, 8.19 [17:21:32] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:21:58] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 9.995 second response time [17:22:01] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 8.414 second response time [17:22:10] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:13] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:22:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.88, 19.80, 19.26 [17:22:40] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.950 second response time [17:22:53] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.919 second response time [17:23:00] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 2.452 second response time [17:23:14] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.991 second response time [17:23:19] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:23:22] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 5.117 second response time [17:23:29] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 3.800 second response time [17:23:32] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 1.553 second response time [17:24:15] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 5.895 second response time [17:24:17] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.732 second response time [17:27:21] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:27:44] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:27:49] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:27:57] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:28:26] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:28:41] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:28:42] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:28:42] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.32, 21.24, 20.08 [17:30:48] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.516 second response time [17:30:48] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.214 second response time [17:30:49] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 7.472 second response time [17:32:02] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.001 second response time [17:32:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.94, 3.70 [17:33:36] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.208 second response time [17:34:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.15, 20.38, 19.97 [17:34:50] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 9.946 second response time [17:34:55] RECOVERY - www.publictestwiki.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/ [17:34:55] gios/plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - www.publictestwiki.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [17:35:41] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 8.083 second response time [17:36:06] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 8.874 second response time [17:36:07] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 8.344 second response time [17:36:16] hmm, the check is using a deprecated function [17:36:26] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 9.380 second response time [17:36:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.95, 3.55, 3.60 [17:37:08] PROBLEM - publictestwiki.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['dns2.registrar-servers.com.', 'dns101.registrar-servers.com.', 'dns102.registrar-servers.com.', 'dns1.registrar-servers.com.'], 'CNAME': None} [17:37:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:39:23] PROBLEM - www.publictestwiki.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['dns1.registrar-servers.com.', 'dns2.registrar-servers.com.'], 'CNAME': None} [17:41:11] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:41:14] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:41:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:41:59] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:42:11] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:42:21] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:42:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.52, 3.08, 3.40 [17:43:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:43:53] RECOVERY - publictestwiki.com - reverse DNS on sslhost is OK: SSL OK - publictestwiki.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [17:44:06] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.225 second response time [17:45:16] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.633 second response time [17:45:19] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.915 second response time [17:46:11] RECOVERY - www.publictestwiki.com - reverse DNS on sslhost is OK: SSL OK - www.publictestwiki.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [17:46:35] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.216 second response time [17:47:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:48:30] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23541 bytes in 6.203 second response time [17:48:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.57, 3.83, 3.58 [17:50:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.74, 3.59 [17:51:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [17:52:13] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26704 MB (5% inode=97%); [17:54:18] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:56:11] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:57:12] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.91, 1.75, 1.30 [17:57:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:58:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.81, 3.92, 3.64 [17:59:00] [02puppet] 07Universal-Omega synchronize pull request 03#2259: MediaWiki: configure shellbox - 13https://git.io/JSDjg [17:59:12] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.82, 1.34, 1.20 [18:00:13] [02puppet] 07Universal-Omega synchronize pull request 03#2259: MediaWiki: configure shellbox - 13https://git.io/JSDjg [18:02:15] [02puppet] 07Universal-Omega synchronize pull request 03#2259: MediaWiki: configure shellbox - 13https://git.io/JSDjg [18:02:46] [02puppet] 07Universal-Omega synchronize pull request 03#2259: MediaWiki: configure shellbox - 13https://git.io/JSDjg [18:02:50] PROBLEM - publictestwiki.com - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/ [18:02:50] gios/plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL WARNING - rDNS OK but records conflict. {'NS': ['dns1.registrar-servers.com.', 'dns2.registrar-servers.c [18:02:50] 'CNAME': None} [18:07:34] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [18:08:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [18:08:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.75, 3.88, 3.91 [18:09:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.86, 6.61, 7.84 [18:09:47] RECOVERY - publictestwiki.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [18:09:47] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - publictestwiki.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [18:09:59] [02puppet] 07Universal-Omega synchronize pull request 03#2259: MediaWiki: configure shellbox - 13https://git.io/JSDjg [18:10:21] @Reception123 hello - i saw your site notice at https://robloxapi.wiki/w/index.php?title=MediaWiki:Sitenotice. we have no DNS errors, you can visit our@custom domain at https://robloxapi.wiki and it works fine! please do not remove it [18:10:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.61, 4.46, 4.13 [18:10:50] [url] MediaWiki:Sitenotice - RoAPI | robloxapi.wiki [18:10:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.81, 6.59, 7.88 [18:11:11] [url] RoAPI | robloxapi.wiki [18:11:40] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:11:49] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:58] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:12:24] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:12:26] @jmk: let me see why [18:12:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.86, 3.82, 3.93 [18:12:54] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:06] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:08] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:21] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:21] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:26] Don't you use CloudFlare to proxy connections? That's in violation of our Custom domains policy [18:13:52] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 8.671 second response time [18:14:26] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:14:46] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:15:13] Agent: yep, that's what our records show [18:17:32] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.946 second response time [18:17:34] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:17:51] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:17:59] RhinosF1, how far back do I have to scrollback? [18:18:13] dmehus: yesterday evening [18:18:15] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.43, 20.25, 19.62 [18:18:23] RhinosF1, ack, looking [18:18:25] [02puppet] 07Universal-Omega synchronize pull request 03#2259: MediaWiki: configure shellbox - 13https://git.io/JSDjg [18:18:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 4.10, 4.01 [18:18:33] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.507 second response time [18:18:58] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.363 second response time [18:18:59] [02puppet] 07Universal-Omega synchronize pull request 03#2259: MediaWiki: configure shellbox - 13https://git.io/JSDjg [18:19:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.26, 7.46, 7.55 [18:19:17] [02puppet] 07Universal-Omega synchronize pull request 03#2259: MediaWiki: configure shellbox - 13https://git.io/JSDjg [18:20:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.96, 3.98 [18:20:44] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:20:54] [02puppet] 07Universal-Omega synchronize pull request 03#2259: MediaWiki: configure shellbox - 13https://git.io/JSDjg [18:20:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.14, 7.11, 7.44 [18:21:33] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 3.668 second response time [18:21:34] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.448 second response time [18:21:39] ```RhinosF1> [18:21:39] dmehus: 21:45 next Friday will be the stop for renames``` [18:21:39] RhinosF1: ack. Is that 21:45 UTC, GMT, or my time? And if UTC/GMT, how many hours ahead are you right now? [18:21:58] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.325 second response time [18:22:01] dmehus: UTC and it's 18:21 now [18:22:11] RhinosF1, ack, thanks :) [18:22:19] so 8 hours ahead [18:22:39] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.919 second response time [18:22:51] I won't plan any global renames for all of next Friday, to be absolutely safe [18:23:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.95, 7.49, 7.62 [18:23:27] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.177 second response time [18:24:25] PROBLEM - publictestwiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address publictestwiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [18:24:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.14, 7.53, 7.60 [18:25:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.19, 6.96, 7.88 [18:26:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.87, 4.28, 4.07 [18:26:59] Why is icinga-new saying crit [18:27:13] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.685 second response time [18:27:29] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.010 second response time [18:27:47] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 1.155 second response time [18:27:59] dmehus: has NDK changed the DNS records [18:28:04] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.772 second response time [18:28:19] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.640 second response time [18:28:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.50, 8.41, 7.93 [18:29:42] Agent: can you ask NDK to check DNS is CNAMEs on both www. and without [18:29:56] New infra is complaining that it can't reach and it looks like A records [18:29:57] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.52, 20.30, 20.22 [18:30:06] RhinosF1: ok, I'll ask him [18:30:54] RhinosF1, dunno, I asked him in -cvt-private [18:31:05] (with a ping on Discord) [18:31:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.69, 7.91, 7.91 [18:31:08] RECOVERY - publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Sun 27 Mar 2022 23:41:23 GMT +0000. [18:31:42] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:32:37] dmehus Check #miraheze-offtopic-relay [18:33:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.58, 7.59, 7.40 [18:33:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:37:09] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-1/±0] 13https://git.io/J9sHF [18:37:10] [02miraheze/ErrorPages] 07Universal-Omega 037853871 - Convert 502 ErrorPage to html [18:37:12] [02ErrorPages] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/JuBZG [18:37:17] [02ErrorPages] 07Universal-Omega opened pull request 03#4: Convert 502 error page to html - 13https://git.io/J9sHb [18:37:32] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J9sHA [18:37:34] [02miraheze/ErrorPages] 07Universal-Omega 030cb1b57 - Update en.json [18:37:35] [02ErrorPages] 07Universal-Omega synchronize pull request 03#4: Convert 502 error page to html - 13https://git.io/J9sHb [18:37:45] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J9sHh [18:37:47] [02miraheze/ErrorPages] 07Universal-Omega 03de212fe - Update qqq.json [18:37:48] [02ErrorPages] 07Universal-Omega synchronize pull request 03#4: Convert 502 error page to html - 13https://git.io/J9sHb [18:38:18] [02puppet] 07Universal-Omega edited pull request 03#2287: MediaWiki: fix 502 error page and enable for custom domains - 13https://git.io/J9si6 [18:38:18] miraheze/ErrorPages - Universal-Omega the build passed. [18:38:36] miraheze/ErrorPages - Universal-Omega the build passed. [18:38:42] [02puppet] 07Universal-Omega synchronize pull request 03#2287: MediaWiki: fix 502 error page and enable for custom domains - 13https://git.io/J9si6 [18:38:48] miraheze/ErrorPages - Universal-Omega the build passed. [18:39:03] [02puppet] 07Universal-Omega edited pull request 03#2287: MediaWiki: fix 502 error page to be html - 13https://git.io/J9si6 [18:40:32] dmehus, Agent: it looks ok now [18:40:53] RhinosF1, ah, ok, looks like he must've fixed it I guess then [18:41:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [18:41:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [18:41:33] I’ve forgotten why it isn’t allowed - could you remind me? [18:41:44] Privacy concerns [18:42:11] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J9sQE [18:42:12] CosmicAlpha: it'd be nice if it gave a traceback if it was a php error [18:42:13] [02miraheze/ErrorPages] 07Universal-Omega 03249c99d - Use logo from Wikimedia Commons [18:42:14] [02ErrorPages] 07Universal-Omega synchronize pull request 03#4: Convert 502 error page to html - 13https://git.io/J9sHb [18:42:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.20, 3.47, 4.00 [18:42:33] what was @jmk asking about? [18:43:08] custom domain removal notice from Reception123 for policy inconsistencies [18:43:12] miraheze/ErrorPages - Universal-Omega the build passed. [18:43:16] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:43:21] RhinosF1: that can't be done in pure HTML, which as paladox mentioned it can't be php, since it can be php failing that causes 502s. [18:43:33] RhinosF1, ah [18:43:39] ohh it’s privacy reasons [18:43:40] CosmicAlpha: yeah true [18:43:43] Yes [18:43:54] that’s frustrating. i’ll have to get rid of it then [18:44:03] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:44:10] You need to disable the orange cloud next to the record [18:44:21] i know but then i miss out on the point of using cloud flare [18:44:28] https://phabricator.miraheze.org/T5644 [18:44:29] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:44:29] [url] ⚓ T5644 Create custom domain policy | phabricator.miraheze.org [18:44:31] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:44:37] But we don't need caching on our domains [18:44:38] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:44:41] this task was repsonsible for creating the policy [18:44:46] That's why we have cache proxies [18:44:49] RhinosF1: jmk wants statistics [18:44:54] yep [18:44:57] It's a privacy and security risk [18:45:00] On what? [18:45:08] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:45:12] site visits, requests etc [18:45:29] nothing in-depth, they don’t give it to you [18:45:32] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9sQ5 [18:45:33] just general stats [18:45:34] [02miraheze/puppet] 07Universal-Omega 031880c85 - MediaWiki: fix 502 error page to be html (#2287) [18:45:35] [02puppet] 07Reception123 closed pull request 03#2287: MediaWiki: fix 502 error page to be html - 13https://git.io/J9si6 [18:45:42] You can find them on wiki [18:45:58] [02ErrorPages] 07Universal-Omega closed pull request 03#4: Convert 502 error page to html - 13https://git.io/J9sHb [18:46:00] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+1/-1/±2] 13https://git.io/J9sQA [18:46:01] [02miraheze/ErrorPages] 07Universal-Omega 03660b41e - Convert 502 error page to html (#4) [18:46:26] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.611 second response time [18:46:35] yeah i’ve seen those but they simply aren’t comparable to the charts and visualizations they give you [18:46:54] miraheze/ErrorPages - Universal-Omega the build passed. [18:48:27] PROBLEM - publictestwiki.com - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])rDNS WARNING - reverse DNS entry for publictestwiki.com could not be found [18:50:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.64, 3.54, 3.70 [18:50:47] @jmk: we can provide better data on request infrequently [18:51:01] Unless you disable cloudflare, we will remove your custom domain [18:51:36] Sorry about the inconvenience :( [18:51:54] as RhinosF1 says we will be happy to provide you with better data as a way to minimize the inconvenience [18:52:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.15, 2.99, 3.48 [18:52:48] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.350 second response time [18:52:51] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.315 second response time [18:53:29] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 0.034 second response time [18:53:31] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 0.402 second response time [18:54:17] :( alright [18:54:21] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.523 second response time [18:54:24] I understand [18:54:26] RhinosF1: Re publictestwiki.com probably related to it expiring and being renewed today [18:54:57] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:55:00] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:55:02] which caused it to be down temporarily [18:55:03] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:55:10] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:55:10] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:55:24] RECOVERY - publictestwiki.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [18:55:24] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - publictestwiki.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [18:56:17] That check needs updating [18:56:56] MacFan4000: I have a PR to fix, but it only will work on new infra so waiting till then. [18:57:02] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.122 second response time [18:57:07] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.699 second response time [18:57:07] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.263 second response time [18:57:07] ah [18:57:12] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 7.807 second response time [18:57:14] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.164 second response time [18:57:35] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:57:56] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:58:19] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:58:28] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:58:31] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:59:17] PROBLEM - www.publictestwiki.com - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/ [18:59:17] b/nagios/plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL WARNING - rDNS OK but records conflict. {'NS': ['dns1.registrar-servers.com.', 'dns101.registrar-ser [18:59:17] om.', 'dns102.registrar-servers.com.', 'dns2.registrar-servers.com.'], 'CNAME': None} [18:59:38] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 6.563 second response time [18:59:57] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.078 second response time [19:00:18] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.309 second response time [19:00:23] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.140 second response time [19:00:26] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.985 second response time [19:02:38] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:03:05] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:03:11] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:03:50] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:03:59] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:04:40] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 6.048 second response time [19:05:03] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 3.273 second response time [19:05:09] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 1.346 second response time [19:05:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:05:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:05:46] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 1.486 second response time [19:05:58] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 1.478 second response time [19:06:05] RECOVERY - www.publictestwiki.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/ [19:06:05] gios/plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - www.publictestwiki.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [19:07:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.96, 7.31, 7.93 [19:08:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.49, 3.07, 3.36 [19:09:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.22, 7.62, 7.96 [19:11:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb [19:12:27] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:12:57] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:13:08] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.14, 20.79, 19.71 [19:13:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:13:53] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:14:00] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:14:11] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:14:14] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:14:24] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:14:26] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:14:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.42, 3.41 [19:15:22] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:15:25] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.309 second response time [19:15:57] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.889 second response time [19:16:17] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.572 second response time [19:16:18] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 8.077 second response time [19:16:25] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 0.562 second response time [19:16:29] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.197 second response time [19:17:01] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 3.075 second response time [19:17:24] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 6.558 second response time [19:17:30] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 8.510 second response time [19:18:05] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 5.716 second response time [19:20:22] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:20:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.95, 3.31, 3.38 [19:20:37] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:20:45] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:20:55] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:21:45] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:21:56] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:22:25] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:22:54] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.50, 6.93, 7.96 [19:23:28] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:24:29] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 9.274 second response time [19:24:58] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 6.510 second response time [19:25:00] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 7.165 second response time [19:25:05] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.188 second response time [19:25:09] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.862 second response time [19:25:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.15, 7.74, 8.15 [19:26:09] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 9.500 second response time [19:26:30] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.65, 20.32, 20.22 [19:26:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 3.59, 3.49 [19:26:38] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.983 second response time [19:26:47] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.902 second response time [19:27:41] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 8.602 second response time [19:29:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:29:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:29:57] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.393 second response time [19:32:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.42, 3.17, 3.36 [19:33:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [19:33:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.20, 7.52, 7.93 [19:35:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.62, 8.46, 8.21 [19:36:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.28, 3.84, 3.58 [19:38:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.51, 3.49 [19:41:36] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:41:44] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:42:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:42:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.85, 3.13, 3.38 [19:42:47] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:42:57] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:43:36] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:43:43] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:43:44] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:43:50] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:44:17] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:44:42] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.794 second response time [19:44:54] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.316 second response time [19:45:34] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.001 second response time [19:45:45] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.605 second response time [19:45:56] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 3.469 second response time [19:46:03] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:46:13] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:46:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.47, 3.72, 3.54 [19:49:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:50:07] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [19:50:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.68, 3.58 [19:52:03] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 2.765 second response time [19:52:07] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 3.032 second response time [19:52:09] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 3.396 second response time [19:52:14] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:17] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 6.721 second response time [19:52:19] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:52:23] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:52:37] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:52:58] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:53:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:53:39] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:54:00] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:54:09] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:54:21] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 8.999 second response time [19:54:26] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.016 second response time [19:54:30] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.508 second response time [19:54:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.83, 3.65 [19:54:40] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.295 second response time [19:54:57] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.732 second response time [19:54:57] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:18] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:55:43] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.938 second response time [19:56:32] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:56:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.81, 3.67 [19:56:33] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:39] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:56:56] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.55, 20.31, 19.64 [19:57:01] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 6.659 second response time [19:57:17] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.532 second response time [19:58:06] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.412 second response time [19:58:11] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.898 second response time [19:58:27] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:58:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.74, 20.14, 19.66 [20:00:26] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 4.318 second response time [20:00:40] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 4.426 second response time [20:00:44] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 3.251 second response time [20:00:44] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 2.576 second response time [20:00:59] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 9.363 second response time [20:01:13] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:03:10] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.314 second response time [20:03:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:07:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:11:09] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:11:13] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:11:18] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.254 second response time [20:11:19] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:11:23] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:11:24] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:11:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:11:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:11:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.47, 7.50, 7.98 [20:11:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:12:01] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:12:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.42, 3.25, 3.40 [20:14:14] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:14:16] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:14:44] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:15:29] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.563 second response time [20:15:29] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.036 second response time [20:15:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:15:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.92, 8.52, 8.26 [20:16:15] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.515 second response time [20:16:21] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 6.097 second response time [20:16:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.93, 3.69, 3.54 [20:16:49] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.898 second response time [20:19:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.52, 7.39, 7.93 [20:19:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:19:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.18, 7.18, 7.76 [20:19:55] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:20:40] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:20:48] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:21:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.81, 8.30, 8.21 [20:21:12] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:21:53] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:21:54] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:21:57] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:22:15] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.541 second response time [20:22:20] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:23:16] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.918 second response time [20:23:34] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 1.830 second response time [20:23:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.80, 7.74, 7.83 [20:23:40] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.336 second response time [20:23:42] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.343 second response time [20:23:49] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.016 second response time [20:23:51] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.324 second response time [20:23:52] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.349 second response time [20:23:58] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 0.358 second response time [20:24:00] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.500 second response time [20:24:21] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 1.523 second response time [20:24:43] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:24:43] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.389 second response time [20:24:57] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 0.494 second response time [20:24:59] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:26:22] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:26:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.68, 3.83, 3.61 [20:26:35] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:50] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.037 second response time [20:28:19] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.567 second response time [20:28:35] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 2.226 second response time [20:29:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.86, 7.95, 7.97 [20:29:57] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:29:59] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:30:08] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:30:20] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:30:25] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:31:16] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.310 second response time [20:31:24] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:31:28] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:32:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.82, 3.74 [20:32:56] [02mw-config] 07RhinosF1 commented on pull request 03#4354: DC-Switch: Disable renames & wiki creation - 13https://git.io/J9sjr [20:33:23] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.015 second response time [20:33:23] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 0.389 second response time [20:33:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:33:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.63, 7.89, 7.86 [20:35:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.22, 7.52, 7.88 [20:35:05] [02mw-config] 07dmehus commented on pull request 03#4354: DC-Switch: Disable renames & wiki creation - 13https://git.io/J9sj5 [20:36:05] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.041 second response time [20:36:08] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.350 second response time [20:36:22] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 4.375 second response time [20:36:29] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.020 second response time [20:36:34] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 0.342 second response time [20:36:42] [02mw-config] 07RhinosF1 commented on pull request 03#4354: DC-Switch: Disable renames & wiki creation - 13https://git.io/J9Gef [20:36:58] dmehus: no benefit here [20:37:09] upload anyone can add to any group [20:37:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:37:37] Anyone who can control them rights being in other groups can by pass $wgRevokePermissions [20:38:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.53, 4.06, 3.77 [20:38:50] [02mw-config] 07dmehus commented on pull request 03#4354: DC-Switch: Disable renames & wiki creation - 13https://git.io/J9Gem [20:39:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.08, 7.98, 7.95 [20:39:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.58, 7.58, 7.79 [20:39:42] dmehus: I trust you to not override it. I was taking wc off and it was there though. [20:40:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.80, 3.72 [20:40:39] [02mw-config] 07dmehus edited a comment on pull request 03#4354: DC-Switch: Disable renames & wiki creation - 13https://git.io/J9Gem [20:41:13] RhinosF1, oh, I thought Agent said you were doing that for both user rights [20:41:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.36, 8.33, 8.04 [20:41:42] I guess that's okay, but still would prefer wgRevokePermissions for `wikicreator`/`createwiki` [20:42:57] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.48, 6.95, 7.57 [20:43:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.98, 6.55, 7.87 [20:44:20] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:44:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [20:44:34] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:44:46] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:44:46] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:47:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 51.195.220.68/cpweb [20:49:27] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:49:58] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:49:59] [02mw-config] 07dmehus opened pull request 03#4364: Re-add user rights to steward group - 13https://git.io/J9Gvz [20:50:06] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:50:35] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:50:49] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:50:59] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 5.364 second response time [20:51:00] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 5.020 second response time [20:51:02] miraheze/mw-config - dmehus the build passed. [20:51:04] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:19] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 0.504 second response time [20:51:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.35, 20.59, 20.24 [20:51:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.74, 7.50, 7.52 [20:51:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.75, 8.17, 7.99 [20:51:52] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:52:04] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:52:06] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 8.624 second response time [20:52:10] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.843 second response time [20:52:14] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:52:32] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.025 second response time [20:52:34] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.629 second response time [20:52:45] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.344 second response time [20:53:29] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.394 second response time [20:53:53] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.258 second response time [20:54:06] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.324 second response time [20:54:23] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.688 second response time [20:54:59] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.163 second response time [20:55:16] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 3.740 second response time [20:55:16] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.66, 20.24, 20.21 [20:55:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:56:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.72, 3.57 [20:58:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.95, 3.47, 3.50 [20:59:05] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.78, 20.77, 20.47 [20:59:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:00:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.78, 3.61 [21:03:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.78, 7.79, 7.75 [21:04:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.11, 3.39, 3.48 [21:05:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:05:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.24, 8.08, 7.86 [21:05:56] [02mw-config] 07RhinosF1 closed pull request 03#4364: Re-add user rights to steward group - 13https://git.io/J9Gvz [21:05:57] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/J9GJG [21:05:59] [02miraheze/mw-config] 07dmehus 039631bb6 - Re-add user rights to steward group (#4364) [21:06:01] [02mw-config] 07RhinosF1 synchronize pull request 03#4354: DC-Switch: Disable renames & wiki creation - 13https://git.io/J9ORf [21:06:31] [02mw-config] 07RhinosF1 edited pull request 03#4354: DC-Switch: Disable wiki creation (trust Stews for rename) - 13https://git.io/J9ORf [21:06:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.26, 3.10, 3.37 [21:06:43] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.61, 22.12, 21.10 [21:07:07] miraheze/mw-config - RhinosF1 the build passed. [21:07:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.65, 7.77, 7.77 [21:07:42] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:08:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.36, 20.03, 20.48 [21:08:41] [02mw-config] 07Universal-Omega reviewed pull request 03#4354 commit - 13https://git.io/J9GJo [21:09:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.99, 6.86, 7.93 [21:09:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.06, 8.28, 7.94 [21:10:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.14, 19.68, 20.30 [21:11:47] [02mw-config] 07dmehus reviewed pull request 03#4354 commit - 13https://git.io/J9GJF [21:12:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.74, 3.55 [21:13:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.93, 7.65, 7.92 [21:13:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [21:14:05] [02mw-config] 07dmehus opened pull request 03#4365: Comment out `createwiki` user right - 13https://git.io/J9GUU [21:14:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 3.59, 3.52 [21:15:07] miraheze/mw-config - dmehus the build passed. [21:16:05] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [21:16:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.15, 3.07, 3.33 [21:19:28] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:19:28] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:19:36] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:19:40] CosmicAlpha: I couldn't remember if groups got deleted with no rights assigned [21:19:44] Like ManageWiki does [21:20:07] Didn't want it to disappear from Special:UserRights etc [21:20:12] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:20:15] Even if it was only for half an hour [21:21:09] RhinosF1, yeah I wondered that, too, I'm not entirely certain. In theory, the group should reappear, but it's possible it won't [21:21:14] so I'll withdraw my PR [21:21:27] RhinosF1: hey that's that's good point, that could be tested also I guess (for future) [21:21:43] dmehus Can I talk to you in private about something? [21:22:04] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:06] [02mw-config] 07dmehus commented on pull request 03#4365: Comment out `createwiki` user right - 13https://git.io/J9GUX [21:22:15] [02mw-config] 07dmehus closed pull request 03#4365: Comment out `createwiki` user right - 13https://git.io/J9GUU [21:22:26] darkmatterman450, sure [21:22:28] CosmicAlpha, dmehus: np [21:22:51] and yeah we could test that on betaheze I suppose [21:22:53] Most of the stuff I do has some reason behind [21:23:00] There's method to most of my madness [21:23:30] I don't think anyone can create wikis there at moment [21:25:49] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 8.099 second response time [21:25:50] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 7.494 second response time [21:26:11] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 4.155 second response time [21:26:21] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 2.791 second response time [21:27:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:27:46] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 6.677 second response time [21:30:34] dmehus: Betaheze doesn't have CW cluster yet,so can't be tested there, I'll do that after the migration, probably a week or 2 after so we can just finish that, and so we can have other test wikis. [21:34:17] CosmicAlpha: oh, yeah [21:34:17] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [21:34:20] that makes sense [21:34:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.14, 4.43, 3.74 [21:34:55] so no ManageWiki on Betaheze I guess then either [21:35:12] PROBLEM - wiki.helioss.co - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.helioss.co and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:35:13] It has ManageWiki, just uses production still. [21:35:17] dmehus [21:35:38] ^ [21:35:48] It will get C/MW soon [21:36:02] but i need CosmicAlpha's help with that [21:38:05] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:38:49] Yeah I can do the CW setup in probably 2 or 3 weeks RhinosF1. Then you should be good to finish ManageWiki tables, which just need creating on testglobal. Same with GlobalNewFiles, and MatomoAnalytics. (Which all use the CW database) [21:38:59] CosmicAlpha, ah okay, that makes sense. I didn't think ManageWiki worked without CreateWiki though? [21:38:59] Yes [21:39:09] dmehus: it doesn't [21:39:16] CW works without MW [21:39:21] But not the other way round [21:39:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:39:46] RhinosF1, ah, so ManageWiki doesn't work on Betaheze then currently? [21:39:58] dmehus: it does [21:40:03] It still uses production [21:40:05] dmehus: CW, and MW are production usage on betaheze. [21:40:07] As with CreateWiki [21:40:24] They don't use testglobal, but mhglobal, even on betaheze. [21:40:27] For now. [21:40:28] oh! [21:40:43] so you can create a wiki on Betaheze but it'll create a wiki on production? [21:40:49] Yes [21:40:52] ah [21:40:55] that makes sense [21:41:14] Good think we haven't assigned `createwiki` to anyone on Betaheze then :P [21:41:47] If the server name were just changed it would load betaheze config though. But making mre wikis on betaheze now would make switching harder as they'd all have to be manually moved to testglobal later. [21:41:52] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb [21:42:23] dmehus: wait no you can't create a wiki on betaheze, but rather from production and move it to betaheze if wanted. [21:42:23] Yeah [21:42:39] No one has createwiki rights [21:42:53] Because it's blacklisted in ManageWiki and no one can edit global groups [21:43:02] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:43:40] RhinosF1: when we do CW we can get global groups setup on betaheze also, as we are making a global cluster at that stage. [21:43:46] CosmicAlpha: do we ever run purgeExpiredUserrights.php [21:43:53] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:43:59] CosmicAlpha: and yeah, I've just never copied the config over from meta [21:44:08] No one has needed any rights yet [21:44:09] RhinosF1: no, we don't. [21:44:20] CosmicAlpha: we possibly should [21:44:21] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:44:29] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:44:31] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:44:31] And yeah don't do that until CW is on betaheze. [21:44:50] https://www.mediawiki.org/wiki/Manual:PurgeExpiredUserrights.php [21:44:50] [url] Manual:purgeExpiredUserrights.php - MediaWiki | www.mediawiki.org [21:45:03] I wonder how much nonsense we are storing [21:45:40] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:46:07] CosmicAlpha: does it sound sensible? [21:46:21] Not sure if worth waiting until new cluster [21:46:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.74, 3.63, 3.89 [21:46:37] As it could be quite big [21:46:42] Might cause lag [21:46:54] cc paladox [21:47:33] RhinosF1: we should wait. But it won't really benefit anything would it? It just moves them from one table to another. [21:48:30] CosmicAlpha: possibly not [21:49:16] I noticed it on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/752300/ [21:49:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:49:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:49:58] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 1.476 second response time [21:50:41] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 6.418 second response time [21:50:42] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 6.003 second response time [21:50:49] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 6.837 second response time [21:51:15] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:20] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 7.482 second response time [21:51:23] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:51:32] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:37] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:39] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:51] PROBLEM - wiki.helioss.co - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])rDNS WARNING - reverse DNS entry for wiki.helioss.co could not be found [21:52:14] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:52:17] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:52:17] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:52:40] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:52:46] PROBLEM - wiki.helioss.co - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.helioss.co could not be found [21:53:17] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.554 second response time [21:53:27] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 5.444 second response time [21:53:30] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.359 second response time [21:53:36] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.261 second response time [21:53:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.16, 7.43, 7.98 [21:53:43] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.071 second response time [21:53:52] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:53:57] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:54:13] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:54:15] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.405 second response time [21:54:21] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 5.339 second response time [21:54:27] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:54:38] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.033 second response time [21:55:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.34, 8.17, 8.19 [21:55:56] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.477 second response time [21:56:24] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 4.685 second response time [21:56:28] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:56:30] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.518 second response time [21:56:34] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:56:46] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:56:49] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:57:00] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:57:05] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:57:19] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:58:30] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 1.032 second response time [21:58:57] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.728 second response time [21:59:05] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.922 second response time [21:59:17] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.022 second response time [21:59:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.76, 19.97, 19.45 [21:59:36] RECOVERY - wiki.helioss.co - reverse DNS on sslhost is OK: SSL OK - wiki.helioss.co reverse DNS resolves to cp21.miraheze.org - CNAME OK [22:00:06] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.843 second response time [22:00:25] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 1.147 second response time [22:00:26] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.009 second response time [22:00:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.32, 3.95, 3.77 [22:00:49] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.457 second response time [22:00:53] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.343 second response time [22:01:13] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.76, 19.76, 19.43 [22:01:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:02:24] I can't see a use case for `purgeExpiredUserRights.php`. Also, would it potentially conflict with ManageWiki at all? [22:03:04] RhinosF1: That CA script actually deletes, not moves also. icinga-miraheze [22:03:25] Lol, I pinged icinga-miraheze! Whoops [22:04:22] dmehus: no how would it conflict with ManageWiki [22:05:12] CosmicAlpha: ah, so the CA would potentially be worth it [22:05:24] RhinosF1, yeah, maybe not. I was just thinking since ManageWiki user group assignments don't use the `userrights` user right [22:05:30] But then temp global groups aren't gonna take up much as I imagine them more rare [22:05:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [22:05:40] dmehus: still don't know why it would impact [22:05:56] yeah, probably not now that I think it through [22:05:57] CosmicAlpha: not sure it would actually do anything useful for us then no [22:06:46] though if user rights are set in LocalSettings.php, how would it purge expired user rights from wikis? [22:07:03] (for those that are set in LS.php of course) [22:07:08] dmehus: you don't assign people rights from LS.php [22:07:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:07:30] I have no idea what you think that script does [22:07:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:07:33] oh, so this purges the expired user groups from the user tables, nothing to do with the user rights? [22:07:49] dmehus: well moves them to the former table [22:07:56] RhinosF1, oh [22:07:57] ok [22:08:01] forget I said anything [22:08:07] I think that's best [22:08:09] [[w:WP:MEMORYHOLE]] [22:08:09] https://en.wikipedia.org/wiki/WP:MEMORYHOLE [22:08:16] oops [22:08:19] wrong link [22:08:24] [[w:WP:1984]] [22:08:24] https://en.wikipedia.org/wiki/WP:1984 [22:08:28] PROBLEM - wiki.helioss.co - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns01.domaincontrol.com.', 'ns02.domaincontrol.com.'], 'CNAME': 'pr-suspensions.go.co.'} [22:10:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.88, 3.96, 3.95 [22:13:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:14:17] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [22:16:09] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:16:12] PROBLEM - wiki.helioss.co - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.helioss.co and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [22:16:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 3.86, 3.85 [22:18:09] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.393 second response time [22:18:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.34, 3.30, 3.64 [22:19:59] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:20:00] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:20:33] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:20:34] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:20:34] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:22:22] [02ssl] 07Universal-Omega opened pull request 03#465: Remove wiki.helioss.co - 13https://git.io/J9GIh [22:22:33] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 4.100 second response time [22:22:33] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 3.301 second response time [22:22:35] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 3.419 second response time [22:22:56] [02ssl] 07Universal-Omega synchronize pull request 03#465: Remove wiki.helioss.co - 13https://git.io/J9GIh [22:24:03] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 4.291 second response time [22:24:06] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 4.533 second response time [22:25:02] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:25:23] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:25:28] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:26:03] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:26:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.25, 3.46 [22:26:51] [02ssl] 07dmehus commented on pull request 03#465: Remove wiki.helioss.co - 13https://git.io/J9GLJ [22:27:02] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.593 second response time [22:27:22] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.665 second response time [22:27:28] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.697 second response time [22:27:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:27:35] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:27:58] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 0.921 second response time [22:28:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.11, 3.23, 3.43 [22:30:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 2.97, 3.30 [22:33:40] [02ssl] 07Universal-Omega opened pull request 03#466: Remove thesimswiki.com redirect - 13https://git.io/J9GLZ [22:37:46] dmehus: do you see a use for temp global groups for us [22:39:21] [02ssl] 07Universal-Omega opened pull request 03#467: Remove wiki.raghuveer.net - 13https://git.io/J9GL0 [22:39:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.62, 7.52, 7.98 [22:40:12] [02ssl] 07Universal-Omega synchronize pull request 03#467: Remove wiki.raghuveer.net - 13https://git.io/J9GL0 [22:41:01] [02ssl] 07Universal-Omega edited pull request 03#467: Remove wiki.raghuveer.net - 13https://git.io/J9GL0 [22:41:19] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.62, 4.30, 3.75 [22:41:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.54, 8.16, 8.15 [22:42:12] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [22:46:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:47:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.42, 3.54, 3.62 [22:48:07] RhinosF1, potentially, yes, for testing CentralAuth changes and MediaWiki extension improvements [22:49:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.79, 3.69 [22:49:07] dmehus: how would temporary global groups help with that? [22:49:11] Not on beta [22:49:17] Like as a mediawiki feature [22:49:24] For things like GIPBE [22:49:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:49:38] You can get it perm on beta once it exists [22:49:59] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [22:50:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.54, 20.23, 19.16 [22:50:36] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:50:36] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:50:40] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:50:49] RhinosF1, oh temporary global groups, on production [22:50:51] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:50:52] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:50:55] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:50:56] I didn't know that was technically possible [22:51:09] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:10] but for `flood` and `global-ipblock-exempt`, yeah that could be useful [22:51:10] dmehus: it's coming in possibly 1.38 [22:51:17] RhinosF1, cool :) [22:51:19] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:20] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:51:23] That's why I found out about them scripts earlier [22:51:29] oh nice :) [22:51:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.58, 7.40, 7.87 [22:51:38] I wondered how much you'd use it [22:51:59] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:52:00] we don't get a lot of GIPBE requests, and can always remove it for users long inactive [22:52:07] but potentially a bit useful, yeah [22:52:34] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:52:36] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.580 second response time [22:52:36] If it's low volume then we don't need to run the CA script really ever [22:52:38] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.431 second response time [22:52:39] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.419 second response time [22:52:43] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:52:43] Like maybe every few years [22:52:52] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.287 second response time [22:52:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.63, 3.68 [22:52:56] Because 1 or 2 rows don't have much size impact [22:52:56] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.662 second response time [22:52:57] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.767 second response time [22:52:59] But yes [22:53:01] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:53:12] I will keep my eye out [22:53:13] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.613 second response time [22:53:16] Anyway I sleep now [22:53:22] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.466 second response time [22:53:26] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 7.566 second response time [22:54:04] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 9.479 second response time [22:54:09] okay, sounds good [22:54:16] won't ping you, since you're sleeping [22:54:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.65, 4.14, 3.87 [22:55:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:56:17] dmehus: don't rename a steward too [22:56:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.64, 19.45, 19.25 [22:56:26] that apparently doesn't work [22:56:38] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 3.200 second response time [22:56:38] without fixing global blocks manually [22:56:44] I'm sleeping now [22:56:49] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 2.721 second response time [22:56:55] No more leaning about central auth from recent patches [22:56:59] That's too much of a hole [22:57:11] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 4.584 second response time [22:58:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.38, 3.63, 3.71 [22:59:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:00:01] !log [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:03] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:05] !log [@mw11] starting deploy of {'l10nupdate': True} to ovlon [23:00:18] RhinosF1, oh, yeah, really? Interesting. I know a steward can't rename *themselves*, so I'd have to request another steward rename me, but you're saying a steward can't be renamed *at all*? [23:00:38] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.24, 3.77, 3.75 [23:01:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:31] dmehus: not without Sysadmin oversight [23:01:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.14, 8.50, 8.01 [23:01:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:45] It apparently breaks global blocks [23:02:01] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.86, 20.23, 18.95 [23:02:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:03:48] RhinosF1, oh, weird. Because the steward's username is used in the global block tables? [23:04:05] Yep [23:04:07] If so, I actually wondered how that would work, but assumed it'd just be updated [23:04:08] oh wow [23:04:13] any plans to fix that? [23:04:35] technically shouldn't be limited to stewards then either, no? [23:04:52] like I couldn't rename Reception123 or Raidarr if they requested it? [23:05:24] Good point [23:05:25] or really anyone with their username in the global block tables, so like Owen, too [23:05:38] err Owen (Miraheze)* [23:05:48] It failed once on WMF [23:05:52] oh, wow [23:06:00] I think it more of a race than a hard fail though [23:06:09] oh [23:06:10] Where it wasn't fully successful [23:06:15] You could do it [23:06:17] Not a race [23:06:24] so did they fix it with fixStuckGlobalRename.php? [23:06:28] But like it failed part way renaming [23:06:35] or manually repairing the global block tables? [23:06:35] No it just left dodgy rows [23:06:37] Rename worked [23:06:38] oh [23:06:47] There is now a script to repair the table [23:06:51] ah [23:06:56] what's that script called? [23:07:00] So if it ever needed to be done we should watch carefully [23:07:10] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GlobalBlocking/+/751992/6/maintenance/FixBlockerUsername.php [23:07:13] ah [23:07:21] interesting, thanks :) [23:07:29] Mediawiki is [23:07:32] that's included in our mediawiki repo, right? [23:07:35] since it's on MW 1.37 [23:07:55] No it'll be 1.38 but I'd just pull it if we needed it [23:08:15] Nothing would fail [23:08:22] oh [23:08:30] So it would be easy enough to apply ourselves [23:08:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.77, 3.78, 3.87 [23:08:35] yeah [23:08:43] script doesn't look overly complex either [23:08:48] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 4.63, 1.99, 0.81 [23:08:53] No it doesn't [23:09:21] * dmehus bets CosmicAlpha could probably write that if it didn't exist [23:09:32] Even Reception123 could [23:09:36] true [23:09:51] Reception123 added transclusion to {{Special:GlobalUser}} in MW 1.36! :D [23:09:51] https://meta.miraheze.org/wiki/Special:Template:GlobalUser [23:09:55] oops [23:10:02] {{Special:GlobalUsers}}* [23:10:02] https://meta.miraheze.org/wiki/Special:Template:GlobalUsers [23:10:08] lol [23:10:13] I hate how wm-bot does that [23:10:17] Tbh we could probably just run the update for us directly [23:10:28] yeah [23:10:39] If we had someone watching the DV [23:10:41] DB [23:13:00] !log [@mwtask111] finished deploy of {'l10nupdate': True} to scsvg - SUCCESS in 778s [23:13:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [23:13:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:13:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:14:27] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.88, 3.86 [23:16:25] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.024 second response time [23:16:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.89, 3.87 [23:16:47] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.11, 1.75, 1.32 [23:17:20] !log [@test101] finished deploy of {'l10nupdate': True} to skip - FAIL: [35072, 0, 0] in 1039s [23:17:37] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.20, 19.46, 19.78 [23:17:42] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:18:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:18:27] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:18:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.42, 4.06, 3.93 [23:18:47] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 0.60, 1.43, 1.26 [23:19:19] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:19:57] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:19:59] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:21:56] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 8.037 second response time [23:22:00] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 7.882 second response time [23:22:02] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 7.866 second response time [23:22:28] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 031_38-support [+0/-0/±1] 13https://git.io/J9Gtg [23:22:29] [02miraheze/RemovePII] 07Universal-Omega 03d38defb - Update SpecialRemovePII.php [23:22:31] [02RemovePII] 07Universal-Omega created branch 031_38-support - 13https://git.io/Jsfpj [23:22:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.13, 3.76, 3.85 [23:22:37] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 6.720 second response time [23:22:39] [02RemovePII] 07Universal-Omega opened pull request 03#41: 1.38 support - 13https://git.io/J9Gta [23:24:30] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 031_38-support [+0/-0/±1] 13https://git.io/J9Gtr [23:24:32] [02miraheze/RemovePII] 07Universal-Omega 03335955d - Update extension.json [23:24:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 3.97, 3.92 [23:24:33] [02RemovePII] 07Universal-Omega synchronize pull request 03#41: 1.38 support - 13https://git.io/J9Gta [23:25:31] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 6.291 second response time [23:25:39] miraheze/RemovePII - Universal-Omega the build has errored. [23:26:31] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:26:49] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:27:18] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:27:18] miraheze/RemovePII - Universal-Omega the build has errored. [23:27:23] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:27:29] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:27:33] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.47, 21.31, 20.19 [23:28:24] PROBLEM - test3 Current Load on test3 is CRITICAL: CRITICAL - load average: 5.04, 3.51, 1.89 [23:28:35] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.409 second response time [23:28:50] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.131 second response time [23:29:14] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.008 second response time [23:29:21] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 1.923 second response time [23:29:29] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.645 second response time [23:29:49] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:30:15] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:30:21] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:30:34] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:31:41] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:32:23] RECOVERY - test3 Current Load on test3 is OK: OK - load average: 2.28, 3.30, 2.21 [23:33:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:35:25] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 031_38-support [+0/-0/±1] 13https://git.io/J9GtH [23:35:27] [02miraheze/RemovePII] 07Universal-Omega 037193cbf - Update SpecialRemovePII.php [23:35:28] [02RemovePII] 07Universal-Omega synchronize pull request 03#41: 1.38 support - 13https://git.io/J9Gta [23:36:49] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 031_38-support [+0/-0/±1] 13https://git.io/J9GtQ [23:36:50] [02miraheze/RemovePII] 07Universal-Omega 03623fa54 - Update extension.json [23:36:52] [02RemovePII] 07Universal-Omega synchronize pull request 03#41: 1.38 support - 13https://git.io/J9Gta [23:36:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.26, 6.98, 7.87 [23:37:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.60, 6.80, 7.87 [23:37:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.10, 6.47, 8.00 [23:37:45] miraheze/RemovePII - Universal-Omega the build has errored. [23:38:40] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 2317s [23:39:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.13, 7.70, 8.06 [23:39:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 6.58, 7.04, 8.06 [23:39:16] miraheze/RemovePII - Universal-Omega the build has errored. [23:39:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb [23:39:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:40:59] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.50, 8.38, 8.22 [23:41:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.19, 6.96, 7.89 [23:42:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.79, 4.00 [23:43:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.47, 7.86, 8.11 [23:43:43] !log [@mw11] finished deploy of {'l10nupdate': True} to ovlon - SUCCESS in 2618s [23:44:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:44:18] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 9.131 second response time [23:44:25] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.578 second response time [23:44:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 4.10, 4.11 [23:46:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.19, 3.80, 4.00 [23:47:10] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.309 second response time [23:47:21] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.934 second response time [23:47:25] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.047 second response time [23:49:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:50:57] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.65, 21.21, 19.60 [23:52:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.40, 20.73, 19.61 [23:52:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/J9Gqt [23:52:56] [02miraheze/mw-config] 07Universal-Omega 038dc3f1e - DC-Switch: Switch to mon111 for IRC RecentChanges feed [23:52:57] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [23:53:01] [02mw-config] 07Universal-Omega opened pull request 03#4366: DC-Switch: Switch to mon111 for IRC RecentChanges feed - 13https://git.io/J9Gqq [23:53:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:54:04] miraheze/mw-config - Universal-Omega the build passed. [23:54:45] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.83, 20.02, 19.48 [23:58:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.09, 2.99, 3.39 [23:59:33] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.74, 18.83, 20.13