[00:01:32] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw9 [00:01:43] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw9 [00:01:48] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.713 second response time [00:01:55] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.72, 22.80, 20.72 [00:02:39] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:02:45] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:02:49] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:02:50] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:02:55] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23577 bytes in 4.750 second response time [00:02:58] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:02:59] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.38, 21.43, 20.62 [00:03:05] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.480 second response time [00:03:21] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.720 second response time [00:03:31] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.996 second response time [00:03:51] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.69, 22.36, 20.81 [00:04:13] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:04:17] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:04:18] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 136 failures. Last run 3 minutes ago with 136 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [00:04:26] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:04:32] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:04:40] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:04:42] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.670 second response time [00:04:51] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:04:53] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:55] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.432 second response time [00:05:28] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 18 backends are healthy [00:05:39] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 18 backends are healthy [00:06:29] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.291 second response time [00:06:30] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.075 second response time [00:06:41] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.088 second response time [00:06:54] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.638 second response time [00:06:55] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.189 second response time [00:08:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:08:57] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.320 second response time [00:09:16] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.355 second response time [00:09:25] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.981 second response time [00:09:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.77, 3.79, 3.46 [00:10:28] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.768 second response time [00:10:39] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.676 second response time [00:11:32] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.34, 19.75, 20.25 [00:12:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:13:16] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:28] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:13:34] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:53] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:16:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.98, 25.19, 22.67 [00:18:05] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:55] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:58] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:58] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.51, 5.33, 4.43 [00:21:13] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:21:23] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:21:31] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.377 second response time [00:21:44] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:21:57] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 3.950 second response time [00:22:05] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.587 second response time [00:22:13] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 5.602 second response time [00:22:18] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.561 second response time [00:23:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.93, 3.86 [00:26:31] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:26:38] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:26:40] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:26:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:26:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.80, 4.95, 4.58 [00:27:35] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.341 second response time [00:27:44] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.382 second response time [00:27:59] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.532 second response time [00:28:31] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.481 second response time [00:28:34] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.042 second response time [00:28:37] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.320 second response time [00:29:06] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.010 second response time [00:29:11] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.362 second response time [00:29:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.99, 3.90 [00:30:27] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [00:30:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:33:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.94, 3.82, 3.89 [00:34:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [00:34:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.79, 5.51, 4.92 [00:35:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.80, 4.34, 4.08 [00:36:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.87, 21.31, 23.20 [00:36:52] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.91, 4.79, 4.72 [00:37:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.17, 6.68, 7.99 [00:40:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:40:56] [02puppet] 07Universal-Omega opened pull request 03#2300: Fix shellbox - 13https://github.com/miraheze/puppet/pull/2300 [00:40:56] [url] Page not found · GitHub · GitHub | github.com [00:41:14] [02puppet] 07Universal-Omega synchronize pull request 03#2300: Fix shellbox - 13https://github.com/miraheze/puppet/pull/2300 [00:41:15] ... [00:41:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.55, 6.81, 7.67 [00:41:51] [02puppet] 07Universal-Omega synchronize pull request 03#2300: Fix shellbox - 13https://github.com/miraheze/puppet/pull/2300 [00:41:51] [url] Page not found · GitHub · GitHub | github.com [00:42:02] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.99, 20.59, 19.99 [00:42:07] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:42:19] [02puppet] 07Universal-Omega synchronize pull request 03#2300: Fix shellbox - 13https://github.com/miraheze/puppet/pull/2300 [00:42:20] [url] Page not found · GitHub · GitHub | github.com [00:43:58] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.28, 19.98, 19.83 [00:44:08] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.935 second response time [00:45:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.87, 3.84, 4.00 [00:46:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:46:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:47:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 4.03, 4.05 [00:48:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.65, 18.15, 20.26 [00:50:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [00:52:06] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.15, 20.48, 20.74 [00:52:51] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:53] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:57] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:53:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.89, 7.83, 7.98 [00:53:22] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:54:48] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.018 second response time [00:54:50] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.007 second response time [00:54:56] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.737 second response time [00:55:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.12, 7.95, 8.00 [00:55:22] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.339 second response time [00:55:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.85, 3.97 [00:56:12] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.311 second response time [00:57:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 4.06, 4.04 [00:59:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.85, 7.90, 7.96 [01:00:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:01:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.16, 8.27, 8.08 [01:04:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.46, 21.30, 20.80 [01:06:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.91, 19.97, 20.36 [01:07:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.48, 7.10, 7.71 [01:07:22] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb [01:08:02] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.27, 21.06, 19.98 [01:08:39] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 7716s [01:08:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:08:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:09:14] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:09:53] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:09:57] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.64, 19.70, 19.63 [01:10:21] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:11:13] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.337 second response time [01:11:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.39, 7.49, 7.70 [01:11:52] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.514 second response time [01:12:17] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.728 second response time [01:12:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.81, 23.05, 21.48 [01:12:40] !log [@mw11] finished deploy of {'l10nupdate': True} to ovlon - SUCCESS in 7958s [01:13:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:13:14] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:13:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.18, 7.32, 7.61 [01:14:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:15:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.69, 7.65, 7.68 [01:16:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.33, 7.40, 7.99 [01:17:40] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:18:40] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.84, 20.25, 19.93 [01:18:44] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.65, 8.11, 8.16 [01:19:59] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:20:28] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.11, 23.51, 22.02 [01:20:36] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:20:41] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:21:02] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:21:37] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:22] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.37, 4.79, 4.23 [01:22:25] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:22:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.85, 23.99, 22.41 [01:22:38] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:45] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:46] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:22:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb [01:22:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4f5e254499b4...73fe0a48f78b [01:22:54] [url] Comparing 4f5e254499b4...73fe0a48f78b · miraheze/mw-config · GitHub | github.com [01:22:55] [02miraheze/mw-config] 07Universal-Omega 0373fe0a4 - fix port for `$wgShellboxUrl` [01:23:04] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:23:14] !log [universalomega@test3] starting deploy of {'pull': 'config', 'config': True} to skip [01:23:19] !log [universalomega@test3] finished deploy of {'pull': 'config', 'config': True} to skip - SUCCESS in 4s [01:23:19] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:24:03] miraheze/mw-config - Universal-Omega the build passed. [01:24:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:25:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:17] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.97, 4.23, 4.15 [01:26:23] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.85, 20.33, 20.20 [01:26:38] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23579 bytes in 7.385 second response time [01:26:44] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.856 second response time [01:26:47] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.019 second response time [01:26:50] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.020 second response time [01:26:56] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.822 second response time [01:26:58] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 3.165 second response time [01:27:07] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.027 second response time [01:27:29] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.779 second response time [01:27:53] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.034 second response time [01:28:17] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.522 second response time [01:28:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/73fe0a48f78b...cdbe43e324e3 [01:28:23] [url] Comparing 73fe0a48f78b...cdbe43e324e3 · miraheze/mw-config · GitHub | github.com [01:28:24] [02miraheze/mw-config] 07Universal-Omega 03cdbe43e - use localhost for shellbox [01:28:43] !log [universalomega@test3] starting deploy of {'pull': 'config', 'config': True} to skip [01:28:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:28:52] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:29:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:29:36] !log [universalomega@test3] starting deploy of {'pull': 'config', 'config': True} to skip [01:29:37] !log [universalomega@test3] finished deploy of {'pull': 'config', 'config': True} to skip - SUCCESS in 0s [01:29:51] miraheze/mw-config - Universal-Omega the build passed. [01:30:24] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:30:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:30:29] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:30:50] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:31:10] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:31:17] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:31:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:32:13] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.13, 21.16, 20.51 [01:32:36] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.937 second response time [01:32:37] [02puppet] 07Universal-Omega synchronize pull request 03#2300: Fix shellbox - 13https://github.com/miraheze/puppet/pull/2300 [01:32:38] [url] Page not found · GitHub · GitHub | github.com [01:32:47] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:32:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 149.56.141.75/cpweb [01:32:53] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.021 second response time [01:33:10] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.653 second response time [01:33:18] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.315 second response time [01:34:09] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.36, 20.09, 20.21 [01:34:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.13, 6.83, 7.91 [01:34:27] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.207 second response time [01:34:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:34:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:36:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.64, 7.50, 8.02 [01:38:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.97, 18.90, 20.28 [01:39:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.31, 6.83, 7.73 [01:41:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.27, 7.35, 7.81 [01:41:23] !log [@mw11] starting deploy of {'config': True} to ovlon [01:41:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:41:51] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.64, 20.57, 20.26 [01:41:52] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 29s [01:42:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:42:50] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 51.195.220.68/cpweb [01:44:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.78, 21.77, 20.94 [01:45:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.64, 7.85, 7.98 [01:45:43] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.89, 20.18, 20.20 [01:46:52] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:47:01] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:26] !log [@test101] starting deploy of {'config': True} to skip [01:47:27] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:48:02] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:48:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:48:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:49:24] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:50:53] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:01] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:19] !log [@mwtask111] starting deploy of {'config': True} to scsvg [01:51:26] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 6s [01:51:41] !log [universalomega@test3] starting deploy of {'world': True} to skip [01:52:02] !log [universalomega@test101] starting deploy of {'world': True} to skip [01:52:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:52:13] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.702 second response time [01:52:22] !log [universalomega@mw11] starting deploy of {'world': True} to ovlon [01:52:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:52:45] !log [universalomega@mwtask111] starting deploy of {'world': True} to scsvg [01:52:50] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.325 second response time [01:52:56] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.012 second response time [01:53:00] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.448 second response time [01:53:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:53:16] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.189 second response time [01:53:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:53:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:54:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:54:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.54, 19.75, 20.32 [01:54:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:55:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.26, 8.05, 7.93 [01:55:21] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.28, 20.50, 20.21 [01:55:27] !log [universalomega@test101] finished deploy of {'world': True} to skip - SUCCESS in 205s [01:55:53] [02puppet] 07Universal-Omega synchronize pull request 03#2300: Fix shellbox - 13https://github.com/miraheze/puppet/pull/2300 [01:55:54] [url] Page not found · GitHub · GitHub | github.com [01:56:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:57:03] !log [universalomega@mwtask111] finished deploy of {'world': True} to scsvg - SUCCESS in 257s [01:57:12] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.195.220.68/cpweb [01:57:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:57:49] !log [universalomega@test3] finished deploy of {'world': True} to skip - SUCCESS in 367s [01:58:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:59:57] !log [universalomega@mw11] starting deploy of {'world': True} to ovlon [02:00:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:01:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.85, 7.67, 7.88 [02:01:47] !log [universalomega@mw11] finished deploy of {'world': True} to ovlon - SUCCESS in 565s [02:02:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:02:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:03:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:03:39] !log [universalomega@mw11] finished deploy of {'world': True} to ovlon - SUCCESS in 222s [02:04:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:05:14] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.87, 19.42, 20.00 [02:11:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.95, 7.82, 7.68 [02:12:01] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.88, 1.64, 1.22 [02:13:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:13:39] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:13:46] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:13:49] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:14:01] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.90, 1.68, 1.28 [02:14:27] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:41] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:14:51] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:15:46] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23579 bytes in 7.911 second response time [02:15:51] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.394 second response time [02:16:02] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.33, 1.55, 1.28 [02:18:23] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:18:25] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:18:37] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.830 second response time [02:21:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.98, 7.53, 7.70 [02:21:41] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:23:07] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:25:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.08, 7.55, 7.63 [02:26:38] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.134 second response time [02:26:50] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.218 second response time [02:27:13] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.255 second response time [02:27:14] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.403 second response time [02:27:35] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.819 second response time [02:30:50] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:31:01] [02puppet] 07Universal-Omega commented on pull request 03#2299: varnish: don't pass cookies to backend for load.php - 13https://github.com/miraheze/puppet/pull/2299#issuecomment-1010569537 [02:31:02] [url] varnish: don't pass cookies to backend for load.php by RhinosF1 · Pull Request #2299 · miraheze/puppet · GitHub | github.com [02:32:33] [02puppet] 07Universal-Omega commented on pull request 03#2299: varnish: don't pass cookies to backend for load.php - 13https://github.com/miraheze/puppet/pull/2299#issuecomment-1010570164 [02:32:33] [url] varnish: don't pass cookies to backend for load.php by RhinosF1 · Pull Request #2299 · miraheze/puppet · GitHub | github.com [02:32:42] [02puppet] 07Universal-Omega edited a comment on pull request 03#2299: varnish: don't pass cookies to backend for load.php - 13https://github.com/miraheze/puppet/pull/2299#issuecomment-1010570164 [02:32:42] [url] varnish: don't pass cookies to backend for load.php by RhinosF1 · Pull Request #2299 · miraheze/puppet · GitHub | github.com [02:33:33] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.04, 4.82, 3.72 [02:33:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.91, 20.54, 20.02 [02:34:22] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:34:56] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:35:19] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:35:30] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.77, 4.56, 3.75 [02:35:36] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.23, 20.17, 19.93 [02:35:45] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:35:46] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:36:11] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:36:57] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.867 second response time [02:37:18] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.408 second response time [02:37:41] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.112 second response time [02:37:41] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.653 second response time [02:38:11] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.007 second response time [02:38:17] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:38:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:42:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:42:35] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:42:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:43:23] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.59, 20.70, 20.16 [02:44:33] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23566 bytes in 0.574 second response time [02:44:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.63, 7.09, 7.91 [02:45:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.11, 20.32, 20.10 [02:46:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.28, 7.44, 7.93 [02:48:02] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:49:15] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.99, 20.44, 20.25 [02:51:11] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.06, 20.28, 20.21 [02:51:48] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:51:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.195.220.68/cpweb [02:53:00] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:53:04] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:53:29] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:53:34] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:59:23] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.701 second response time [02:59:25] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 9.644 second response time [02:59:47] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:59:50] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.392 second response time [02:59:51] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.238 second response time [02:59:52] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:59:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.63, 3.67, 3.97 [03:00:05] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:00:08] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:00:16] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:00:20] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 5.668 second response time [03:01:47] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.013 second response time [03:01:52] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.406 second response time [03:02:01] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.965 second response time [03:02:09] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23579 bytes in 3.356 second response time [03:02:15] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.798 second response time [03:02:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:03:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.67, 3.89 [03:05:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:09:40] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb [03:12:34] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.56, 20.16, 19.63 [03:12:59] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:13:00] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:13:03] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:13:50] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:13:55] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:14:58] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.348 second response time [03:15:02] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.541 second response time [03:15:06] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.581 second response time [03:15:51] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.758 second response time [03:15:57] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.020 second response time [03:16:27] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.84, 19.42, 19.49 [03:19:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.13, 3.82, 4.00 [03:21:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.34, 4.14, 4.10 [03:22:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [03:25:03] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:25:32] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:26:28] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:26:57] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:27:36] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.391 second response time [03:27:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.65, 3.71, 3.95 [03:28:28] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.970 second response time [03:28:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:29:00] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.657 second response time [03:29:13] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.102 second response time [03:31:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.05, 4.10, 4.05 [03:32:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:33:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.13, 3.35, 3.78 [03:35:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.58, 7.42, 7.96 [03:37:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.73, 3.84 [03:43:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.61, 7.91, 7.79 [03:47:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.14, 3.81, 3.90 [03:51:01] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:51:04] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:51:15] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:51:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.94, 3.93 [03:52:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [03:52:57] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.037 second response time [03:53:04] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.302 second response time [03:53:15] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.331 second response time [03:56:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:57:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.83, 7.60, 7.89 [03:59:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 3.90, 3.97 [04:04:24] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.70, 7.43, 7.97 [04:05:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.03, 4.20, 3.99 [04:07:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.91, 7.55, 7.59 [04:08:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.01, 8.05, 8.08 [04:11:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.81, 3.70, 3.87 [04:13:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.50, 7.86, 7.78 [04:14:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.12, 7.63, 7.97 [04:15:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.83, 7.40, 7.93 [04:17:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.67, 7.80, 8.00 [04:18:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.00, 8.18, 8.07 [04:19:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.85, 8.12, 7.88 [04:19:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.27, 3.80, 3.76 [04:22:12] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2607:5300:201:3100::5ebc/cpweb [04:22:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.59, 7.60, 7.88 [04:22:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.19, 19.77, 18.90 [04:24:09] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:24:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.59, 7.96, 7.97 [04:24:35] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.31, 19.36, 18.84 [04:25:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.83, 3.80 [04:26:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.12, 7.93, 7.97 [04:27:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.15, 7.57, 7.89 [04:27:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.47, 4.20, 3.94 [04:29:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.82, 3.84 [04:30:35] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.wikimicrofinanza.it All nameservers failed to answer the query. [04:33:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.02, 7.37, 7.68 [04:35:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.09, 6.80, 7.43 [04:36:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.17, 7.69, 7.78 [04:37:15] RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [04:38:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.15, 7.11, 7.56 [04:38:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.70, 6.33, 7.67 [04:39:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 2.93, 6.26, 7.83 [04:41:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.17, 3.58, 3.62 [04:42:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.65, 7.62, 7.63 [04:42:40] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:43:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.36, 3.09, 3.43 [04:44:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.11, 7.34, 7.52 [04:45:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.31, 7.49, 7.83 [04:46:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:47:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.56, 7.25, 7.69 [04:48:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.71, 7.81, 7.58 [04:49:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.89, 7.33, 7.24 [04:51:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.17, 7.38, 7.88 [04:53:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.21, 7.99, 8.03 [04:53:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.98, 7.83, 7.77 [04:55:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.59, 7.76, 7.55 [04:55:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.62, 7.45, 7.65 [04:55:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.72, 4.09, 3.77 [04:56:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.86, 7.03, 7.10 [04:59:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.23, 7.64, 7.66 [05:01:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.20, 7.95, 7.65 [05:03:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.54, 7.16, 7.41 [05:03:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.92, 3.83 [05:05:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.47, 6.86, 7.86 [05:06:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.90, 6.97, 7.82 [05:07:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.83, 3.79 [05:10:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.40, 7.09, 7.61 [05:11:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.39, 7.24, 7.24 [05:11:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.32, 7.43, 7.66 [05:13:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.85, 7.48, 7.33 [05:13:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.25, 7.36, 7.61 [05:15:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.26, 3.98, 3.96 [05:17:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.19, 4.55, 4.18 [05:17:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.18, 7.56, 7.63 [05:19:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.36, 7.67, 7.41 [05:21:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.86, 3.96 [05:21:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.78, 7.79, 7.72 [05:23:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.85, 7.63, 7.49 [05:23:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.64, 4.20, 4.08 [05:23:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.70, 8.10, 7.85 [05:25:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.88, 3.73, 3.92 [05:25:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.92, 7.72, 7.75 [05:27:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.92, 8.06, 7.86 [05:29:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.28, 4.10, 4.03 [05:33:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.77, 3.92 [05:35:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.65, 3.91, 3.94 [05:36:44] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.64, 7.13, 7.87 [05:37:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.60, 7.17, 7.75 [05:39:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.52, 5.71, 6.63 [05:41:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:41:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.88, 3.95 [05:42:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.15, 6.45, 7.98 [05:43:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.54, 7.46, 8.00 [05:43:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.93, 3.95 [05:43:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.10, 7.18, 7.52 [05:45:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.03, 6.97, 7.96 [05:45:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.18, 3.68, 3.86 [05:45:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.38, 6.90, 7.39 [05:47:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.52, 7.71, 7.94 [05:48:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:49:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.43, 6.99, 6.83 [05:49:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.39, 3.68, 3.81 [05:49:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.21, 8.33, 7.82 [05:50:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.26, 7.95, 7.60 [05:51:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.73, 6.45, 6.67 [05:51:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.87, 7.64, 7.88 [05:51:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.42, 3.30, 3.66 [05:52:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.80, 7.73, 7.56 [05:53:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.46, 7.65, 7.69 [05:54:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.73, 5.92, 6.80 [05:55:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.02, 7.76, 7.79 [05:55:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.34, 2.83, 3.37 [05:55:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.94, 8.10, 7.85 [05:58:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.77, 6.52, 6.87 [05:58:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:59:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.49, 7.28, 7.00 [05:59:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.41, 3.49 [05:59:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.12, 7.75, 7.84 [06:01:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.42, 7.60, 7.14 [06:01:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.37, 3.00, 3.33 [06:03:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.04, 7.37, 7.42 [06:03:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.48, 7.11, 7.02 [06:03:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:05:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.25, 7.15, 7.35 [06:07:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.15, 7.65, 7.51 [06:07:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.52, 6.26, 6.72 [06:08:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:09:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.92, 7.87, 7.62 [06:11:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.04, 8.43, 7.84 [06:11:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:11:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.55, 6.65, 7.10 [06:12:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.51, 8.72, 7.64 [06:12:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.10, 7.34, 7.12 [06:13:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.37, 7.96, 7.14 [06:14:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.81, 7.57, 7.24 [06:15:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.57, 7.52, 7.09 [06:15:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.60, 6.74, 7.11 [06:17:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.60, 3.47 [06:19:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.86, 3.58 [06:19:59] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.18, 7.17, 7.12 [06:20:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.07, 8.38, 7.61 [06:21:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.09, 7.37, 7.11 [06:22:15] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:22:24] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:22:37] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [06:22:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [06:22:55] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:23:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.30, 7.53, 7.22 [06:23:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.44, 6.85, 7.07 [06:24:19] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.580 second response time [06:24:32] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.289 second response time [06:24:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.17, 7.05, 7.29 [06:25:00] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.545 second response time [06:26:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:27:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.25, 7.21, 7.12 [06:28:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:29:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.92, 7.22, 7.15 [06:31:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:31:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.18, 7.62, 7.30 [06:32:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.00, 7.81, 7.35 [06:33:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.58, 7.33, 7.13 [06:33:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.81, 3.88 [06:34:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.31, 7.52, 7.30 [06:35:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.37, 7.17, 7.09 [06:36:50] [02puppet] 07Reception123 closed pull request 03#2300: Fix shellbox - 13https://github.com/miraheze/puppet/pull/2300 [06:36:51] [url] Page not found · GitHub · GitHub | github.com [06:36:52] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/50da6a6543dc...089ef529b260 [06:36:52] [url] Comparing 50da6a6543dc...089ef529b260 · miraheze/puppet · GitHub | github.com [06:36:53] [02miraheze/puppet] 07Universal-Omega 03089ef52 - Fix shellbox (#2300) [06:37:05] Thanks Reception123! [06:37:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.33, 6.35, 6.79 [06:37:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:37:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.06, 7.40, 7.43 [06:38:19] CosmicAlpha: no problem [06:38:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.03, 7.12, 7.78 [06:41:59] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.46, 7.79, 7.53 [06:42:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:43:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.49, 7.22, 7.95 [06:43:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.00, 7.64, 7.11 [06:45:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.12, 7.43, 7.11 [06:46:11] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=joeyslikk --username-prefix joeyslikk --report 1 --no-updates /home/reception/Pages_Categories_for_Joey_Slikk_4th_New_Wiki.xml (START) [06:46:12] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=joeyslikk --username-prefix joeyslikk --report 1 --no-updates /home/reception/Pages_Categories_for_Joey_Slikk_4th_New_Wiki.xml (END - exit=65280) [06:46:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:46:21] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=joeyslikkwiki --username-prefix joeyslikk --report 1 --no-updates /home/reception/Pages_Categories_for_Joey_Slikk_4th_New_Wiki.xml (START) [06:46:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.68, 7.20, 7.42 [06:46:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:46:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:46:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:46:53] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=joeyslikkwiki --username-prefix joeyslikk --report 1 --no-updates /home/reception/Pages_Categories_for_Joey_Slikk_4th_New_Wiki.xml (END - exit=0) [06:47:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:47:05] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.33, 7.74, 7.97 [06:47:42] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=joeyslikkwiki (START) [06:47:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:48:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.41, 6.95, 7.31 [06:49:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.99, 7.88, 8.00 [06:49:45] Reception123: that interwiki URL is broken; see -interwiki-feed [06:49:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.56, 6.96, 7.87 [06:50:09] can you pause the import, Reception123? [06:50:20] and fix the interwiki URL and restart it? [06:50:38] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=teamballswiki --username-prefix wikia:teamballs --report 1 --no-updates /home/reception/esteamballs_pages_full.xml%281%29_%281%29.xml (START) [06:50:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:51:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.88, 8.47, 8.22 [06:51:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.10, 6.48, 6.78 [06:51:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.01, 4.25, 3.93 [06:52:05] dmehus: unfortunately as you can see above it says (EXIT) meaning it's done :( [06:52:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.69, 7.41, 7.39 [06:53:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.09, 7.92, 8.00 [06:54:24] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.68, 7.31, 7.35 [06:55:01] Reception123: ack; we continued this in -interwiki-feed anyway [06:55:08] yeah [06:55:10] lol Voidwalker's quit message [06:55:16] * dmehus should be asleep too [06:58:05] lol, a lot of people should be asleep right now [06:58:41] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=joeyslikkwiki (END - exit=0) [06:58:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:59:07] Agent: except those who just woke up :P [06:59:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.96, 3.94 [07:00:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.58, 7.25, 7.08 [07:00:59] Reception123: you who woke up could also use a little extra sleep :) [07:01:06] heh :D [07:01:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.45, 6.96, 7.59 [07:01:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.60, 4.41, 4.10 [07:01:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.69, 7.23, 7.87 [07:02:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.76, 7.78, 7.45 [07:02:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.69, 7.19, 7.10 [07:03:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.17, 3.94, 3.96 [07:04:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.34, 7.85, 7.53 [07:06:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.28, 8.25, 7.71 [07:06:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.58, 6.16, 6.67 [07:07:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.01, 7.33, 7.98 [07:09:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.73, 7.84, 8.08 [07:09:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.18, 5.44, 6.75 [07:10:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.83, 7.76, 7.65 [07:11:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.42, 7.31, 7.86 [07:11:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:12:33] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:13:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.66, 3.88, 3.81 [07:13:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.12, 7.05, 7.09 [07:14:59] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=teamballswiki --username-prefix wikia:teamballs --report 1 --no-updates /home/reception/esteamballs_pages_full.xml%281%29_%281%29.xml (END - exit=0) [07:15:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.14, 7.56, 7.83 [07:15:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:15:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.32, 6.86, 6.49 [07:15:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.73, 3.76 [07:16:50] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=teamballswiki (START) [07:16:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:17:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.70, 7.43, 7.74 [07:17:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.35, 6.70, 6.48 [07:17:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:17:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.26, 7.11, 7.11 [07:17:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.45, 7.31, 7.25 [07:19:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.93, 8.24, 7.59 [07:21:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.23, 7.42, 7.66 [07:21:20] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.11, 7.64, 6.87 [07:21:27] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.57, 8.53, 7.45 [07:21:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.68, 3.69 [07:22:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.25, 7.87, 7.48 [07:23:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.17, 7.39, 7.63 [07:23:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.68, 6.87, 6.68 [07:23:21] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.45, 7.91, 7.36 [07:23:26] [02puppet] 07RhinosF1 commented on pull request 03#2299: varnish: don't pass cookies to backend for load.php - 13https://github.com/miraheze/puppet/pull/2299#issuecomment-1010719347 [07:23:27] [url] varnish: don't pass cookies to backend for load.php by RhinosF1 · Pull Request #2299 · miraheze/puppet · GitHub | github.com [07:23:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.73, 3.59, 3.65 [07:23:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.61, 7.93, 7.64 [07:24:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.00, 7.38, 7.36 [07:24:35] [02puppet] 07Universal-Omega commented on pull request 03#2299: varnish: don't pass cookies to backend for load.php - 13https://github.com/miraheze/puppet/pull/2299#issuecomment-1010720109 [07:24:35] [url] varnish: don't pass cookies to backend for load.php by RhinosF1 · Pull Request #2299 · miraheze/puppet · GitHub | github.com [07:25:00] [02puppet] 07Universal-Omega edited a comment on pull request 03#2299: varnish: don't pass cookies to backend for load.php - 13https://github.com/miraheze/puppet/pull/2299#issuecomment-1010720109 [07:25:01] ... [07:25:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.88, 8.09, 7.86 [07:25:55] [02puppet] 07RhinosF1 commented on pull request 03#2299: varnish: don't pass cookies to backend for load.php - 13https://github.com/miraheze/puppet/pull/2299#issuecomment-1010721130 [07:25:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.63, 3.64 [07:25:55] ... [07:25:59] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.60, 8.14, 7.74 [07:27:16] [02puppet] 07Universal-Omega commented on pull request 03#2299: varnish: don't pass cookies to backend for load.php - 13https://github.com/miraheze/puppet/pull/2299#issuecomment-1010722037 [07:27:16] [url] varnish: don't pass cookies to backend for load.php by RhinosF1 · Pull Request #2299 · miraheze/puppet · GitHub | github.com [07:27:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.10, 6.49, 6.57 [07:27:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.17, 7.94, 7.67 [07:27:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.87, 3.34, 3.54 [07:29:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.48, 7.76, 7.80 [07:29:05] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.34, 7.80, 7.43 [07:30:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.37, 7.39, 7.33 [07:30:59] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.05, 7.56, 7.39 [07:31:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.95, 7.22, 6.83 [07:31:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.65, 7.66, 7.55 [07:31:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.10, 3.04, 3.36 [07:31:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.75, 7.30, 7.60 [07:33:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.67, 8.43, 8.04 [07:33:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.40, 7.61, 7.02 [07:34:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.02, 7.49, 7.41 [07:35:48] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.74, 7.97, 7.73 [07:36:53] [02puppet] 07Universal-Omega opened pull request 03#2301: shellbox: ensure `/var/tmp/shellbox` exists - 13https://github.com/miraheze/puppet/pull/2301 [07:36:54] [url] Page not found · GitHub · GitHub | github.com [07:37:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.55, 7.91, 7.93 [07:37:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.84, 8.37, 7.92 [07:37:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.51, 7.75, 7.72 [07:38:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.97, 8.60, 7.88 [07:41:46] !log delete personal me from search [07:41:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:44:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.77, 3.52 [07:45:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.66, 7.78, 7.88 [07:46:48] [02puppet] 07Universal-Omega synchronize pull request 03#2301: shellbox: ensure `/var/tmp/shellbox` exists - 13https://github.com/miraheze/puppet/pull/2301 [07:46:49] [url] Page not found · GitHub · GitHub | github.com [07:47:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.19, 6.93, 7.34 [07:47:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.14, 8.00, 7.77 [07:47:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.84, 8.35, 8.08 [07:47:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.28, 7.78, 7.95 [07:48:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.07, 3.46, 3.46 [07:49:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.02, 6.47, 7.12 [07:49:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.05, 8.73, 8.06 [07:49:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.75, 8.45, 8.17 [07:50:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.64, 3.12, 3.33 [07:51:26] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [07:51:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.41, 7.21, 7.70 [07:52:25] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.97, 6.40, 6.80 [07:53:03] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.25, 6.88, 7.09 [07:53:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:53:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.52, 7.90, 7.90 [07:54:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.45, 7.91, 8.00 [07:56:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 7.93, 8.01, 8.03 [07:57:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.36, 7.39, 7.26 [07:58:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.44, 7.15, 7.72 [07:59:19] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.19, 6.87, 6.89 [07:59:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.61, 7.33, 7.72 [08:01:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.15, 7.48, 7.29 [08:01:18] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.28, 6.57, 6.77 [08:01:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.27, 7.81, 7.85 [08:02:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/0891a0b99235 [08:02:20] [02miraheze/mw-config] 07Universal-Omega 030891a0b - Set score configs [08:02:22] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-5 - 13https://github.com/miraheze/mw-config [08:02:22] [url] Page not found · GitHub · GitHub | github.com [08:02:23] [02mw-config] 07Universal-Omega opened pull request 03#4374: Set score configs - 13https://github.com/miraheze/mw-config/pull/4374 [08:02:24] [url] Page not found · GitHub · GitHub | github.com [08:03:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.54, 7.02, 7.87 [08:03:31] miraheze/mw-config - Universal-Omega the build passed. [08:04:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.34, 3.28 [08:05:14] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.05, 7.35, 7.09 [08:05:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.98, 7.65, 7.81 [08:06:43] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0891a0b99235...1bf34c87a8b1 [08:06:44] [url] Comparing 0891a0b99235...1bf34c87a8b1 · miraheze/mw-config · GitHub | github.com [08:06:45] [02miraheze/mw-config] 07Universal-Omega 031bf34c8 - Update LocalSettings.php [08:06:46] [02mw-config] 07Universal-Omega synchronize pull request 03#4374: Set score configs - 13https://github.com/miraheze/mw-config/pull/4374 [08:06:47] [url] Page not found · GitHub · GitHub | github.com [08:06:50] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.58, 2.73, 3.06 [08:07:12] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.28, 7.76, 7.27 [08:07:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:07:49] miraheze/mw-config - Universal-Omega the build passed. [08:09:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.21, 7.61, 7.53 [08:09:11] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 8.00, 7.59, 7.25 [08:09:16] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=teamballswiki (END - exit=0) [08:09:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.45, 6.98, 7.48 [08:09:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:09:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.45, 7.83, 7.79 [08:09:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1bf34c87a8b1...609753dd6c7a [08:09:54] [url] Comparing 1bf34c87a8b1...609753dd6c7a · miraheze/mw-config · GitHub | github.com [08:09:55] [02miraheze/mw-config] 07Universal-Omega 03609753d - Update LocalSettings.php [08:09:57] [02mw-config] 07Universal-Omega synchronize pull request 03#4374: Set score configs - 13https://github.com/miraheze/mw-config/pull/4374 [08:09:57] ... [08:10:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/609753dd6c7a...9b1723e92456 [08:10:22] [url] Comparing 609753dd6c7a...9b1723e92456 · miraheze/mw-config · GitHub | github.com [08:10:23] [02miraheze/mw-config] 07Universal-Omega 039b1723e - Update Defines.php [08:10:25] [02mw-config] 07Universal-Omega synchronize pull request 03#4374: Set score configs - 13https://github.com/miraheze/mw-config/pull/4374 [08:10:25] [url] Page not found · GitHub · GitHub | github.com [08:10:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:10:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.22, 5.95, 6.79 [08:11:00] miraheze/mw-config - Universal-Omega the build passed. [08:11:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.85, 8.43, 7.84 [08:11:10] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.55, 7.98, 7.44 [08:11:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.97, 7.05, 7.43 [08:11:31] miraheze/mw-config - Universal-Omega the build passed. [08:11:41] [02mw-config] 07Universal-Omega edited pull request 03#4374: Re-enable score with shellbox - 13https://github.com/miraheze/mw-config/pull/4374 [08:11:41] [url] Page not found · GitHub · GitHub | github.com [08:11:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.66, 7.44, 7.99 [08:13:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.25, 7.41, 7.55 [08:13:08] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.04, 7.58, 7.39 [08:13:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9b1723e92456...499036447d16 [08:13:20] [url] Comparing 9b1723e92456...499036447d16 · miraheze/mw-config · GitHub | github.com [08:13:21] [02miraheze/mw-config] 07Universal-Omega 034990364 - Update ManageWikiExtensions.php [08:13:23] [02mw-config] 07Universal-Omega synchronize pull request 03#4374: Re-enable score with shellbox - 13https://github.com/miraheze/mw-config/pull/4374 [08:13:23] [url] Page not found · GitHub · GitHub | github.com [08:13:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.53, 7.85, 7.87 [08:14:38] miraheze/mw-config - Universal-Omega the build passed. [08:15:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:17:05] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.50, 6.02, 6.79 [08:19:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.53, 5.83, 6.73 [08:19:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.34, 8.41, 8.15 [08:21:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.16, 6.78, 7.01 [08:21:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:21:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.49, 7.32, 7.40 [08:22:25] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostnam [08:22:26] t_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1089, in query return self.resolve(qname, rdtype, rdclass, tcp, source, File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifet [08:22:26] ile "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407216548919678 seconds [08:22:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.15, 7.75, 7.02 [08:23:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.78, 7.26, 7.17 [08:23:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.91, 3.83, 3.25 [08:23:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.95, 6.59, 6.89 [08:23:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.89, 7.29, 7.39 [08:24:44] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.82, 7.74, 7.10 [08:25:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.05, 3.43, 3.17 [08:25:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.21, 8.19, 7.51 [08:25:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.69, 1.80, 1.10 [08:26:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.21, 3.29, 3.15 [08:27:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.12, 6.88, 7.11 [08:27:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.76, 1.51, 1.09 [08:31:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:32:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.49, 6.44, 6.77 [08:33:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.79, 6.22, 6.74 [08:36:50] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.85, 6.34, 6.25 [08:37:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.94, 7.12, 6.97 [08:37:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.20, 6.22, 6.67 [08:37:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.02, 7.07, 7.04 [08:39:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:39:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.67, 6.87, 6.97 [08:41:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.11, 6.37, 6.73 [08:42:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.69, 7.10, 6.91 [08:42:47] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/nagios [08:42:47] ns/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - diamowiki.ga reverse DNS resolves to cp20.miraheze.org - CNAME OK [08:43:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.08, 7.34, 7.93 [08:44:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.06, 7.40, 7.04 [08:44:45] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.09, 5.83, 6.12 [08:46:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.40, 6.87, 6.88 [08:47:46] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.86, 6.36, 6.75 [08:47:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.31, 7.74, 7.94 [08:48:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.42, 7.41, 7.07 [08:49:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:51:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.28, 7.50, 6.98 [08:51:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.28, 6.93, 6.91 [08:52:09] [02puppet] 07Universal-Omega opened pull request 03#2302: shellbox: use composer.phar - 13https://github.com/miraheze/puppet/pull/2302 [08:52:10] [url] Page not found · GitHub · GitHub | github.com [08:53:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.10, 7.21, 6.96 [08:53:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.41, 4.17, 3.50 [08:54:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:55:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.04, 7.16, 6.69 [08:55:47] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.83, 7.21, 6.95 [08:56:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.36, 7.79, 7.45 [08:57:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.82, 6.31, 6.67 [08:57:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.27, 7.07, 6.70 [08:57:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.83, 3.53 [08:59:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.73, 7.48, 6.89 [09:01:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.39, 7.06, 6.80 [09:01:46] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.70, 6.43, 6.73 [09:03:31] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.47, 6.00, 6.64 [09:03:31] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.52, 6.79, 6.73 [09:04:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:07:27] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.96, 6.90, 6.82 [09:07:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:08:01] [02puppet] 07Reception123 closed pull request 03#2301: shellbox: ensure `/var/tmp/shellbox` exists - 13https://github.com/miraheze/puppet/pull/2301 [09:08:02] [url] Page not found · GitHub · GitHub | github.com [09:08:03] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/089ef529b260...b7d733c9e9db [09:08:04] [url] Comparing 089ef529b260...b7d733c9e9db · miraheze/puppet · GitHub | github.com [09:08:04] [02miraheze/puppet] 07Universal-Omega 03b7d733c - shellbox: ensure `/var/tmp/shellbox` exists (#2301) [09:08:11] [02puppet] 07Reception123 closed pull request 03#2302: shellbox: use composer.phar - 13https://github.com/miraheze/puppet/pull/2302 [09:08:12] ... [09:08:13] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b7d733c9e9db...efbd2df0efc4 [09:08:13] [url] Comparing b7d733c9e9db...efbd2df0efc4 · miraheze/puppet · GitHub | github.com [09:08:14] [02miraheze/puppet] 07Universal-Omega 03efbd2df - shellbox: use composer.phar (#2302) [09:08:39] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.20, 7.13, 6.97 [09:09:26] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.25, 6.74, 6.77 [09:10:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.14, 6.97, 7.06 [09:11:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.80, 2.92, 3.31 [09:15:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.00, 6.72, 6.54 [09:18:29] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.23, 6.39, 6.73 [09:19:46] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.32, 7.43, 6.85 [09:21:41] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.37, 7.24, 6.84 [09:23:36] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.16, 7.93, 7.14 [09:27:12] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.00, 6.61, 6.36 [09:27:16] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.15, 6.90, 6.70 [09:27:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:29:11] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.37, 6.72, 6.44 [09:29:13] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.17, 6.64, 6.64 [09:29:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.12, 7.43, 7.26 [09:30:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.48, 7.53, 7.92 [09:35:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.93, 6.00, 6.69 [09:37:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.57, 6.40, 7.74 [09:41:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.29, 7.65, 7.85 [09:43:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.68, 7.47, 6.99 [09:43:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.94, 7.56, 7.79 [09:45:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.83, 7.18, 6.94 [09:45:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.58, 8.38, 8.07 [09:47:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.92, 7.60, 7.82 [09:48:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.28, 7.22, 7.12 [09:49:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.54, 7.88, 7.25 [09:49:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.65, 8.68, 8.18 [09:50:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.11, 7.59, 7.30 [09:51:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.04, 7.47, 7.17 [09:51:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.64, 7.95, 7.98 [09:52:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.02, 8.16, 7.54 [09:53:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.61, 7.52, 6.73 [09:53:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:54:50] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/na [09:54:50] lugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main [09:54:50] e "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1089, in query return self.resolve(qname, rdtype, rdclass, tcp, source, File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in res [09:54:50] imeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.408265829086304 seconds [09:55:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.85, 7.97, 6.99 [09:55:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.41, 7.64, 7.28 [09:55:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.27, 8.46, 8.16 [09:57:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.26, 7.83, 7.06 [09:57:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.26, 8.03, 6.99 [09:58:48] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.59, 7.45, 6.74 [09:59:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.22, 7.99, 7.21 [10:01:37] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/nagios [10:01:37] ns/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - diamowiki.ga reverse DNS resolves to cp21.miraheze.org - CNAME OK [10:02:47] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.99, 7.51, 6.96 [10:03:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.39, 7.25, 7.10 [10:06:46] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.21, 6.72, 6.75 [10:07:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.08, 7.59, 7.44 [10:09:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.70, 7.78, 7.97 [10:11:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.05, 7.52, 7.42 [10:11:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.24, 7.88, 7.98 [10:13:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.04, 6.77, 7.15 [10:13:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.51, 7.86, 7.97 [10:13:46] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.56, 6.19, 6.64 [10:15:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.29, 7.52, 7.38 [10:17:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.35, 7.22, 7.28 [10:17:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.86, 7.17, 6.99 [10:18:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.70, 7.55, 7.96 [10:19:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.90, 7.66, 7.42 [10:19:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.76, 7.67, 7.81 [10:19:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.83, 7.65, 7.18 [10:20:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.40, 8.24, 8.16 [10:21:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.65, 7.37, 7.69 [10:21:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.39, 7.51, 7.18 [10:23:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.19, 7.27, 6.96 [10:23:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.63, 8.23, 7.50 [10:25:30] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.49, 7.26, 6.99 [10:25:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.40, 7.72, 7.41 [10:27:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.16, 7.37, 7.51 [10:29:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.97, 7.87, 7.72 [10:29:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.22, 6.94, 7.34 [10:31:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.51, 7.92, 7.74 [10:31:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.11, 7.80, 7.44 [10:33:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.72, 7.57, 7.65 [10:33:24] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.30, 7.24, 6.99 [10:33:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:34:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.87, 7.22, 7.93 [10:35:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.64, 6.83, 6.87 [10:35:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.12, 7.11, 7.96 [10:37:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.92, 6.52, 6.75 [10:37:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.12, 7.13, 7.39 [10:39:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.88, 6.14, 6.76 [10:39:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.21, 2.80 [10:41:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.64, 3.05, 2.79 [10:45:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.04, 6.02, 6.80 [10:48:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.42, 7.95, 7.66 [10:49:03] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.64, 7.00, 7.00 [10:49:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.92, 8.18, 7.55 [10:49:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.40, 7.65, 7.39 [10:50:03] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.77, 7.08, 6.90 [10:50:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:51:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.41, 7.07, 7.02 [10:53:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.74, 7.67, 7.49 [10:55:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:55:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.33, 8.29, 7.73 [10:57:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.48, 6.11, 6.71 [10:57:05] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.47, 6.86, 6.72 [10:57:41] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.40, 7.75, 7.26 [10:59:04] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.58, 6.65, 6.65 [10:59:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.00, 7.50, 7.23 [11:01:30] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.44, 7.91, 7.42 [11:03:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.18, 6.48, 6.60 [11:04:59] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.10, 7.13, 6.85 [11:05:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.40, 7.86, 7.52 [11:05:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.16, 7.90, 7.94 [11:06:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:06:57] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.96, 6.67, 6.71 [11:07:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.98, 7.49, 6.98 [11:08:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.75, 7.56, 7.99 [11:09:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.95, 7.39, 7.02 [11:09:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.16, 7.78, 7.54 [11:10:53] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.29, 7.29, 6.98 [11:11:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.87, 8.11, 7.34 [11:11:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.80, 7.58, 7.48 [11:11:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.40, 7.56, 7.73 [11:12:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.54, 8.02, 8.03 [11:13:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.73, 7.65, 7.27 [11:13:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.18, 7.71, 7.53 [11:13:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.32, 7.54, 7.71 [11:14:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.12, 7.78, 7.95 [11:15:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.93, 7.56, 7.49 [11:15:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.81, 7.64, 7.94 [11:16:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:16:49] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.66, 7.52, 7.18 [11:17:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.17, 7.37, 7.20 [11:17:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.27, 7.81, 7.96 [11:17:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.10, 8.09, 7.87 [11:18:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.03, 7.59, 7.83 [11:19:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.95, 7.68, 7.34 [11:19:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:20:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.60, 7.94, 7.95 [11:20:47] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.60, 7.56, 7.31 [11:21:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.80, 7.55, 7.91 [11:24:45] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.04, 7.36, 7.24 [11:26:45] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.84, 7.47, 7.29 [11:27:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.44, 8.40, 7.76 [11:28:45] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.70, 8.08, 7.55 [11:29:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:29:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.34, 8.62, 8.16 [11:33:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:36:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.64, 7.84, 7.71 [11:37:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.27, 7.99, 7.96 [11:38:38] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.62, 7.48, 7.66 [11:39:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.30, 7.93, 7.39 [11:40:37] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.91, 8.16, 7.87 [11:41:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.89, 8.64, 8.20 [11:41:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.89, 7.81, 7.41 [11:42:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.02, 7.89, 7.82 [11:44:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.54, 8.22, 7.95 [11:49:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.12, 8.07, 7.59 [11:50:30] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.61, 8.00, 8.00 [11:51:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.61, 7.37, 7.39 [11:57:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.36, 8.70, 7.95 [11:57:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.48, 7.15, 7.90 [12:01:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.95, 8.02, 8.04 [12:02:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.22, 7.62, 7.58 [12:03:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.89, 7.96, 7.94 [12:04:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.45, 7.18, 7.43 [12:06:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.32, 7.70, 7.56 [12:08:44] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.98, 6.60, 7.85 [12:13:14] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.60, 20.04, 18.72 [12:14:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.65, 7.30, 7.68 [12:15:13] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 20.09, 20.10, 18.91 [12:15:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.83, 6.76, 7.82 [12:17:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.59, 7.60, 8.00 [12:21:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.55, 3.12 [12:21:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.75, 7.43, 7.91 [12:23:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.67, 3.22 [12:24:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.14, 7.34, 7.80 [12:25:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.17, 5.72, 6.60 [12:25:45] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:25:50] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:50] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:25:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.15, 3.95, 3.36 [12:25:56] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:27:31] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [12:27:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.38, 3.61, 3.30 [12:28:23] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:29:49] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.946 second response time [12:29:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.80, 3.41 [12:29:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.17, 7.67, 7.62 [12:30:03] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.303 second response time [12:30:03] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.553 second response time [12:30:08] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.050 second response time [12:30:21] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.054 second response time [12:31:20] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:31:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.86, 7.60, 7.60 [12:32:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.94, 7.97, 7.73 [12:33:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.48, 8.44, 7.92 [12:35:14] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::5ebc/cpweb [12:35:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.47, 7.01, 6.72 [12:35:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.75, 3.54 [12:36:18] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:36:41] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:37:01] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:37:11] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:37:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.09, 7.17, 6.80 [12:38:46] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.586 second response time [12:39:00] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.697 second response time [12:39:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.65, 7.17, 6.86 [12:39:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.63, 3.04, 3.34 [12:40:27] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 1.326 second response time [12:41:05] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [12:41:58] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:42:42] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:43:30] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:43:33] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:44:06] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.191 second response time [12:44:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.59, 6.93, 7.98 [12:44:46] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.332 second response time [12:45:35] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.936 second response time [12:45:38] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.722 second response time [12:45:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb [12:46:57] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:49:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.35, 7.82, 7.32 [12:51:40] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:51:55] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:51:58] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:52:02] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:52:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [12:53:11] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:56:08] RECOVERY - wiki.fbpml.org - reverse DNS on sslhost is OK: SSL OK - wiki.fbpml.org reverse DNS resolves to cp20.miraheze.org - CNAME OK [12:56:45] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:59:23] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.563 second response time [13:00:11] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.127 second response time [13:01:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.93, 6.92, 7.94 [13:02:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.70, 7.81, 7.65 [13:02:24] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.248 second response time [13:02:29] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.713 second response time [13:02:31] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.880 second response time [13:03:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.05, 7.92, 8.17 [13:03:03] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:05:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.21, 7.13, 7.85 [13:09:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.14, 7.22, 7.69 [13:10:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.47, 7.90, 7.87 [13:11:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.26, 6.61, 7.45 [13:11:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 3.48, 6.57, 7.65 [13:11:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.33, 6.75, 7.64 [13:12:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 7.28, 8.04, 7.95 [13:14:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.82, 7.64, 7.81 [13:15:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.49, 7.54, 7.62 [13:15:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.16, 6.83, 7.47 [13:16:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.61, 7.96, 7.91 [13:17:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.38, 7.30, 7.53 [13:18:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.51, 7.83, 7.87 [13:20:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [13:20:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.72, 8.55, 8.13 [13:21:23] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [13:22:42] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:22:47] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:23:46] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:24:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.46, 7.89, 7.99 [13:24:49] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.157 second response time [13:24:53] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.097 second response time [13:24:57] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:25:49] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:25:51] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.291 second response time [13:26:05] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:26:46] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:27:21] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:29:03] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:29:24] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/na [13:29:24] lugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main [13:29:24] e "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1089, in query return self.resolve(qname, rdtype, rdclass, tcp, source, File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in res [13:29:24] imeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406147718429565 seconds [13:30:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.97, 8.13, 8.01 [13:30:55] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.947 second response time [13:31:21] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.745 second response time [13:31:30] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.544 second response time [13:32:06] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 1.306 second response time [13:32:15] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.171 second response time [13:32:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.48, 7.12, 7.65 [13:33:06] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:33:12] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:33:19] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:33:20] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:33:24] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:35:06] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.148 second response time [13:35:11] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.213 second response time [13:35:16] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.774 second response time [13:35:21] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.378 second response time [13:35:27] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.038 second response time [13:35:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:36:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.43, 7.47, 7.64 [13:37:50] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.94, 1.40, 0.81 [13:38:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.50, 4.32, 3.50 [13:39:50] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.54, 1.05, 0.75 [13:42:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.89, 3.50 [13:42:32] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [13:42:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [13:43:50] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.30, 20.69, 19.89 [13:45:47] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.14, 20.12, 19.77 [13:45:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.73, 3.35, 3.37 [13:46:37] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:49:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.88, 3.44, 3.43 [13:50:00] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:50:01] RECOVERY - wiki.fbpml.org - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/nagi [13:50:01] gins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - wiki.fbpml.org reverse DNS resolves to cp20.miraheze.org - CNAME OK [13:50:21] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:50:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [13:51:31] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:51:31] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:51:34] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:51:50] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:51:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.67, 3.52 [13:51:58] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:52:03] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:52:17] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:52:21] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:53:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.54, 3.48 [13:54:35] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.85, 20.46, 20.03 [13:55:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.35, 3.13, 3.34 [13:56:31] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.05, 21.45, 20.40 [13:57:38] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.587 second response time [13:57:39] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.637 second response time [13:57:48] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.008 second response time [13:58:02] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.218 second response time [13:58:22] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.480 second response time [13:58:28] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.31, 20.47, 20.17 [14:00:18] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.935 second response time [14:00:38] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.341 second response time [14:00:43] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 8.530 second response time [14:00:47] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.025 second response time [14:01:04] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.853 second response time [14:02:01] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:02:02] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:02:11] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:02:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.71, 19.69, 19.92 [14:02:28] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:02:47] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:35] PROBLEM - wiki.fbpml.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns2.hostnetbv.com.', 'ns1.hostnet.nl.', 'ns3.hostnetbv.nl.'], 'CNAME': 'fbpml.miraheze.org.'} [14:06:18] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.468 second response time [14:06:26] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.133 second response time [14:06:37] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.406 second response time [14:07:01] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.308 second response time [14:10:21] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.276 second response time [14:10:46] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.58, 3.31 [14:11:45] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:11:55] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:12:14] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:12:24] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:12:30] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:29] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:49] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:14:43] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:14:58] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:15:11] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:16:30] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.65, 3.40, 3.33 [14:18:34] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.517 second response time [14:18:45] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.991 second response time [14:18:54] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 7.155 second response time [14:22:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.24, 20.67, 20.16 [14:22:56] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:23:07] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:23:19] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:45] RECOVERY - wiki.fbpml.org - reverse DNS on sslhost is OK: SSL OK - wiki.fbpml.org reverse DNS resolves to cp21.miraheze.org - CNAME OK [14:28:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.46, 20.18, 20.19 [14:29:50] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.897 second response time [14:29:57] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.268 second response time [14:32:34] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.443 second response time [14:32:38] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.646 second response time [14:34:25] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:35:02] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.936 second response time [14:35:11] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.492 second response time [14:35:26] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.385 second response time [14:35:47] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 5.965 second response time [14:36:09] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 7.499 second response time [14:36:26] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.601 second response time [14:37:37] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.686 second response time [14:38:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:44:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:45:07] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:45:24] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:47:00] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:47:58] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:48:08] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:50:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:50:57] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:51:01] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:51:06] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.52, 21.18, 20.54 [14:52:16] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:53:00] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:53:24] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:54:31] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.805 second response time [14:54:58] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.15, 20.09, 20.22 [14:55:25] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.629 second response time [14:55:27] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.784 second response time [14:58:08] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.248 second response time [14:58:31] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.156 second response time [15:05:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.65, 3.37 [15:08:03] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 6.346 second response time [15:08:04] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.774 second response time [15:08:08] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.954 second response time [15:09:05] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.446 second response time [15:09:35] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.027 second response time [15:10:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:10:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:13:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.69, 3.77, 3.50 [15:14:00] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:14:09] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:14:11] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:14:15] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:14:23] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:14:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:14:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:15:31] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:16:02] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:16:32] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:16:36] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:16:42] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:17:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.59, 3.49 [15:18:25] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.671 second response time [15:18:26] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.246 second response time [15:18:31] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.571 second response time [15:18:36] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.428 second response time [15:20:22] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.775 second response time [15:22:12] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.896 second response time [15:22:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.31, 20.78, 20.18 [15:22:42] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.027 second response time [15:22:54] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.008 second response time [15:23:00] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.619 second response time [15:26:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.69, 20.39, 20.15 [15:26:55] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:28:06] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:28:23] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:28:35] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:28:42] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:29:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.42, 3.16, 3.39 [15:30:42] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:30:43] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:30:46] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:30:50] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:30:52] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:02] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:31:14] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:31:24] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:31:39] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:32:22] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.453 second response time [15:32:31] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.313 second response time [15:32:46] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.866 second response time [15:32:57] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.562 second response time [15:33:12] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.006 second response time [15:33:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:35:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.58, 7.27, 7.95 [15:37:47] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 7.56, 7.63, 8.01 [15:38:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:38:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:39:16] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.949 second response time [15:39:16] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.972 second response time [15:39:45] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.505 second response time [15:40:04] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.560 second response time [15:40:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:40:55] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.872 second response time [15:40:57] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.16, 1.74, 1.38 [15:41:14] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.912 second response time [15:41:34] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.510 second response time [15:41:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:41:42] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.417 second response time [15:41:49] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.169 second response time [15:41:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.29, 3.69, 3.48 [15:42:55] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.62, 1.39, 1.29 [15:43:10] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.482 second response time [15:43:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.51, 3.33, 3.38 [15:44:08] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:44:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:44:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:45:58] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:04] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:46:16] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:46:34] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:22] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:47:33] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:47:39] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:47:41] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:48:12] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.271 second response time [15:48:14] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:49:46] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.898 second response time [15:50:06] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.585 second response time [15:50:15] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.881 second response time [15:50:24] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.142 second response time [15:51:38] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.632 second response time [15:51:42] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:51:48] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:52:09] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:52:21] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:52:53] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.56, 3.92, 3.55 [15:53:41] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23566 bytes in 0.850 second response time [15:53:41] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.258 second response time [15:53:44] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.952 second response time [15:53:53] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.451 second response time [15:54:08] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.030 second response time [15:54:29] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.767 second response time [15:54:35] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.946 second response time [15:54:51] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:55:16] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23577 bytes in 6.701 second response time [15:56:01] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:56:13] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:56:35] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:41] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:56:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.80, 3.59 [15:57:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.64, 20.55, 19.66 [16:01:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.49, 20.40, 19.81 [16:02:17] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.190 second response time [16:02:27] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.809 second response time [16:02:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.72, 3.59 [16:02:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:02:53] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.402 second response time [16:02:58] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.029 second response time [16:03:15] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.874 second response time [16:04:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:05:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.84, 7.36, 7.93 [16:06:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.97, 3.74 [16:11:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.93, 8.22, 8.04 [16:11:41] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [16:12:04] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 3.00, 3.40 [16:12:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 149.56.141.75/cpweb [16:13:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.75, 7.98, 7.97 [16:14:35] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:14:37] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:15:23] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:15:29] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:15:45] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:17:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.70, 8.12, 7.96 [16:17:31] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.957 second response time [16:19:20] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:19:31] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:19:36] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:20:59] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:21:17] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.123 second response time [16:21:32] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.662 second response time [16:21:35] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.672 second response time [16:21:44] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.837 second response time [16:21:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.94, 3.49, 3.37 [16:21:57] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.744 second response time [16:22:05] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.01, 20.65, 20.12 [16:23:02] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.638 second response time [16:23:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.96, 3.31, 3.32 [16:24:02] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.36, 20.04, 19.95 [16:26:07] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:26:11] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:26:24] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:26:38] [02puppet] 07RhinosF1 opened pull request 03#2303: DC-Switch: Failover LE - 13https://github.com/miraheze/puppet/pull/2303 [16:26:39] [url] Page not found · GitHub · GitHub | github.com [16:27:19] Reception123: ready? [16:27:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.29, 3.29 [16:28:36] Merging now [16:29:10] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.211 second response time [16:29:14] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.035 second response time [16:29:26] [02puppet] 07Reception123 closed pull request 03#2303: DC-Switch: Failover LE - 13https://github.com/miraheze/puppet/pull/2303 [16:29:26] [url] Page not found · GitHub · GitHub | github.com [16:29:27] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/efbd2df0efc4...6828040a8538 [16:29:28] [url] Comparing efbd2df0efc4...6828040a8538 · miraheze/puppet · GitHub | github.com [16:29:28] [02miraheze/puppet] 07RhinosF1 036828040 - DC-Switch: Failover LE (#2303) [16:29:37] RhinosF1: ^ [16:29:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.42, 2.94, 3.16 [16:29:56] Reception123: pull to puppet pls [16:30:01] on new DC [16:30:02] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:30:06] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:22] Reception123: no, old DC [16:30:30] you just need to run puppet on cp* [16:30:49] both already done [16:31:42] Reception123: and ran puppet on cp* [16:31:42] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:31:45] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:31:46] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:33:46] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.66, 19.76, 19.65 [16:34:30] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.758 second response time [16:34:36] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.665 second response time [16:34:42] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.320 second response time [16:35:42] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.08, 19.31, 19.51 [16:35:53] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.393 second response time [16:35:56] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.258 second response time [16:36:00] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.812 second response time [16:36:19] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 5.864 second response time [16:36:32] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 9.830 second response time [16:40:45] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:41:01] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:41:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.45, 20.50, 19.86 [16:42:55] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:43:21] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:43:43] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:43:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.86, 3.65, 3.40 [16:44:52] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.930 second response time [16:45:04] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:45:12] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.376 second response time [16:45:13] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:18] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.009 second response time [16:45:21] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:45:44] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:45:48] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.554 second response time [16:45:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.23, 4.16, 3.61 [16:46:05] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:46:21] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:46:24] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:46:55] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:47:07] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.854 second response time [16:47:08] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.278 second response time [16:47:19] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.364 second response time [16:47:22] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.62, 19.74, 19.83 [16:50:10] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:13] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.093 second response time [16:51:31] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:44] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:51:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.99, 3.73 [16:52:04] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:52:11] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.119 second response time [16:52:16] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:52:21] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:52:25] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:52:33] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.927 second response time [16:52:39] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.246 second response time [16:52:57] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:53:36] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:54:00] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:54:08] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:54:09] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:54:17] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:54:48] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:42] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:55:59] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.947 second response time [16:56:07] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.021 second response time [16:56:12] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.728 second response time [16:56:17] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.290 second response time [16:56:32] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.581 second response time [16:56:40] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 8.532 second response time [16:56:41] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:56:51] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.145 second response time [16:57:02] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:57:05] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:57:14] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.958 second response time [16:57:52] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.223 second response time [16:58:25] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.884 second response time [16:58:46] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.832 second response time [16:58:53] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.469 second response time [16:59:02] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.486 second response time [16:59:03] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.452 second response time [17:00:38] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.358 second response time [17:01:40] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.88, 1.61, 1.06 [17:01:52] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.925 second response time [17:01:58] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.580 second response time [17:02:09] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.497 second response time [17:02:20] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.708 second response time [17:02:42] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.844 second response time [17:03:39] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.61, 2.18, 1.31 [17:04:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:05:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.99, 20.59, 20.14 [17:05:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.54, 4.24, 3.85 [17:06:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:07:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.79, 1.54, 1.26 [17:07:48] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.65, 19.54, 19.82 [17:08:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [17:12:59] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:13:04] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:13:39] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb [17:13:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.87, 20.77, 20.24 [17:13:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.84, 3.86, 3.89 [17:14:39] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:14:45] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:15:01] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:15:36] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.22, 20.01, 20.01 [17:16:27] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:16:41] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:16:58] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:05] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:20:40] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.066 second response time [17:20:47] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.018 second response time [17:20:51] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.692 second response time [17:21:01] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.580 second response time [17:21:08] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.010 second response time [17:21:13] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.658 second response time [17:21:14] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.532 second response time [17:21:22] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.506 second response time [17:21:36] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.942 second response time [17:25:27] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:31] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:25:44] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:26:05] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:26:34] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:27:07] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:27:16] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:27:22] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:27:40] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:27:44] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:31:22] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.273 second response time [17:31:42] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.873 second response time [17:31:48] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 4.313 second response time [17:31:59] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.149 second response time [17:32:22] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.332 second response time [17:33:04] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03gdnsd3 [+0/-0/±2] 13https://github.com/miraheze/dns/commit/43f987f0fe13 [17:33:06] [02miraheze/dns] 07JohnFLewis 0343f987f - push gdnsd3 config [17:33:07] [02dns] 07JohnFLewis created branch 03gdnsd3 - 13https://github.com/miraheze/dns [17:33:08] [url] Page not found · GitHub · GitHub | github.com [17:34:06] [02dns] 07JohnFLewis opened pull request 03#243: push gdnsd3 config - 13https://github.com/miraheze/dns/pull/243 [17:34:07] [url] Page not found · GitHub · GitHub | github.com [17:34:11] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.60, 21.30, 20.70 [17:34:13] [02dns] 07JohnFLewis closed pull request 03#243: push gdnsd3 config - 13https://github.com/miraheze/dns/pull/243 [17:34:14] [url] Page not found · GitHub · GitHub | github.com [17:34:15] [02miraheze/dns] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±4] 13https://github.com/miraheze/dns/compare/52ceba44c580...9ec550afa1dd [17:34:15] [url] Comparing 52ceba44c580...9ec550afa1dd · miraheze/dns · GitHub | github.com [17:34:16] [02miraheze/dns] 07JohnFLewis 039ec550a - Merge pull request #243 from miraheze/gdnsd3 [17:34:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:34:57] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.621 second response time [17:35:48] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.217 second response time [17:36:00] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.745 second response time [17:37:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.70, 3.56 [17:38:16] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.143 second response time [17:38:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:39:49] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.802 second response time [17:39:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.64, 3.55 [17:41:53] RECOVERY - ns2 Auth DNS on ns2 is OK: DNS OK: 0.043 seconds response time. miraheze.org returns 149.56.141.75,2607:5300:201:3100::929a [17:42:20] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [17:42:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb [17:42:40] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:42:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:44:11] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:44:22] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:44:24] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:44:36] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:45:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.61, 3.53 [17:47:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.24, 3.45, 3.48 [17:49:26] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: John, last run 24 minutes ago with 0 failures [17:49:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.49, 3.16, 3.38 [17:50:08] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:50:20] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:50:26] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.019 second response time [17:50:28] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [17:50:30] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.015 second response time [17:50:33] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.306 second response time [17:50:44] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.307 second response time [17:50:53] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:51:02] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.349 second response time [17:51:07] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:51:26] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:51:40] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:51:55] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:52:29] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:31] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:52:49] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:56] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:53:00] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:53:09] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:53:24] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:53:35] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.38, 19.61, 20.28 [17:53:43] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.990 second response time [17:53:49] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:53:49] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:53:52] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:53:53] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:53:54] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 2.012 second response time [17:53:55] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:56] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:18] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.374 second response time [17:54:29] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 1.001 second response time [17:54:30] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.333 second response time [17:54:59] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:55:58] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.330 second response time [17:56:47] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:56:50] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:56:59] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:57:03] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.020 second response time [17:57:04] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:57:16] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.101 second response time [17:57:16] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.087 second response time [17:57:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.79, 7.20, 7.95 [17:57:24] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:32] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.985 second response time [17:58:05] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.626 second response time [17:58:08] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.531 second response time [17:59:23] RECOVERY - bn.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - bn.gyaanipedia.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [18:00:10] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.079 second response time [18:00:10] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.936 second response time [18:00:12] RECOVERY - de.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - de.gyaanipedia.com reverse DNS resolves to cp31.miraheze.org - CNAME OK [18:00:17] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.682 second response time [18:00:29] RECOVERY - commons.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - commons.gyaanipedia.com reverse DNS resolves to cp21.miraheze.org - CNAME OK [18:00:45] RECOVERY - en.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [18:00:46] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - en.gyaanipedia.com reverse DNS resolves to cp21.miraheze.org - CNAME OK [18:01:12] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.594 second response time [18:01:18] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.792 second response time [18:01:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.01, 7.54, 7.93 [18:01:22] RECOVERY - fr.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - fr.gyaanipedia.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [18:01:30] RECOVERY - mr.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [18:01:30] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - mr.gyaanipedia.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [18:01:47] PROBLEM - bn.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'bn.gyaanipedia.com' expired on Wed 17 Nov 2021 04:06:31 GMT +0000. [18:01:54] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.337 second response time [18:01:54] RECOVERY - lg.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - lg.gyaanipedia.com reverse DNS resolves to cp21.miraheze.org - CNAME OK [18:01:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.73, 3.34, 3.19 [18:02:04] PROBLEM - commons.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'commons.gyaanipedia.com' expired on Sat 06 Nov 2021 14:23:16 GMT +0000. [18:02:10] RECOVERY - pa.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - pa.gyaanipedia.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [18:02:23] PROBLEM - bn.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'bn.gyaanipedia.com' expired on Wed 17 Nov 2021 04:06:31 GMT +0000. [18:02:36] RECOVERY - es.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - es.gyaanipedia.com reverse DNS resolves to cp21.miraheze.org - CNAME OK [18:02:41] RECOVERY - fr.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [18:02:41] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - fr.gyaanipedia.com reverse DNS resolves to cp21.miraheze.org - CNAME OK [18:02:53] RECOVERY - zh.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - zh.gyaanipedia.com reverse DNS resolves to cp21.miraheze.org - CNAME OK [18:03:09] RECOVERY - pa.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [18:03:09] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - pa.gyaanipedia.com reverse DNS resolves to cp30.miraheze.org - CNAME OK [18:03:23] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.610 second response time [18:03:36] RECOVERY - meta.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/li [18:03:36] os/plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - meta.gyaanipedia.com reverse DNS resolves to cp30.miraheze.org - CNAME OK [18:03:44] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.671 second response time [18:03:48] RECOVERY - hi.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - hi.gyaanipedia.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [18:03:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.17, 3.25, 3.18 [18:04:08] PROBLEM - en.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'en.gyaanipedia.com' expired on Mon 10 Jan 2022 06:04:23 GMT +0000. [18:04:08] PROBLEM - lg.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'lg.gyaanipedia.com' expires in 16 hours (Thu 13 Jan 2022 10:21:40 GMT +0000) [18:04:18] RECOVERY - hi.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [18:04:18] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - hi.gyaanipedia.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [18:04:21] RECOVERY - bn.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [18:04:21] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - bn.gyaanipedia.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [18:04:34] PROBLEM - zh.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'zh.gyaanipedia.com' expired on Wed 24 Nov 2021 06:09:39 GMT +0000. [18:04:39] RECOVERY - meta.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - meta.gyaanipedia.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [18:04:43] RECOVERY - commons.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr [18:04:43] agios/plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - commons.gyaanipedia.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [18:04:57] RECOVERY - de.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [18:04:57] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - de.gyaanipedia.com reverse DNS resolves to cp30.miraheze.org - CNAME OK [18:05:17] PROBLEM - de.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'de.gyaanipedia.com' expired on Wed 20 Oct 2021 09:04:41 GMT +0000. [18:05:22] PROBLEM - pa.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'pa.gyaanipedia.com' expired on Sat 06 Nov 2021 14:11:17 GMT +0000. [18:05:24] RECOVERY - en.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - en.gyaanipedia.com reverse DNS resolves to cp31.miraheze.org - CNAME OK [18:05:54] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.722 second response time [18:05:55] RECOVERY - es.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [18:05:55] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - es.gyaanipedia.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [18:06:04] PROBLEM - mr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'mr.gyaanipedia.com' expired on Sat 06 Nov 2021 14:05:54 GMT +0000. [18:06:20] PROBLEM - meta.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'meta.gyaanipedia.com' expired on Sat 06 Nov 2021 14:00:24 GMT +0000. [18:06:41] RECOVERY - lg.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [18:06:41] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - lg.gyaanipedia.com reverse DNS resolves to cp21.miraheze.org - CNAME OK [18:06:47] PROBLEM - fr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'fr.gyaanipedia.com' expired on Mon 10 Jan 2022 05:01:26 GMT +0000. [18:07:16] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.308 second response time [18:07:19] RECOVERY - mr.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - mr.gyaanipedia.com reverse DNS resolves to cp21.miraheze.org - CNAME OK [18:07:21] Oh f [18:07:21] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.989 second response time [18:07:30] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.107 second response time [18:07:30] PROBLEM - zh.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'zh.gyaanipedia.com' expired on Wed 24 Nov 2021 06:09:39 GMT +0000. [18:07:31] PROBLEM - en.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'en.gyaanipedia.com' expired on Mon 10 Jan 2022 06:04:23 GMT +0000. [18:07:32] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.080 second response time [18:07:33] PROBLEM - meta.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'meta.gyaanipedia.com' expired on Sat 06 Nov 2021 14:00:24 GMT +0000. [18:07:33] PROBLEM - es.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'es.gyaanipedia.com' expired on Wed 20 Oct 2021 07:55:00 GMT +0000. [18:07:47] PROBLEM - de.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'de.gyaanipedia.com' expired on Wed 20 Oct 2021 09:04:41 GMT +0000. [18:07:53] PROBLEM - lg.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'lg.gyaanipedia.com' expires in 16 hours (Thu 13 Jan 2022 10:21:40 GMT +0000) [18:07:53] PROBLEM - commons.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'commons.gyaanipedia.com' expired on Sat 06 Nov 2021 14:23:16 GMT +0000. [18:08:14] PROBLEM - hi.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'hi.gyaanipedia.com' expired on Sat 06 Nov 2021 14:08:24 GMT +0000. [18:08:37] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:08:40] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:08:46] RECOVERY - zh.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [18:08:46] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - zh.gyaanipedia.com reverse DNS resolves to cp21.miraheze.org - CNAME OK [18:08:52] PROBLEM - fr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'fr.gyaanipedia.com' expired on Mon 10 Jan 2022 05:01:26 GMT +0000. [18:09:01] PROBLEM - hi.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'hi.gyaanipedia.com' expired on Sat 06 Nov 2021 14:08:24 GMT +0000. [18:09:34] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:09:35] PROBLEM - mr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'mr.gyaanipedia.com' expired on Sat 06 Nov 2021 14:05:54 GMT +0000. [18:09:36] PROBLEM - pa.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'pa.gyaanipedia.com' expired on Sat 06 Nov 2021 14:11:17 GMT +0000. [18:09:55] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:10:42] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.330 second response time [18:10:46] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.256 second response time [18:11:35] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.239 second response time [18:11:59] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.396 second response time [18:12:22] PROBLEM - es.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'es.gyaanipedia.com' expired on Wed 20 Oct 2021 07:55:00 GMT +0000. [18:13:18] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.006 second response time [18:13:27] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:39] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:39] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:48] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:49] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:56] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:14:10] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:14:10] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:14:15] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:15:08] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:15:14] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:15:17] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.010 second response time [18:15:25] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.480 second response time [18:15:36] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.137 second response time [18:15:46] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.767 second response time [18:15:46] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.146 second response time [18:15:56] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:16:26] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:16:40] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:17:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.00, 7.30, 7.78 [18:18:32] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:19:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.62, 7.99, 7.96 [18:19:43] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:19:51] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.269 second response time [18:19:52] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.14, 21.07, 20.20 [18:19:55] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:20:00] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.323 second response time [18:20:02] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:03] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.533 second response time [18:20:09] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:20:09] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:20:29] looks like the various le certs for gyaanipedia.com have expired (have been since november) - seems like Reception123 renewed them in october, but the newer ones aren't being used for some reason [18:20:36] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.645 second response time [18:20:36] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.653 second response time [18:20:36] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.036 second response time [18:20:37] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.202 second response time [18:20:53] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.262 second response time [18:21:28] MacFan4000: it's because they were proxying [18:21:31] And finally removed it [18:21:32] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.879 second response time [18:21:35] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw12 [18:21:37] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.265 second response time [18:21:42] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.038 second response time [18:21:47] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.95, 20.29, 20.02 [18:21:53] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.535 second response time [18:21:55] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.86, 3.92, 3.44 [18:22:01] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.013 second response time [18:22:01] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:22:06] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.341 second response time [18:22:06] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.597 second response time [18:22:14] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.27, 1.77, 1.26 [18:22:19] RhinosF1: yess but https://github.com/miraheze/ssl/blob/master/certificates/en.gyaanipedia.com.crt [18:22:20] [url] ssl/en.gyaanipedia.com.crt at master · miraheze/ssl · GitHub | github.com [18:22:20] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:37] the latest change to that suggests that it was renewed in october [18:22:42] it should be valid\ [18:23:31] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 18 backends are healthy [18:23:38] MacFan4000: nope [18:23:41] Read the date [18:23:52] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.013 second response time [18:23:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.95, 4.18, 3.58 [18:23:59] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.324 second response time [18:24:13] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.85, 1.37, 1.17 [18:24:17] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:24:19] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.841 second response time [18:24:26] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:24:57] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:25:05] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:25:07] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:25:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.95, 7.48, 7.87 [18:25:39] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.56, 20.83, 20.28 [18:25:57] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:26:07] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:27:34] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.87, 20.25, 20.14 [18:28:00] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.333 second response time [18:28:12] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.408 second response time [18:29:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.76, 3.59 [18:30:42] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.449 second response time [18:30:45] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.944 second response time [18:31:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.04, 7.17, 7.54 [18:31:22] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.995 second response time [18:31:29] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.654 second response time [18:31:31] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.215 second response time [18:31:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.51, 3.97, 3.69 [18:32:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:33:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.19, 3.81, 3.68 [18:35:06] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:35:08] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:35:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.42, 7.69, 7.69 [18:35:52] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:35:58] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:02] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:36:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [18:37:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.37, 8.05, 7.81 [18:37:58] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.133 second response time [18:38:06] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.579 second response time [18:38:10] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.695 second response time [18:39:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.29, 7.88, 7.79 [18:39:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 3.78, 3.66 [18:41:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.57, 7.97, 7.82 [18:41:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.55, 3.46, 3.56 [18:42:14] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:42:28] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:42:36] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:41] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:43:42] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:43:43] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:43:46] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:44:01] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:44:19] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.440 second response time [18:45:11] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.06, 19.61, 19.14 [18:46:00] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:46:38] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:46:38] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:46:44] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:46:53] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:47:07] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.97, 19.67, 19.22 [18:47:56] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.589 second response time [18:47:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.21, 2.89, 3.29 [18:47:59] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.013 second response time [18:47:59] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.300 second response time [18:48:25] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:48:25] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:48:45] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:48:46] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:50:02] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:50:07] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:50:18] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:50:46] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:51:02] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:51:05] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.902 second response time [18:51:06] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.06, 20.46, 19.65 [18:51:12] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.810 second response time [18:51:19] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.333 second response time [18:52:20] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:52:21] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:52:22] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:52:26] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:52:26] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:53:05] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.25, 20.32, 19.70 [18:53:13] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.88, 20.29, 19.95 [18:53:51] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.021 second response time [18:53:53] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.909 second response time [18:54:24] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.254 second response time [18:54:58] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.249 second response time [18:55:06] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.417 second response time [18:55:10] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.544 second response time [18:55:13] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.11, 21.52, 20.43 [18:55:17] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.540 second response time [18:56:11] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.323 second response time [18:56:21] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.025 second response time [18:56:27] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.325 second response time [18:56:30] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.600 second response time [18:56:35] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.034 second response time [18:56:43] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.232 second response time [18:56:49] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.778 second response time [18:56:49] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.090 second response time [18:56:59] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.40, 21.08, 20.22 [18:57:13] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.88, 21.16, 20.43 [18:57:16] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.000 second response time [18:57:19] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.108 second response time [18:58:52] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03revert-4770-dependabot/submodules/REL1_37/extensions/DisplayTitle-6def2db [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/fa4539529a9f [18:58:54] [02miraheze/mediawiki] 07Universal-Omega 03fa45395 - Revert "Bump extensions/DisplayTitle from `d480066` to `6def2db` (#4770)" [18:58:55] [02mediawiki] 07Universal-Omega opened pull request 03#4939: Revert "Bump extensions/DisplayTitle from `d480066` to `6def2db`" - 13https://github.com/miraheze/mediawiki/pull/4939 [18:58:56] [url] Page not found · GitHub · GitHub | github.com [18:58:57] [02mediawiki] 07Universal-Omega created branch 03revert-4770-dependabot/submodules/REL1_37/extensions/DisplayTitle-6def2db - 13https://github.com/miraheze/mediawiki [18:58:57] ... [18:59:21] [02mediawiki] 07github-actions[bot] labeled pull request 03#4939: Revert "Bump extensions/DisplayTitle from `d480066` to `6def2db`" - 13https://github.com/miraheze/mediawiki/pull/4939 [18:59:21] [url] Page not found · GitHub · GitHub | github.com [18:59:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#4939: Revert "Bump extensions/DisplayTitle from `d480066` to `6def2db`" - 13https://github.com/miraheze/mediawiki/pull/4939 [18:59:23] ... [18:59:35] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 6.882 second response time [19:00:04] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.31, 1.95, 1.29 [19:00:27] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.319 second response time [19:00:46] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.771 second response time [19:00:52] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.71, 20.25, 20.11 [19:00:52] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 5.260 second response time [19:01:06] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.761 second response time [19:01:58] RhinosF1: ^ FYI. It causes citations to mess up with the cite extension (making them always display as ampersand (&)) so should probably be reverted for now until it can be fixed. [19:02:04] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.30, 1.54, 1.21 [19:04:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:05:15] CosmicAlpha: ok [19:06:20] CosmicAlpha: I don't see it in this week's queue [19:06:30] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.10, 3.74, 3.42 [19:06:30] Feel free to deploy or I will tonight [19:06:31] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:06:37] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:06:49] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:06:57] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:07:06] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:07:42] !log changed ns2's glue in name.com to new Infra [19:08:25] I'd imagine https://github.com/wikimedia/mediawiki-extensions-DisplayTitle/commit/6def2dbfee56823a63dead16a5d54ca30e2157c8 at fault [19:08:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.70, 3.44 [19:08:25] [url] Make links to anchors visible · wikimedia/mediawiki-extensions-DisplayTitle@6def2db · GitHub | github.com [19:08:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:49] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/9ec550afa1dd...eec138bc6367 [19:08:50] [url] Comparing 9ec550afa1dd...eec138bc6367 · miraheze/dns · GitHub | github.com [19:08:51] [02miraheze/dns] 07JohnFLewis 03eec138b - deploy ns2 into production [19:09:25] RhinosF1: yeah that's what I was thinking also. [19:10:06] CosmicAlpha: I raised it in #mediawiki [19:10:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.72, 3.39, 3.36 [19:10:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [19:10:47] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.190 second response time [19:10:58] CosmicAlpha: could you link in there to an example oage [19:11:28] Let me set up reproduction on betawiki then. [19:13:15] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.25, 19.38, 20.07 [19:14:44] Done [19:14:47] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.591 second response time [19:15:12] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23579 bytes in 4.544 second response time [19:15:15] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.164 second response time [19:15:39] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.585 second response time [19:16:17] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:16:44] CosmicAlpha: cool, do you want me to roll it back if no fix when I do the weekly run [19:16:50] Probably in like 2.5 hours? [19:17:47] RhinosF1: yeah, just merge my PR I already opened to revert it. I'm currently mobile or I would. [19:17:59] Perfect [19:18:23] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 6.499 second response time [19:18:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.49, 3.41 [19:20:28] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [19:20:42] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:20:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.72, 3.50 [19:21:13] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:21:29] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:21:32] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:21:41] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:21:45] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:21:47] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:21:53] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:22:51] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:05] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:23:17] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.827 second response time [19:23:46] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.739 second response time [19:23:54] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.347 second response time [19:23:59] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.252 second response time [19:24:46] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.04, 3.44, 3.45 [19:24:59] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 9.560 second response time [19:25:07] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.432 second response time [19:25:30] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.010 second response time [19:25:37] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.325 second response time [19:25:55] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.538 second response time [19:26:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.69, 3.22, 3.37 [19:27:40] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:28:08] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:28:19] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:28:26] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:30:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.29, 3.86, 3.60 [19:32:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.62, 3.77, 3.60 [19:38:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.23, 3.08, 3.37 [19:38:44] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.793 second response time [19:39:26] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.896 second response time [19:40:10] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.585 second response time [19:41:03] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.754 second response time [19:41:04] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.999 second response time [19:41:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:42:29] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:42:40] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:42:53] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:43:05] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:43:14] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:43:38] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:43:45] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:44:11] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:44:25] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:44:26] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:47:46] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.049 second response time [19:47:59] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.888 second response time [19:48:14] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.051 second response time [19:48:19] gyaanipedia domains are expired. [19:48:28] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [19:48:41] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.680 second response time [19:48:41] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.675 second response time [19:48:44] Reception123: ^ [19:49:09] No access currently :( [19:49:14] They all need renewing after CF was finally turned off [19:49:37] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.99, 20.79, 20.09 [19:49:48] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:49:53] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:50:31] RECOVERY - en.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [19:50:40] No [19:50:41] No [19:50:42] No [19:50:49] CosmicAlpha: why they don't that [19:50:57] We can't renew while they have it set [19:51:17] Also that should say critical [19:51:18] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:51:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.16, 20.27, 20.00 [19:51:35] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:51:38] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:51:41] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:51:46] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:07] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:52:26] PROBLEM - en.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.gyaanipedia.com could not be found [19:52:32] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:53:09] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:53:11] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:53:23] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.128 second response time [19:54:16] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:54:35] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.109 second response time [19:54:36] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:54:39] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:55:09] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.197 second response time [19:55:11] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.320 second response time [19:55:16] [02puppet] 07Universal-Omega opened pull request 03#2304: dns: update ns2 IPs in recursor - 13https://github.com/miraheze/puppet/pull/2304 [19:55:16] [url] Page not found · GitHub · GitHub | github.com [19:55:31] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.225 second response time [19:55:36] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.009 second response time [19:55:44] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.321 second response time [19:56:15] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.998 second response time [19:56:36] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw9 [19:56:42] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.739 second response time [19:56:43] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.807 second response time [19:57:02] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:04] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:57:04] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:57:13] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:57:28] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:57:31] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:57:50] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.250 second response time [19:58:05] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 8.363 second response time [19:58:26] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.203 second response time [19:58:32] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 18 backends are healthy [19:59:01] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 0.364 second response time [19:59:01] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.008 second response time [19:59:02] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:59:04] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.938 second response time [19:59:10] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.521 second response time [19:59:29] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.246 second response time [19:59:34] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 4.468 second response time [19:59:45] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.013 second response time [19:59:56] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.347 second response time [20:00:03] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:00:09] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.449 second response time [20:00:09] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.904 second response time [20:00:10] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:00:13] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.266 second response time [20:00:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:01:00] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.007 second response time [20:02:09] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.843 second response time [20:02:13] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.349 second response time [20:02:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:03:12] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.339 second response time [20:03:25] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.018 second response time [20:03:47] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.674 second response time [20:13:24] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:13:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:14:00] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:14:09] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:14:19] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:15:09] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:15:33] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:15:35] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:17:37] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.745 second response time [20:18:17] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.524 second response time [20:18:21] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.495 second response time [20:18:37] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.496 second response time [20:20:13] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:20:34] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:20:35] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:20:49] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:20:59] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:21:22] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:21:23] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.470 second response time [20:21:34] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:21:55] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:21:57] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.677 second response time [20:22:16] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:22:39] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:22:40] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:22:40] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.341 second response time [20:22:52] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:23:22] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.434 second response time [20:24:12] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/commit/ae4004ce64ea [20:24:14] [02miraheze/RemovePII] 07Universal-Omega 03ae4004c - RemovePIIJob: select PII rather than update/delete it [20:24:15] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.255 second response time [20:24:15] [02RemovePII] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/RemovePII [20:24:16] [url] Page not found · GitHub · GitHub | github.com [20:24:17] [02RemovePII] 07Universal-Omega opened pull request 03#42: RemovePIIJob: select PII rather than update/delete it - 13https://github.com/miraheze/RemovePII/pull/42 [20:24:17] [url] Page not found · GitHub · GitHub | github.com [20:24:39] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.601 second response time [20:24:40] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.574 second response time [20:24:51] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.159 second response time [20:25:38] [02miraheze/RemovePII] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/ae4004ce64ea...cbd78762380f [20:25:38] [url] Comparing ae4004ce64ea...cbd78762380f · miraheze/RemovePII · GitHub | github.com [20:25:39] [02miraheze/RemovePII] 07github-actions 03cbd7876 - CI: lint code to MediaWiki standards [20:25:41] [02RemovePII] 07github-actions[bot] synchronize pull request 03#42: RemovePIIJob: select PII rather than update/delete it - 13https://github.com/miraheze/RemovePII/pull/42 [20:25:41] [url] Page not found · GitHub · GitHub | github.com [20:25:43] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.048 second response time [20:26:01] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.110 second response time [20:26:22] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:26:23] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:28] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.232 second response time [20:26:45] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:26:51] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.309 second response time [20:26:52] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:27:06] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.923 second response time [20:27:43] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:28:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.04, 4.81, 2.86 [20:28:09] miraheze/RemovePII - Universal-Omega the build has errored. [20:29:22] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:29:54] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:30:06] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:30:30] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:30:56] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.621 second response time [20:31:06] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.693 second response time [20:31:19] [02RemovePII] 07Universal-Omega edited pull request 03#42: RemovePIIJob: select PII rather than update/delete it - 13https://github.com/miraheze/RemovePII/pull/42 [20:31:20] [url] Page not found · GitHub · GitHub | github.com [20:31:38] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.813 second response time [20:31:43] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.328 second response time [20:31:50] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:32:39] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.042 second response time [20:32:40] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.316 second response time [20:33:34] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.019 second response time [20:33:41] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/cbd78762380f...679093d52ec2 [20:33:42] [url] Comparing cbd78762380f...679093d52ec2 · miraheze/RemovePII · GitHub | github.com [20:33:43] [02miraheze/RemovePII] 07Universal-Omega 03679093d - Update RemovePIIJob.php [20:33:44] [02RemovePII] 07Universal-Omega synchronize pull request 03#42: RemovePIIJob: select PII rather than update/delete it - 13https://github.com/miraheze/RemovePII/pull/42 [20:33:45] ... [20:33:45] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.047 second response time [20:33:56] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.005 second response time [20:34:17] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.103 second response time [20:34:42] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.544 second response time [20:34:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:36:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.16, 20.66, 19.61 [20:36:46] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/679093d52ec2...2de36378a8d3 [20:36:47] [url] Comparing 679093d52ec2...2de36378a8d3 · miraheze/RemovePII · GitHub | github.com [20:36:48] [02miraheze/RemovePII] 07Universal-Omega 032de3637 - Update RemovePIIJob.php [20:36:49] [02RemovePII] 07Universal-Omega synchronize pull request 03#42: RemovePIIJob: select PII rather than update/delete it - 13https://github.com/miraheze/RemovePII/pull/42 [20:36:50] [url] Page not found · GitHub · GitHub | github.com [20:37:28] miraheze/RemovePII - Universal-Omega the build has errored. [20:38:09] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:38:18] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:38:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.62, 20.37, 19.64 [20:39:30] RECOVERY - zh.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [20:39:45] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.80, 20.31, 19.32 [20:39:50] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.18, 1.53, 1.93 [20:40:08] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.564 second response time [20:40:09] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:40:14] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.063 second response time [20:40:40] miraheze/RemovePII - Universal-Omega the build has errored. [20:40:44] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:40:50] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:40:59] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:41:24] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:41:35] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:41:52] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:42:17] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:42:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.54, 3.21 [20:42:48] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.630 second response time [20:43:03] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.744 second response time [20:43:10] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:14] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:43:18] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:43:26] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.938 second response time [20:43:34] RECOVERY - zh.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [20:43:35] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.96, 19.98, 19.41 [20:43:45] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.85, 1.09, 1.65 [20:44:20] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.865 second response time [20:44:22] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.111 second response time [20:44:26] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.007 second response time [20:44:30] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:44:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.45, 3.21 [20:44:47] PROBLEM - zh.gyaanipedia.com - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')rDNS [20:44:47] G - reverse DNS entry for zh.gyaanipedia.com could not be found [20:45:14] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:45:34] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:45:54] PROBLEM - zh.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for zh.gyaanipedia.com could not be found [20:45:58] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:46:14] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/2de36378a8d3...ffb119eb5ce2 [20:46:15] [url] Comparing 2de36378a8d3...ffb119eb5ce2 · miraheze/RemovePII · GitHub | github.com [20:46:16] [02miraheze/RemovePII] 07Universal-Omega 03ffb119e - Update RemovePIIJob.php [20:46:17] [02RemovePII] 07Universal-Omega synchronize pull request 03#42: RemovePIIJob: select PII rather than update/delete it - 13https://github.com/miraheze/RemovePII/pull/42 [20:46:18] [url] Page not found · GitHub · GitHub | github.com [20:46:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.51, 3.64, 3.29 [20:47:22] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.730 second response time [20:47:24] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.597 second response time [20:47:31] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.211 second response time [20:47:52] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.023 second response time [20:48:01] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.302 second response time [20:48:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.51, 3.29 [20:49:37] miraheze/RemovePII - Universal-Omega the build has errored. [20:50:05] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw12 [20:50:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.65, 20.97, 20.21 [20:50:27] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:50:28] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:42] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.103 second response time [20:50:53] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.727 second response time [20:50:58] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:51:07] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:51:14] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:51:23] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:57] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:52:04] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 18 backends are healthy [20:52:27] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.104 second response time [20:52:28] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 1.724 second response time [20:52:34] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:52:52] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:52:54] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.366 second response time [20:53:05] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.329 second response time [20:53:10] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.040 second response time [20:53:29] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23566 bytes in 9.192 second response time [20:54:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.68, 3.73, 3.43 [20:54:30] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.023 second response time [20:54:37] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:54:47] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:54:49] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:54:52] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.182 second response time [20:55:04] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:55:15] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:55:39] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:55:53] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.323 second response time [20:56:05] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 0.022 second response time [20:56:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.49, 20.24, 20.19 [20:56:33] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.169 second response time [20:57:05] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.530 second response time [20:57:14] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.50, 20.56, 20.04 [20:57:16] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.540 second response time [20:59:13] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.97, 19.54, 19.72 [20:59:53] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.684 second response time [20:59:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.67, 3.48, 3.46 [20:59:55] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.532 second response time [21:00:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.69, 20.53, 20.28 [21:00:49] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.661 second response time [21:01:07] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.690 second response time [21:01:12] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.643 second response time [21:02:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:02:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:03:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.28, 3.68, 3.52 [21:04:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.93, 19.32, 19.82 [21:06:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:07:18] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:07:41] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:07:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.66, 3.55 [21:08:30] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+1/-0/±0] 13https://github.com/miraheze/RemovePII/commit/584083b34298 [21:08:31] [02miraheze/RemovePII] 07Universal-Omega 03584083b - Create GeneratePII maintenance script [21:08:33] [02RemovePII] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/RemovePII [21:08:33] [url] Page not found · GitHub · GitHub | github.com [21:08:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:08:34] [02RemovePII] 07Universal-Omega opened pull request 03#43: Create GeneratePII maintenance script - 13https://github.com/miraheze/RemovePII/pull/43 [21:08:35] [url] Page not found · GitHub · GitHub | github.com [21:09:24] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.314 second response time [21:09:52] [02miraheze/RemovePII] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/584083b34298...8c390cc67550 [21:09:52] [url] Comparing 584083b34298...8c390cc67550 · miraheze/RemovePII · GitHub | github.com [21:09:53] [02miraheze/RemovePII] 07github-actions 038c390cc - CI: lint code to MediaWiki standards [21:09:55] [02RemovePII] 07github-actions[bot] synchronize pull request 03#43: Create GeneratePII maintenance script - 13https://github.com/miraheze/RemovePII/pull/43 [21:09:55] [url] Page not found · GitHub · GitHub | github.com [21:11:02] miraheze/RemovePII - Universal-Omega the build has errored. [21:11:11] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/8c390cc67550...0203a1e12ca0 [21:11:11] [url] Comparing 8c390cc67550...0203a1e12ca0 · miraheze/RemovePII · GitHub | github.com [21:11:12] [02miraheze/RemovePII] 07Universal-Omega 030203a1e - Syntax [21:11:14] [02RemovePII] 07Universal-Omega synchronize pull request 03#43: Create GeneratePII maintenance script - 13https://github.com/miraheze/RemovePII/pull/43 [21:11:14] ... [21:12:16] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:12:19] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:12:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.74, 20.47, 20.07 [21:13:14] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:14:01] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.664 second response time [21:14:14] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.020 second response time [21:14:17] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.343 second response time [21:14:20] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:14:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.23, 20.29, 20.04 [21:14:40] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:14] miraheze/RemovePII - Universal-Omega the build passed. [21:16:48] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:16:53] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:16:55] [02RemovePII] 07Universal-Omega closed pull request 03#42: RemovePIIJob: select PII rather than update/delete it - 13https://github.com/miraheze/RemovePII/pull/42 [21:16:55] [url] Page not found · GitHub · GitHub | github.com [21:16:58] [02RemovePII] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/RemovePII [21:16:58] ... [21:17:00] [02miraheze/RemovePII] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:18:24] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.77, 2.79, 1.46 [21:19:06] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/0203a1e12ca0...5decda5d5451 [21:19:07] [url] Comparing 0203a1e12ca0...5decda5d5451 · miraheze/RemovePII · GitHub | github.com [21:19:08] [02miraheze/RemovePII] 07Universal-Omega 035decda5 - Update generatePII.php [21:19:09] [02RemovePII] 07Universal-Omega synchronize pull request 03#43: Create GeneratePII maintenance script - 13https://github.com/miraheze/RemovePII/pull/43 [21:19:10] [url] Page not found · GitHub · GitHub | github.com [21:19:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.61, 3.18, 3.40 [21:20:47] RECOVERY - fr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [21:21:00] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.468 second response time [21:21:02] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 6.401 second response time [21:21:03] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.838 second response time [21:21:26] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.005 second response time [21:21:28] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:21:32] miraheze/RemovePII - Universal-Omega the build has errored. [21:21:33] RECOVERY - meta.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [21:21:33] RECOVERY - es.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [21:21:53] RECOVERY - lg.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [21:21:53] RECOVERY - commons.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [21:22:06] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:16] RECOVERY - hi.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [21:22:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.96, 1.85, 1.38 [21:22:45] PROBLEM - commons.gyaanipedia.com - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.') [21:22:45] ARNING - reverse DNS entry for commons.gyaanipedia.com could not be found [21:22:52] RECOVERY - fr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [21:22:58] PROBLEM - de.gyaanipedia.com - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')rDNS [21:22:58] G - reverse DNS entry for de.gyaanipedia.com could not be found [21:23:01] RECOVERY - hi.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [21:23:03] RECOVERY - commons.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [21:23:04] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:23:15] RECOVERY - de.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [21:23:26] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.012 second response time [21:23:36] RECOVERY - pa.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [21:23:42] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:23:43] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:23:58] PROBLEM - es.gyaanipedia.com - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')rDNS [21:23:59] G - reverse DNS entry for es.gyaanipedia.com could not be found [21:24:03] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:24:06] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.600 second response time [21:24:26] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.43, 1.37, 1.26 [21:24:34] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:24:35] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:24:40] PROBLEM - lg.gyaanipedia.com - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')rDNS [21:24:41] G - reverse DNS entry for lg.gyaanipedia.com could not be found [21:24:52] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:25:01] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.845 second response time [21:25:03] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:25:09] RECOVERY - lg.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [21:25:11] PROBLEM - de.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for de.gyaanipedia.com could not be found [21:25:22] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.565 second response time [21:25:28] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:25:30] PROBLEM - commons.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for commons.gyaanipedia.com could not be found [21:25:36] RECOVERY - es.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [21:25:38] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.015 second response time [21:25:41] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/5decda5d5451...f5fff53b681c [21:25:42] [url] Comparing 5decda5d5451...f5fff53b681c · miraheze/RemovePII · GitHub | github.com [21:25:43] [02miraheze/RemovePII] 07Universal-Omega 03f5fff53 - Update generatePII.php [21:25:43] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.350 second response time [21:25:44] [02RemovePII] 07Universal-Omega synchronize pull request 03#43: Create GeneratePII maintenance script - 13https://github.com/miraheze/RemovePII/pull/43 [21:25:45] [url] Page not found · GitHub · GitHub | github.com [21:25:49] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:25:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.56, 3.51 [21:25:59] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:26:22] RECOVERY - pa.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [21:26:23] PROBLEM - fr.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fr.gyaanipedia.com could not be found [21:26:30] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:26:31] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.145 second response time [21:26:38] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23579 bytes in 3.331 second response time [21:26:46] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:26:49] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:26:54] PROBLEM - lg.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lg.gyaanipedia.com could not be found [21:26:56] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.211 second response time [21:27:03] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.337 second response time [21:27:03] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:27:11] PROBLEM - pa.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pa.gyaanipedia.com could not be found [21:27:12] [02miraheze/RemovePII] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/f5fff53b681c...65d52f129c15 [21:27:13] [url] Comparing f5fff53b681c...65d52f129c15 · miraheze/RemovePII · GitHub | github.com [21:27:14] [02miraheze/RemovePII] 07github-actions 0365d52f1 - CI: lint code to MediaWiki standards [21:27:15] [02RemovePII] 07github-actions[bot] synchronize pull request 03#43: Create GeneratePII maintenance script - 13https://github.com/miraheze/RemovePII/pull/43 [21:27:16] [url] Page not found · GitHub · GitHub | github.com [21:27:20] RECOVERY - meta.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sni.cloudflaressl.com' will expire on Sat 15 Oct 2022 23:59:59 GMT +0000. [21:27:21] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.279 second response time [21:27:30] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23579 bytes in 3.570 second response time [21:27:32] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw13 [21:27:36] PROBLEM - es.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for es.gyaanipedia.com could not be found [21:27:42] PROBLEM - fr.gyaanipedia.com - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')rDNS [21:27:42] G - reverse DNS entry for fr.gyaanipedia.com could not be found [21:27:52] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.186 second response time [21:27:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.50, 3.76, 3.58 [21:27:59] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.034 second response time [21:28:10] PROBLEM - pa.gyaanipedia.com - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')rDNS [21:28:10] G - reverse DNS entry for pa.gyaanipedia.com could not be found [21:28:24] miraheze/RemovePII - Universal-Omega the build has errored. [21:28:34] PROBLEM - hi.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hi.gyaanipedia.com could not be found [21:28:35] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.595 second response time [21:28:36] PROBLEM - meta.gyaanipedia.com - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')rDN [21:28:36] ING - reverse DNS entry for meta.gyaanipedia.com could not be found [21:28:39] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/65d52f129c15...1af0eb658fd7 [21:28:39] [url] Comparing 65d52f129c15...1af0eb658fd7 · miraheze/RemovePII · GitHub | github.com [21:28:40] [02miraheze/RemovePII] 07Universal-Omega 031af0eb6 - Update generatePII.php [21:28:42] [02RemovePII] 07Universal-Omega synchronize pull request 03#43: Create GeneratePII maintenance script - 13https://github.com/miraheze/RemovePII/pull/43 [21:28:42] [url] Page not found · GitHub · GitHub | github.com [21:29:14] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.324 second response time [21:29:30] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [21:29:37] RECOVERY - wiki.wisrail.de - reverse DNS on sslhost is OK: SSL OK - wiki.wisrail.de reverse DNS resolves to cp31.miraheze.org - CNAME OK [21:29:43] RECOVERY - commons.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr [21:29:43] agios/plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - commons.gyaanipedia.com reverse DNS resolves to cp31.miraheze.org - CNAME OK [21:29:48] RECOVERY - de.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [21:29:49] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - de.gyaanipedia.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [21:29:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.98, 3.43, 3.48 [21:29:56] RECOVERY - en.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - en.gyaanipedia.com reverse DNS resolves to cp21.miraheze.org - CNAME OK [21:30:13] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:30:29] RECOVERY - wiki.wisrail.de - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/nag [21:30:29] ugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - wiki.wisrail.de reverse DNS resolves to cp31.miraheze.org - CNAME OK [21:30:32] PROBLEM - en.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'en.gyaanipedia.com' expired on Mon 10 Jan 2022 06:04:23 GMT +0000. [21:30:34] PROBLEM - es.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'es.gyaanipedia.com' expired on Wed 20 Oct 2021 07:55:00 GMT +0000. [21:30:48] RECOVERY - en.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [21:30:49] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - en.gyaanipedia.com reverse DNS resolves to cp21.miraheze.org - CNAME OK [21:30:53] PROBLEM - commons.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'commons.gyaanipedia.com' expired on Sat 06 Nov 2021 14:23:16 GMT +0000. [21:30:54] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:30:58] RECOVERY - es.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [21:30:58] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - es.gyaanipedia.com reverse DNS resolves to cp21.miraheze.org - CNAME OK [21:31:14] PROBLEM - hi.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'hi.gyaanipedia.com' expired on Sat 06 Nov 2021 14:08:24 GMT +0000. [21:31:25] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:31:30] RECOVERY - lg.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [21:31:30] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - lg.gyaanipedia.com reverse DNS resolves to cp21.miraheze.org - CNAME OK [21:31:52] PROBLEM - fr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'fr.gyaanipedia.com' expired on Mon 10 Jan 2022 05:01:26 GMT +0000. [21:32:01] PROBLEM - hi.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'hi.gyaanipedia.com' expired on Sat 06 Nov 2021 14:08:24 GMT +0000. [21:32:06] PROBLEM - commons.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'commons.gyaanipedia.com' expired on Sat 06 Nov 2021 14:23:16 GMT +0000. [21:32:11] PROBLEM - de.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'de.gyaanipedia.com' expired on Wed 20 Oct 2021 09:04:41 GMT +0000. [21:32:13] RECOVERY - de.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - de.gyaanipedia.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [21:32:23] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:32:27] RECOVERY - commons.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - commons.gyaanipedia.com reverse DNS resolves to cp31.miraheze.org - CNAME OK [21:32:29] RECOVERY - zh.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [21:32:29] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - zh.gyaanipedia.com reverse DNS resolves to cp30.miraheze.org - CNAME OK [21:32:29] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.240 second response time [21:32:36] PROBLEM - pa.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'pa.gyaanipedia.com' expired on Sat 06 Nov 2021 14:11:17 GMT +0000. [21:32:54] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.031 second response time [21:33:03] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.504 second response time [21:33:09] RECOVERY - fr.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - fr.gyaanipedia.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [21:33:18] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:33:18] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:33:20] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.027 second response time [21:33:34] miraheze/RemovePII - Universal-Omega the build passed. [21:33:50] RECOVERY - lg.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - lg.gyaanipedia.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [21:34:02] RECOVERY - pa.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - pa.gyaanipedia.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [21:34:06] RECOVERY - zh.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - zh.gyaanipedia.com reverse DNS resolves to cp21.miraheze.org - CNAME OK [21:34:08] PROBLEM - en.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'en.gyaanipedia.com' expired on Mon 10 Jan 2022 06:04:23 GMT +0000. [21:34:10] PROBLEM - lg.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'lg.gyaanipedia.com' expires in 12 hours (Thu 13 Jan 2022 10:21:40 GMT +0000) [21:34:16] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.343 second response time [21:34:22] RECOVERY - fr.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [21:34:23] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - fr.gyaanipedia.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [21:34:33] PROBLEM - es.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'es.gyaanipedia.com' expired on Wed 20 Oct 2021 07:55:00 GMT +0000. [21:34:35] RECOVERY - es.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - es.gyaanipedia.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [21:34:37] PROBLEM - zh.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'zh.gyaanipedia.com' expired on Wed 24 Nov 2021 06:09:39 GMT +0000. [21:34:58] RECOVERY - pa.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/ [21:34:58] /plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - pa.gyaanipedia.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [21:35:02] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.517 second response time [21:35:13] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.44, 19.75, 18.95 [21:35:21] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.994 second response time [21:35:22] PROBLEM - pa.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'pa.gyaanipedia.com' expired on Sat 06 Nov 2021 14:11:17 GMT +0000. [21:35:25] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.742 second response time [21:35:30] RECOVERY - meta.gyaanipedia.com - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/li [21:35:30] os/plugins/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - meta.gyaanipedia.com reverse DNS resolves to cp30.miraheze.org - CNAME OK [21:35:34] RECOVERY - hi.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - hi.gyaanipedia.com reverse DNS resolves to cp31.miraheze.org - CNAME OK [21:35:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.21, 3.23, 3.37 [21:36:20] PROBLEM - meta.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'meta.gyaanipedia.com' expired on Sat 06 Nov 2021 14:00:24 GMT +0000. [21:36:46] PROBLEM - fr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'fr.gyaanipedia.com' expired on Mon 10 Jan 2022 05:01:26 GMT +0000. [21:37:14] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.37, 19.23, 18.86 [21:37:17] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:37:18] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:37:30] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:37:30] PROBLEM - zh.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'zh.gyaanipedia.com' expired on Wed 24 Nov 2021 06:09:39 GMT +0000. [21:37:33] PROBLEM - meta.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'meta.gyaanipedia.com' expired on Sat 06 Nov 2021 14:00:24 GMT +0000. [21:37:46] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:37:53] PROBLEM - lg.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'lg.gyaanipedia.com' expires in 12 hours (Thu 13 Jan 2022 10:21:40 GMT +0000) [21:39:24] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:39:44] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.014 second response time [21:39:46] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:39:50] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.291 second response time [21:40:13] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:40:13] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:40:17] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:40:32] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:40:35] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.682 second response time [21:41:06] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:41:20] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.453 second response time [21:41:23] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.727 second response time [21:41:33] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.229 second response time [21:42:07] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:42:17] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.564 second response time [21:42:19] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.399 second response time [21:42:25] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.603 second response time [21:42:38] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.484 second response time [21:43:04] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.403 second response time [21:43:29] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.658 second response time [21:43:30] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:43:48] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.005 second response time [21:43:53] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.380 second response time [21:44:08] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.319 second response time [21:44:16] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.316 second response time [21:44:40] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:44:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:44:57] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:44:57] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:44:59] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:49:46] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:49:57] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:50:40] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:50:41] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:50:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:51:04] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:07] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:51:51] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.534 second response time [21:51:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.16, 3.18 [21:51:58] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:52:00] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.026 second response time [21:53:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 3.19, 3.20 [21:53:57] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.780 second response time [21:54:51] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.015 second response time [21:54:52] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.405 second response time [21:55:01] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:55:13] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 8.388 second response time [21:55:19] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23566 bytes in 7.660 second response time [21:55:27] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.423 second response time [21:55:27] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:55:30] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.732 second response time [21:55:33] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.323 second response time [21:56:02] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.012 second response time [21:56:13] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:56:18] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:56:47] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 0.319 second response time [21:56:57] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.023 second response time [21:59:43] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 8.279 second response time [22:00:24] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.133 second response time [22:00:31] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.108 second response time [22:00:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:00:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:01:00] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.412 second response time [22:01:13] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.350 second response time [22:02:00] PROBLEM - mw101 APT on mw101 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:02:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.67, 7.04, 8.00 [22:04:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.31, 7.45, 8.02 [22:04:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.98, 3.48, 3.27 [22:06:05] PROBLEM - mw122 APT on mw122 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:06:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.45, 7.53, 7.99 [22:06:44] PROBLEM - mw111 APT on mw111 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:08:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.00, 8.52, 8.30 [22:09:28] I gonna start pushing the weekly updates [22:09:46] [02mediawiki] 07RhinosF1 closed pull request 03#4939: Revert "Bump extensions/DisplayTitle from `d480066` to `6def2db`" - 13https://github.com/miraheze/mediawiki/pull/4939 [22:09:47] [url] Page not found · GitHub · GitHub | github.com [22:09:48] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03revert-4770-dependabot/submodules/REL1_37/extensions/DisplayTitle-6def2db [22:09:49] [02mediawiki] 07RhinosF1 deleted branch 03revert-4770-dependabot/submodules/REL1_37/extensions/DisplayTitle-6def2db - 13https://github.com/miraheze/mediawiki [22:09:50] ... [22:09:51] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/87695d295050...8ee7dcb5c30e [22:09:51] [url] Comparing 87695d295050...8ee7dcb5c30e · miraheze/mediawiki · GitHub | github.com [22:09:52] [02miraheze/mediawiki] 07Universal-Omega 038ee7dcb - Revert "Bump extensions/DisplayTitle from `d480066` to `6def2db` (#4770)" (#4939) [22:10:23] Thanks RhinosF1! [22:10:35] [02mediawiki] 07RhinosF1 closed pull request 03#4930: Bump extensions/PageSchemas from `7645ba0` to `7795965` - 13https://github.com/miraheze/mediawiki/pull/4930 [22:10:36] [url] Page not found · GitHub · GitHub | github.com [22:10:37] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/8ee7dcb5c30e...9f670074e966 [22:10:38] [02miraheze/mediawiki] 07dependabot[bot] 039f67007 - Bump extensions/PageSchemas from `7645ba0` to `7795965` (#4930) [22:10:39] [url] Comparing 8ee7dcb5c30e...9f670074e966 · miraheze/mediawiki · GitHub | github.com [22:10:40] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/PageSchemas-7795965 [22:10:41] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/PageSchemas-7795965 - 13https://github.com/miraheze/mediawiki [22:10:42] [url] Page not found · GitHub · GitHub | github.com [22:10:44] [02mediawiki] 07RhinosF1 closed pull request 03#4925: Bump extensions/CommentStreams from `9e2d8f1` to `702ce1c` - 13https://github.com/miraheze/mediawiki/pull/4925 [22:10:44] [url] Page not found · GitHub · GitHub | github.com [22:10:45] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/CommentStreams-702ce1c [22:10:47] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/CommentStreams-702ce1c - 13https://github.com/miraheze/mediawiki [22:10:47] ... [22:10:48] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/9f670074e966...431d08a6a493 [22:10:49] [url] Comparing 9f670074e966...431d08a6a493 · miraheze/mediawiki · GitHub | github.com [22:10:50] [02miraheze/mediawiki] 07dependabot[bot] 03431d08a - Bump extensions/CommentStreams from `9e2d8f1` to `702ce1c` (#4925) [22:10:53] [02mediawiki] 07RhinosF1 closed pull request 03#4922: Bump extensions/Cargo from `07f465d` to `b1d6c6e` - 13https://github.com/miraheze/mediawiki/pull/4922 [22:10:54] [url] Page not found · GitHub · GitHub | github.com [22:10:55] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/431d08a6a493...29ad25a1fc85 [22:10:55] [url] Comparing 431d08a6a493...29ad25a1fc85 · miraheze/mediawiki · GitHub | github.com [22:10:56] [02miraheze/mediawiki] 07dependabot[bot] 0329ad25a - Bump extensions/Cargo from `07f465d` to `b1d6c6e` (#4922) [22:10:58] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Cargo-b1d6c6e [22:10:59] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Cargo-b1d6c6e - 13https://github.com/miraheze/mediawiki [22:11:00] [url] Page not found · GitHub · GitHub | github.com [22:11:01] [02mediawiki] 07RhinosF1 closed pull request 03#4911: Bump extensions/ApprovedRevs from `609df70` to `083cc00` - 13https://github.com/miraheze/mediawiki/pull/4911 [22:11:01] [url] Page not found · GitHub · GitHub | github.com [22:11:02] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/ApprovedRevs-083cc00 [22:11:04] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/29ad25a1fc85...e262ac453e45 [22:11:04] [url] Comparing 29ad25a1fc85...e262ac453e45 · miraheze/mediawiki · GitHub | github.com [22:11:05] [02miraheze/mediawiki] 07dependabot[bot] 03e262ac4 - Bump extensions/ApprovedRevs from `609df70` to `083cc00` (#4911) [22:11:07] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/ApprovedRevs-083cc00 - 13https://github.com/miraheze/mediawiki [22:11:07] [url] Page not found · GitHub · GitHub | github.com [22:11:08] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/RatePage-5c1e4b0 [22:11:10] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/RatePage-5c1e4b0 - 13https://github.com/miraheze/mediawiki [22:11:10] [url] Page not found · GitHub · GitHub | github.com [22:11:11] [02mediawiki] 07RhinosF1 closed pull request 03#4909: Bump extensions/RatePage from `a9d6b32` to `5c1e4b0` - 13https://github.com/miraheze/mediawiki/pull/4909 [22:11:12] ... [22:11:13] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/e262ac453e45...4c7960bf8fda [22:11:13] [url] Comparing e262ac453e45...4c7960bf8fda · miraheze/mediawiki · GitHub | github.com [22:11:14] [02miraheze/mediawiki] 07dependabot[bot] 034c7960b - Bump extensions/RatePage from `a9d6b32` to `5c1e4b0` (#4909) [22:11:16] [02mediawiki] 07RhinosF1 closed pull request 03#4893: Bump extensions/Translate from `941deb5` to `266785e` - 13https://github.com/miraheze/mediawiki/pull/4893 [22:11:17] [url] Page not found · GitHub · GitHub | github.com [22:11:18] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Translate-266785e - 13https://github.com/miraheze/mediawiki [22:11:18] ... [22:11:19] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Translate-266785e [22:11:21] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/4c7960bf8fda...969459155eed [22:11:21] [url] Comparing 4c7960bf8fda...969459155eed · miraheze/mediawiki · GitHub | github.com [22:11:22] [02miraheze/mediawiki] 07dependabot[bot] 039694591 - Bump extensions/Translate from `941deb5` to `266785e` (#4893) [22:11:24] [02mediawiki] 07RhinosF1 closed pull request 03#4887: Bump skins/Femiwiki from `21a54f2` to `a4f62a4` - 13https://github.com/miraheze/mediawiki/pull/4887 [22:11:25] [url] Page not found · GitHub · GitHub | github.com [22:11:26] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/969459155eed...fd1d7001a4b7 [22:11:27] [url] Comparing 969459155eed...fd1d7001a4b7 · miraheze/mediawiki · GitHub | github.com [22:11:27] [02miraheze/mediawiki] 07dependabot[bot] 03fd1d700 - Bump skins/Femiwiki from `21a54f2` to `a4f62a4` (#4887) [22:11:29] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/skins/Femiwiki-a4f62a4 [22:11:30] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/skins/Femiwiki-a4f62a4 - 13https://github.com/miraheze/mediawiki [22:11:31] [url] Page not found · GitHub · GitHub | github.com [22:11:35] [02mediawiki] 07RhinosF1 closed pull request 03#4880: Bump skins/Metrolook from `577b947` to `eed515c` - 13https://github.com/miraheze/mediawiki/pull/4880 [22:11:35] [url] Page not found · GitHub · GitHub | github.com [22:11:36] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/skins/Metrolook-eed515c [22:11:38] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/fd1d7001a4b7...1211d7d8dd4b [22:11:38] [url] Comparing fd1d7001a4b7...1211d7d8dd4b · miraheze/mediawiki · GitHub | github.com [22:11:39] [02miraheze/mediawiki] 07dependabot[bot] 031211d7d - Bump skins/Metrolook from `577b947` to `eed515c` (#4880) [22:11:41] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/skins/Metrolook-eed515c - 13https://github.com/miraheze/mediawiki [22:11:41] [url] Page not found · GitHub · GitHub | github.com [22:11:42] [02mediawiki] 07RhinosF1 closed pull request 03#4877: Bump extensions/UniversalLanguageSelector from `f33d7ce` to `d2b84d7` - 13https://github.com/miraheze/mediawiki/pull/4877 [22:11:43] [url] Page not found · GitHub · GitHub | github.com [22:11:44] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/1211d7d8dd4b...fdfceb39d81d [22:11:44] [url] Comparing 1211d7d8dd4b...fdfceb39d81d · miraheze/mediawiki · GitHub | github.com [22:11:45] [02miraheze/mediawiki] 07dependabot[bot] 03fdfceb3 - Bump extensions/UniversalLanguageSelector from `f33d7ce` to `d2b84d7` (#4877) [22:11:47] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-d2b84d7 - 13https://github.com/miraheze/mediawiki [22:11:47] ... [22:11:48] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-d2b84d7 [22:11:51] [02mediawiki] 07RhinosF1 closed pull request 03#4875: Bump skins/Medik from `b491fc8` to `833353a` - 13https://github.com/miraheze/mediawiki/pull/4875 [22:11:52] [url] Page not found · GitHub · GitHub | github.com [22:11:53] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/fdfceb39d81d...3aca3013b368 [22:11:53] [url] Comparing fdfceb39d81d...3aca3013b368 · miraheze/mediawiki · GitHub | github.com [22:11:54] [02miraheze/mediawiki] 07dependabot[bot] 033aca301 - Bump skins/Medik from `b491fc8` to `833353a` (#4875) [22:11:56] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/skins/Medik-833353a [22:11:56] PROBLEM - mw121 APT on mw121 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:11:57] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/skins/Medik-833353a - 13https://github.com/miraheze/mediawiki [22:11:58] [url] Page not found · GitHub · GitHub | github.com [22:11:59] [02mediawiki] 07RhinosF1 closed pull request 03#4869: Bump extensions/CreatePageUw from `074295c` to `84db84a` - 13https://github.com/miraheze/mediawiki/pull/4869 [22:11:59] [url] Page not found · GitHub · GitHub | github.com [22:12:00] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/3aca3013b368...dafdbc5004a8 [22:12:01] [url] Comparing 3aca3013b368...dafdbc5004a8 · miraheze/mediawiki · GitHub | github.com [22:12:02] [02miraheze/mediawiki] 07dependabot[bot] 03dafdbc5 - Bump extensions/CreatePageUw from `074295c` to `84db84a` (#4869) [22:12:03] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/CreatePageUw-84db84a [22:12:05] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/CreatePageUw-84db84a - 13https://github.com/miraheze/mediawiki [22:12:05] [url] Page not found · GitHub · GitHub | github.com [22:12:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 149.56.141.75/cpweb [22:12:35] [02mediawiki] 07RhinosF1 closed pull request 03#4938: Bump extensions/WikiForum from `d8b6e08` to `b1e5d4e` - 13https://github.com/miraheze/mediawiki/pull/4938 [22:12:36] [url] Page not found · GitHub · GitHub | github.com [22:12:37] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/dafdbc5004a8...020f74d27705 [22:12:38] [url] Comparing dafdbc5004a8...020f74d27705 · miraheze/mediawiki · GitHub | github.com [22:12:38] [02miraheze/mediawiki] 07dependabot[bot] 03020f74d - Bump extensions/WikiForum from `d8b6e08` to `b1e5d4e` (#4938) [22:12:40] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/WikiForum-b1e5d4e [22:12:41] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/WikiForum-b1e5d4e - 13https://github.com/miraheze/mediawiki [22:12:42] ... [22:12:45] [02mediawiki] 07RhinosF1 closed pull request 03#4937: Bump extensions/Math from `9ec9134` to `4cf0a3a` - 13https://github.com/miraheze/mediawiki/pull/4937 [22:12:46] [url] Page not found · GitHub · GitHub | github.com [22:12:47] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/020f74d27705...7181e8fdfac8 [22:12:48] [url] Comparing 020f74d27705...7181e8fdfac8 · miraheze/mediawiki · GitHub | github.com [22:12:48] [02miraheze/mediawiki] 07dependabot[bot] 037181e8f - Bump extensions/Math from `9ec9134` to `4cf0a3a` (#4937) [22:12:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [22:12:50] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Math-4cf0a3a [22:12:50] PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [22:12:51] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Math-4cf0a3a - 13https://github.com/miraheze/mediawiki [22:12:52] [url] Page not found · GitHub · GitHub | github.com [22:12:54] [02mediawiki] 07RhinosF1 closed pull request 03#4936: Bump extensions/BlogPage from `c18c1c3` to `c884723` - 13https://github.com/miraheze/mediawiki/pull/4936 [22:12:54] [url] Page not found · GitHub · GitHub | github.com [22:12:55] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/BlogPage-c884723 [22:12:57] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/7181e8fdfac8...c750d7acff5c [22:12:58] [url] Comparing 7181e8fdfac8...c750d7acff5c · miraheze/mediawiki · GitHub | github.com [22:12:58] [02miraheze/mediawiki] 07dependabot[bot] 03c750d7a - Bump extensions/BlogPage from `c18c1c3` to `c884723` (#4936) [22:13:00] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/BlogPage-c884723 - 13https://github.com/miraheze/mediawiki [22:13:00] [url] Page not found · GitHub · GitHub | github.com [22:13:03] [02mediawiki] 07RhinosF1 closed pull request 03#4935: Bump extensions/TemplateData from `998163a` to `2896aeb` - 13https://github.com/miraheze/mediawiki/pull/4935 [22:13:04] [url] Page not found · GitHub · GitHub | github.com [22:13:05] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/TemplateData-2896aeb - 13https://github.com/miraheze/mediawiki [22:13:05] ... [22:13:06] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/TemplateData-2896aeb [22:13:08] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/c750d7acff5c...c292a1c6803e [22:13:09] [url] Comparing c750d7acff5c...c292a1c6803e · miraheze/mediawiki · GitHub | github.com [22:13:09] [02miraheze/mediawiki] 07dependabot[bot] 03c292a1c - Bump extensions/TemplateData from `998163a` to `2896aeb` (#4935) [22:13:14] [02mediawiki] 07RhinosF1 closed pull request 03#4933: Bump skins/MinervaNeue from `7beea49` to `15047cd` - 13https://github.com/miraheze/mediawiki/pull/4933 [22:13:15] [url] Page not found · GitHub · GitHub | github.com [22:13:16] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/skins/MinervaNeue-15047cd - 13https://github.com/miraheze/mediawiki [22:13:16] ... [22:13:17] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/c292a1c6803e...1f18f12f79bd [22:13:18] [url] Comparing c292a1c6803e...1f18f12f79bd · miraheze/mediawiki · GitHub | github.com [22:13:19] [02miraheze/mediawiki] 07dependabot[bot] 031f18f12 - Bump skins/MinervaNeue from `7beea49` to `15047cd` (#4933) [22:13:20] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/skins/MinervaNeue-15047cd [22:13:22] [02mediawiki] 07RhinosF1 closed pull request 03#4932: Bump extensions/ArticlePlaceholder from `d226109` to `23d9495` - 13https://github.com/miraheze/mediawiki/pull/4932 [22:13:22] [url] Page not found · GitHub · GitHub | github.com [22:13:23] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/1f18f12f79bd...e86a3591c2d1 [22:13:24] PROBLEM - mw102 APT on mw102 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:13:24] [url] Comparing 1f18f12f79bd...e86a3591c2d1 · miraheze/mediawiki · GitHub | github.com [22:13:25] [02miraheze/mediawiki] 07dependabot[bot] 03e86a359 - Bump extensions/ArticlePlaceholder from `d226109` to `23d9495` (#4932) [22:13:26] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/ArticlePlaceholder-23d9495 [22:13:28] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/ArticlePlaceholder-23d9495 - 13https://github.com/miraheze/mediawiki [22:13:28] [url] Page not found · GitHub · GitHub | github.com [22:13:29] PROBLEM - mwtask1 APT on mwtask1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:13:30] [02mediawiki] 07RhinosF1 closed pull request 03#4931: Bump extensions/VisualEditor from `d12a0aa` to `3aeea63` - 13https://github.com/miraheze/mediawiki/pull/4931 [22:13:30] [url] Page not found · GitHub · GitHub | github.com [22:13:31] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/e86a3591c2d1...0cf3c735af44 [22:13:32] [url] Comparing e86a3591c2d1...0cf3c735af44 · miraheze/mediawiki · GitHub | github.com [22:13:33] [02miraheze/mediawiki] 07dependabot[bot] 030cf3c73 - Bump extensions/VisualEditor from `d12a0aa` to `3aeea63` (#4931) [22:13:34] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/VisualEditor-3aeea63 - 13https://github.com/miraheze/mediawiki [22:13:34] [url] Page not found · GitHub · GitHub | github.com [22:13:36] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/VisualEditor-3aeea63 [22:13:40] [02mediawiki] 07RhinosF1 closed pull request 03#4929: Bump extensions/CentralAuth from `d59e785` to `80029dd` - 13https://github.com/miraheze/mediawiki/pull/4929 [22:13:40] [url] Page not found · GitHub · GitHub | github.com [22:13:41] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/CentralAuth-80029dd [22:13:43] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/0cf3c735af44...6aaa26dbba68 [22:13:44] [url] Comparing 0cf3c735af44...6aaa26dbba68 · miraheze/mediawiki · GitHub | github.com [22:13:44] [02miraheze/mediawiki] 07dependabot[bot] 036aaa26d - Bump extensions/CentralAuth from `d59e785` to `80029dd` (#4929) [22:13:46] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/CentralAuth-80029dd - 13https://github.com/miraheze/mediawiki [22:13:46] ... [22:13:49] [02mediawiki] 07RhinosF1 closed pull request 03#4928: Bump extensions/AbuseFilter from `4bf6a6a` to `601ffab` - 13https://github.com/miraheze/mediawiki/pull/4928 [22:13:49] [url] Page not found · GitHub · GitHub | github.com [22:13:50] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/AbuseFilter-601ffab - 13https://github.com/miraheze/mediawiki [22:13:51] ... [22:13:52] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/6aaa26dbba68...19c2c3f9b391 [22:13:52] [url] Comparing 6aaa26dbba68...19c2c3f9b391 · miraheze/mediawiki · GitHub | github.com [22:13:53] [02miraheze/mediawiki] 07dependabot[bot] 0319c2c3f - Bump extensions/AbuseFilter from `4bf6a6a` to `601ffab` (#4928) [22:13:54] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:13:55] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/AbuseFilter-601ffab [22:13:56] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:13:58] [02mediawiki] 07RhinosF1 closed pull request 03#4927: Bump extensions/GlobalUserPage from `dfb9f81` to `6819809` - 13https://github.com/miraheze/mediawiki/pull/4927 [22:13:58] [url] Page not found · GitHub · GitHub | github.com [22:13:58] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:13:59] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/GlobalUserPage-6819809 - 13https://github.com/miraheze/mediawiki [22:13:59] [url] Page not found · GitHub · GitHub | github.com [22:14:01] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/GlobalUserPage-6819809 [22:14:02] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/19c2c3f9b391...aad38a50f542 [22:14:03] [url] Comparing 19c2c3f9b391...aad38a50f542 · miraheze/mediawiki · GitHub | github.com [22:14:04] [02miraheze/mediawiki] 07dependabot[bot] 03aad38a5 - Bump extensions/GlobalUserPage from `dfb9f81` to `6819809` (#4927) [22:14:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.35, 4.25, 3.69 [22:14:25] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:14:39] [02mediawiki] 07RhinosF1 closed pull request 03#4926: Bump extensions/ConfirmEdit from `2916872` to `a591b21` - 13https://github.com/miraheze/mediawiki/pull/4926 [22:14:40] [url] Page not found · GitHub · GitHub | github.com [22:14:41] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/aad38a50f542...799dc24fc0f1 [22:14:41] [url] Comparing aad38a50f542...799dc24fc0f1 · miraheze/mediawiki · GitHub | github.com [22:14:42] [02miraheze/mediawiki] 07dependabot[bot] 03799dc24 - Bump extensions/ConfirmEdit from `2916872` to `a591b21` (#4926) [22:14:44] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/ConfirmEdit-a591b21 [22:14:45] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/ConfirmEdit-a591b21 - 13https://github.com/miraheze/mediawiki [22:14:46] [url] Page not found · GitHub · GitHub | github.com [22:14:47] [02mediawiki] 07RhinosF1 closed pull request 03#4924: Bump extensions/MediaWikiChat from `cda9558` to `768cb67` - 13https://github.com/miraheze/mediawiki/pull/4924 [22:14:47] [url] Page not found · GitHub · GitHub | github.com [22:14:48] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/799dc24fc0f1...3989ed240009 [22:14:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:14:49] [url] Comparing 799dc24fc0f1...3989ed240009 · miraheze/mediawiki · GitHub | github.com [22:14:50] [02miraheze/mediawiki] 07dependabot[bot] 033989ed2 - Bump extensions/MediaWikiChat from `cda9558` to `768cb67` (#4924) [22:14:51] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/MediaWikiChat-768cb67 [22:14:53] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/MediaWikiChat-768cb67 - 13https://github.com/miraheze/mediawiki [22:14:53] ... [22:14:57] [02mediawiki] 07RhinosF1 closed pull request 03#4923: Bump extensions/CategoryTree from `21c4cb0` to `64c6802` - 13https://github.com/miraheze/mediawiki/pull/4923 [22:14:58] [url] Page not found · GitHub · GitHub | github.com [22:14:58] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/CategoryTree-64c6802 - 13https://github.com/miraheze/mediawiki [22:14:59] [url] Page not found · GitHub · GitHub | github.com [22:15:00] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/CategoryTree-64c6802 [22:15:01] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/3989ed240009...51431d88186e [22:15:02] [url] Comparing 3989ed240009...51431d88186e · miraheze/mediawiki · GitHub | github.com [22:15:03] [02miraheze/mediawiki] 07dependabot[bot] 0351431d8 - Bump extensions/CategoryTree from `21c4cb0` to `64c6802` (#4923) [22:15:04] [02mediawiki] 07RhinosF1 closed pull request 03#4921: Bump extensions/AntiSpoof from `56b4b17` to `ed0a896` - 13https://github.com/miraheze/mediawiki/pull/4921 [22:15:05] [url] Page not found · GitHub · GitHub | github.com [22:15:06] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/51431d88186e...5178e8f81e89 [22:15:07] [url] Comparing 51431d88186e...5178e8f81e89 · miraheze/mediawiki · GitHub | github.com [22:15:07] [02miraheze/mediawiki] 07dependabot[bot] 035178e8f - Bump extensions/AntiSpoof from `56b4b17` to `ed0a896` (#4921) [22:15:09] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/AntiSpoof-ed0a896 - 13https://github.com/miraheze/mediawiki [22:15:09] [url] Page not found · GitHub · GitHub | github.com [22:15:10] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/AntiSpoof-ed0a896 [22:15:47] [02mediawiki] 07RhinosF1 closed pull request 03#4920: Bump extensions/MultimediaViewer from `412fcea` to `6b87177` - 13https://github.com/miraheze/mediawiki/pull/4920 [22:15:47] [url] Page not found · GitHub · GitHub | github.com [22:15:48] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/MultimediaViewer-6b87177 - 13https://github.com/miraheze/mediawiki [22:15:49] ... [22:15:50] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/5178e8f81e89...209fab7e411d [22:15:51] [url] Comparing 5178e8f81e89...209fab7e411d · miraheze/mediawiki · GitHub | github.com [22:15:51] [02miraheze/mediawiki] 07dependabot[bot] 03209fab7 - Bump extensions/MultimediaViewer from `412fcea` to `6b87177` (#4920) [22:15:53] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/MultimediaViewer-6b87177 [22:15:55] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.906 second response time [22:15:56] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.065 second response time [22:15:57] [02mediawiki] 07RhinosF1 closed pull request 03#4919: Bump extensions/Wikibase from `da93956` to `2261946` - 13https://github.com/miraheze/mediawiki/pull/4919 [22:15:58] [url] Page not found · GitHub · GitHub | github.com [22:15:59] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Wikibase-2261946 - 13https://github.com/miraheze/mediawiki [22:15:59] [url] Page not found · GitHub · GitHub | github.com [22:15:59] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.311 second response time [22:16:00] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Wikibase-2261946 [22:16:02] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/209fab7e411d...9dabc56dc4e3 [22:16:03] [url] Comparing 209fab7e411d...9dabc56dc4e3 · miraheze/mediawiki · GitHub | github.com [22:16:03] [02miraheze/mediawiki] 07dependabot[bot] 039dabc56 - Bump extensions/Wikibase from `da93956` to `2261946` (#4919) [22:16:07] PROBLEM - cloud4 APT on cloud4 is CRITICAL: APT CRITICAL: 11 packages available for upgrade (1 critical updates). [22:16:14] [02mediawiki] 07RhinosF1 closed pull request 03#4918: Bump extensions/CreateWiki from `e1ee9ab` to `5698d35` - 13https://github.com/miraheze/mediawiki/pull/4918 [22:16:14] [url] Page not found · GitHub · GitHub | github.com [22:16:15] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/9dabc56dc4e3...ddee0e01aafe [22:16:16] [url] Comparing 9dabc56dc4e3...ddee0e01aafe · miraheze/mediawiki · GitHub | github.com [22:16:17] [02miraheze/mediawiki] 07dependabot[bot] 03ddee0e0 - Bump extensions/CreateWiki from `e1ee9ab` to `5698d35` (#4918) [22:16:18] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/CreateWiki-5698d35 - 13https://github.com/miraheze/mediawiki [22:16:19] [url] Page not found · GitHub · GitHub | github.com [22:16:20] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/CreateWiki-5698d35 [22:16:22] [02mediawiki] 07RhinosF1 closed pull request 03#4915: Bump extensions/Echo from `e2d6620` to `34677a0` - 13https://github.com/miraheze/mediawiki/pull/4915 [22:16:22] [url] Page not found · GitHub · GitHub | github.com [22:16:23] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/ddee0e01aafe...e93996102f58 [22:16:24] [url] Comparing ddee0e01aafe...e93996102f58 · miraheze/mediawiki · GitHub | github.com [22:16:25] [02miraheze/mediawiki] 07dependabot[bot] 03e939961 - Bump extensions/Echo from `e2d6620` to `34677a0` (#4915) [22:16:26] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Echo-34677a0 [22:16:28] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Echo-34677a0 - 13https://github.com/miraheze/mediawiki [22:16:28] ... [22:16:34] [02mediawiki] 07RhinosF1 closed pull request 03#4914: Bump extensions/Interwiki from `25ba830` to `a525e3e` - 13https://github.com/miraheze/mediawiki/pull/4914 [22:16:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:16:34] [url] Page not found · GitHub · GitHub | github.com [22:16:35] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Interwiki-a525e3e - 13https://github.com/miraheze/mediawiki [22:16:35] [url] Page not found · GitHub · GitHub | github.com [22:16:37] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/e93996102f58...cf061479c342 [22:16:37] [url] Comparing e93996102f58...cf061479c342 · miraheze/mediawiki · GitHub | github.com [22:16:38] [02miraheze/mediawiki] 07dependabot[bot] 03cf06147 - Bump extensions/Interwiki from `25ba830` to `a525e3e` (#4914) [22:16:40] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Interwiki-a525e3e [22:16:42] [02mediawiki] 07RhinosF1 closed pull request 03#4913: Bump extensions/Thanks from `06f0c44` to `9421bd5` - 13https://github.com/miraheze/mediawiki/pull/4913 [22:16:42] [url] Page not found · GitHub · GitHub | github.com [22:16:43] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/cf061479c342...aa93ae142e33 [22:16:44] [url] Comparing cf061479c342...aa93ae142e33 · miraheze/mediawiki · GitHub | github.com [22:16:45] [02miraheze/mediawiki] 07dependabot[bot] 03aa93ae1 - Bump extensions/Thanks from `06f0c44` to `9421bd5` (#4913) [22:16:46] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Thanks-9421bd5 [22:16:48] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Thanks-9421bd5 - 13https://github.com/miraheze/mediawiki [22:16:48] [url] Page not found · GitHub · GitHub | github.com [22:17:20] [02mediawiki] 07RhinosF1 closed pull request 03#4912: Bump extensions/WikibaseQualityConstraints from `9752e3d` to `7b9ee7f` - 13https://github.com/miraheze/mediawiki/pull/4912 [22:17:20] [url] Page not found · GitHub · GitHub | github.com [22:17:21] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/WikibaseQualityConstraints-7b9ee7f [22:17:23] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/aa93ae142e33...23c414356f1a [22:17:23] [url] Comparing aa93ae142e33...23c414356f1a · miraheze/mediawiki · GitHub | github.com [22:17:24] [02miraheze/mediawiki] 07dependabot[bot] 0323c4143 - Bump extensions/WikibaseQualityConstraints from `9752e3d` to `7b9ee7f` (#4912) [22:17:26] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/WikibaseQualityConstraints-7b9ee7f - 13https://github.com/miraheze/mediawiki [22:17:26] ... [22:17:29] [02mediawiki] 07RhinosF1 closed pull request 03#4910: Bump extensions/CiteThisPage from `a6721da` to `67239de` - 13https://github.com/miraheze/mediawiki/pull/4910 [22:17:30] [url] Page not found · GitHub · GitHub | github.com [22:17:31] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/23c414356f1a...e8669854e996 [22:17:32] [url] Comparing 23c414356f1a...e8669854e996 · miraheze/mediawiki · GitHub | github.com [22:17:32] [02miraheze/mediawiki] 07dependabot[bot] 03e866985 - Bump extensions/CiteThisPage from `a6721da` to `67239de` (#4910) [22:17:34] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/CiteThisPage-67239de - 13https://github.com/miraheze/mediawiki [22:17:34] [url] Page not found · GitHub · GitHub | github.com [22:17:35] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/CiteThisPage-67239de [22:17:38] [02mediawiki] 07RhinosF1 closed pull request 03#4908: Bump extensions/ContactPage from `d5a4827` to `fbd203c` - 13https://github.com/miraheze/mediawiki/pull/4908 [22:17:39] [url] Page not found · GitHub · GitHub | github.com [22:17:40] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/ContactPage-fbd203c - 13https://github.com/miraheze/mediawiki [22:17:40] ... [22:17:41] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/ContactPage-fbd203c [22:17:43] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/e8669854e996...9f15b0983cd2 [22:17:43] [url] Comparing e8669854e996...9f15b0983cd2 · miraheze/mediawiki · GitHub | github.com [22:17:44] [02miraheze/mediawiki] 07dependabot[bot] 039f15b09 - Bump extensions/ContactPage from `d5a4827` to `fbd203c` (#4908) [22:17:48] [02mediawiki] 07RhinosF1 closed pull request 03#4907: Bump extensions/Cite from `a78f570` to `797d179` - 13https://github.com/miraheze/mediawiki/pull/4907 [22:17:49] [url] Page not found · GitHub · GitHub | github.com [22:17:50] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/9f15b0983cd2...c5d050517fe7 [22:17:51] [url] Comparing 9f15b0983cd2...c5d050517fe7 · miraheze/mediawiki · GitHub | github.com [22:17:51] [02miraheze/mediawiki] 07dependabot[bot] 03c5d0505 - Bump extensions/Cite from `a78f570` to `797d179` (#4907) [22:17:53] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Cite-797d179 [22:17:54] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Cite-797d179 - 13https://github.com/miraheze/mediawiki [22:17:55] [url] Page not found · GitHub · GitHub | github.com [22:17:58] [02mediawiki] 07RhinosF1 closed pull request 03#4906: Bump extensions/Flow from `fd89ee0` to `8643327` - 13https://github.com/miraheze/mediawiki/pull/4906 [22:17:59] [url] Page not found · GitHub · GitHub | github.com [22:18:00] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Flow-8643327 - 13https://github.com/miraheze/mediawiki [22:18:00] ... [22:18:02] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/c5d050517fe7...28cdb84bf311 [22:18:02] [url] Comparing c5d050517fe7...28cdb84bf311 · miraheze/mediawiki · GitHub | github.com [22:18:03] [02miraheze/mediawiki] 07dependabot[bot] 0328cdb84 - Bump extensions/Flow from `fd89ee0` to `8643327` (#4906) [22:18:04] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Flow-8643327 [22:18:12] [02mediawiki] 07RhinosF1 closed pull request 03#4905: Bump extensions/DiscussionTools from `7209e5f` to `2fcc0eb` - 13https://github.com/miraheze/mediawiki/pull/4905 [22:18:12] [url] Page not found · GitHub · GitHub | github.com [22:18:13] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/DiscussionTools-2fcc0eb - 13https://github.com/miraheze/mediawiki [22:18:14] [url] Page not found · GitHub · GitHub | github.com [22:18:15] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/DiscussionTools-2fcc0eb [22:18:16] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/28cdb84bf311...307c1a55941d [22:18:17] [url] Comparing 28cdb84bf311...307c1a55941d · miraheze/mediawiki · GitHub | github.com [22:18:18] [02miraheze/mediawiki] 07dependabot[bot] 03307c1a5 - Bump extensions/DiscussionTools from `7209e5f` to `2fcc0eb` (#4905) [22:18:22] [02mediawiki] 07RhinosF1 closed pull request 03#4904: Bump extensions/DismissableSiteNotice from `29bb34c` to `f88ae6c` - 13https://github.com/miraheze/mediawiki/pull/4904 [22:18:22] [url] Page not found · GitHub · GitHub | github.com [22:18:23] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/DismissableSiteNotice-f88ae6c [22:18:25] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/DismissableSiteNotice-f88ae6c - 13https://github.com/miraheze/mediawiki [22:18:25] ... [22:18:26] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/307c1a55941d...01c7dbbc3170 [22:18:27] [url] Comparing 307c1a55941d...01c7dbbc3170 · miraheze/mediawiki · GitHub | github.com [22:18:28] [02miraheze/mediawiki] 07dependabot[bot] 0301c7dbb - Bump extensions/DismissableSiteNotice from `29bb34c` to `f88ae6c` (#4904) [22:18:29] [02mediawiki] 07RhinosF1 closed pull request 03#4903: Bump extensions/CommonsMetadata from `f06ab37` to `cc10042` - 13https://github.com/miraheze/mediawiki/pull/4903 [22:18:30] [url] Page not found · GitHub · GitHub | github.com [22:18:31] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/01c7dbbc3170...c96cb5a26869 [22:18:31] [url] Comparing 01c7dbbc3170...c96cb5a26869 · miraheze/mediawiki · GitHub | github.com [22:18:32] [02miraheze/mediawiki] 07dependabot[bot] 03c96cb5a - Bump extensions/CommonsMetadata from `f06ab37` to `cc10042` (#4903) [22:18:34] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/CommonsMetadata-cc10042 - 13https://github.com/miraheze/mediawiki [22:18:34] [url] Page not found · GitHub · GitHub | github.com [22:18:35] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/CommonsMetadata-cc10042 [22:18:37] [02mediawiki] 07RhinosF1 closed pull request 03#4902: Bump extensions/AdminLinks from `f385200` to `3daf69d` - 13https://github.com/miraheze/mediawiki/pull/4902 [22:18:37] [url] Page not found · GitHub · GitHub | github.com [22:18:38] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/c96cb5a26869...593d0886b8df [22:18:39] [url] Comparing c96cb5a26869...593d0886b8df · miraheze/mediawiki · GitHub | github.com [22:18:40] [02miraheze/mediawiki] 07dependabot[bot] 03593d088 - Bump extensions/AdminLinks from `f385200` to `3daf69d` (#4902) [22:18:41] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/AdminLinks-3daf69d - 13https://github.com/miraheze/mediawiki [22:18:42] [url] Page not found · GitHub · GitHub | github.com [22:18:43] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/AdminLinks-3daf69d [22:18:44] [02mediawiki] 07RhinosF1 closed pull request 03#4901: Bump extensions/CheckUser from `b2255f3` to `72187a6` - 13https://github.com/miraheze/mediawiki/pull/4901 [22:18:45] [url] Page not found · GitHub · GitHub | github.com [22:18:46] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/593d0886b8df...7296412e2fa0 [22:18:46] [url] Comparing 593d0886b8df...7296412e2fa0 · miraheze/mediawiki · GitHub | github.com [22:18:47] [02miraheze/mediawiki] 07dependabot[bot] 037296412 - Bump extensions/CheckUser from `b2255f3` to `72187a6` (#4901) [22:18:49] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/CheckUser-72187a6 - 13https://github.com/miraheze/mediawiki [22:18:49] ... [22:18:50] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/CheckUser-72187a6 [22:19:07] PROBLEM - test101 APT on test101 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:19:22] [02mediawiki] 07RhinosF1 closed pull request 03#4899: Bump extensions/FlaggedRevs from `796798a` to `f3a8ffc` - 13https://github.com/miraheze/mediawiki/pull/4899 [22:19:22] [url] Page not found · GitHub · GitHub | github.com [22:19:23] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/FlaggedRevs-f3a8ffc [22:19:25] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/7296412e2fa0...55058080807d [22:19:26] [url] Comparing 7296412e2fa0...55058080807d · miraheze/mediawiki · GitHub | github.com [22:19:26] [02miraheze/mediawiki] 07dependabot[bot] 035505808 - Bump extensions/FlaggedRevs from `796798a` to `f3a8ffc` (#4899) [22:19:28] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/FlaggedRevs-f3a8ffc - 13https://github.com/miraheze/mediawiki [22:19:28] [url] Page not found · GitHub · GitHub | github.com [22:19:29] [02mediawiki] 07RhinosF1 closed pull request 03#4898: Bump extensions/ProofreadPage from `6878509` to `a4aedd2` - 13https://github.com/miraheze/mediawiki/pull/4898 [22:19:30] [url] Page not found · GitHub · GitHub | github.com [22:19:31] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/ProofreadPage-a4aedd2 - 13https://github.com/miraheze/mediawiki [22:19:31] ... [22:19:32] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/ProofreadPage-a4aedd2 [22:19:34] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/55058080807d...910bdda9f44a [22:19:35] [url] Comparing 55058080807d...910bdda9f44a · miraheze/mediawiki · GitHub | github.com [22:19:35] [02miraheze/mediawiki] 07dependabot[bot] 03910bdda - Bump extensions/ProofreadPage from `6878509` to `a4aedd2` (#4898) [22:19:39] [02mediawiki] 07RhinosF1 closed pull request 03#4896: Bump extensions/CentralNotice from `d3d1795` to `794de94` - 13https://github.com/miraheze/mediawiki/pull/4896 [22:19:40] [url] Page not found · GitHub · GitHub | github.com [22:19:41] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/910bdda9f44a...ecce16ee1ce3 [22:19:41] [url] Comparing 910bdda9f44a...ecce16ee1ce3 · miraheze/mediawiki · GitHub | github.com [22:19:42] [02miraheze/mediawiki] 07dependabot[bot] 03ecce16e - Bump extensions/CentralNotice from `d3d1795` to `794de94` (#4896) [22:19:44] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/CentralNotice-794de94 - 13https://github.com/miraheze/mediawiki [22:19:44] [url] Page not found · GitHub · GitHub | github.com [22:19:45] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/CentralNotice-794de94 [22:19:49] [02mediawiki] 07RhinosF1 closed pull request 03#4895: Bump extensions/WikiEditor from `1efc3d8` to `f99f482` - 13https://github.com/miraheze/mediawiki/pull/4895 [22:19:49] [url] Page not found · GitHub · GitHub | github.com [22:19:50] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/WikiEditor-f99f482 - 13https://github.com/miraheze/mediawiki [22:19:51] ... [22:19:52] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/ecce16ee1ce3...96c7ca6e0101 [22:19:53] [url] Comparing ecce16ee1ce3...96c7ca6e0101 · miraheze/mediawiki · GitHub | github.com [22:19:54] [02miraheze/mediawiki] 07dependabot[bot] 0396c7ca6 - Bump extensions/WikiEditor from `1efc3d8` to `f99f482` (#4895) [22:19:55] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/WikiEditor-f99f482 [22:19:58] [02mediawiki] 07RhinosF1 closed pull request 03#4894: Bump extensions/GlobalWatchlist from `d3cb716` to `d302af9` - 13https://github.com/miraheze/mediawiki/pull/4894 [22:19:58] [url] Page not found · GitHub · GitHub | github.com [22:19:59] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/96c7ca6e0101...ff80ee677332 [22:20:00] [url] Comparing 96c7ca6e0101...ff80ee677332 · miraheze/mediawiki · GitHub | github.com [22:20:01] [02miraheze/mediawiki] 07dependabot[bot] 03ff80ee6 - Bump extensions/GlobalWatchlist from `d3cb716` to `d302af9` (#4894) [22:20:02] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/GlobalWatchlist-d302af9 - 13https://github.com/miraheze/mediawiki [22:20:03] [url] Page not found · GitHub · GitHub | github.com [22:20:04] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/GlobalWatchlist-d302af9 [22:20:06] [02mediawiki] 07RhinosF1 closed pull request 03#4891: Bump extensions/WikimediaIncubator from `12b9543` to `33d1ba9` - 13https://github.com/miraheze/mediawiki/pull/4891 [22:20:07] [url] Page not found · GitHub · GitHub | github.com [22:20:08] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/WikimediaIncubator-33d1ba9 - 13https://github.com/miraheze/mediawiki [22:20:08] ... [22:20:09] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/WikimediaIncubator-33d1ba9 [22:20:11] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/ff80ee677332...66d06b7fda2c [22:20:12] [url] Comparing ff80ee677332...66d06b7fda2c · miraheze/mediawiki · GitHub | github.com [22:20:12] [02miraheze/mediawiki] 07dependabot[bot] 0366d06b7 - Bump extensions/WikimediaIncubator from `12b9543` to `33d1ba9` (#4891) [22:20:16] [02mediawiki] 07RhinosF1 closed pull request 03#4890: Bump extensions/MassMessage from `bd6de25` to `7e532a9` - 13https://github.com/miraheze/mediawiki/pull/4890 [22:20:17] [url] Page not found · GitHub · GitHub | github.com [22:20:18] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/MassMessage-7e532a9 [22:20:19] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/66d06b7fda2c...a3e10bb3435b [22:20:20] [url] Comparing 66d06b7fda2c...a3e10bb3435b · miraheze/mediawiki · GitHub | github.com [22:20:21] [02miraheze/mediawiki] 07dependabot[bot] 03a3e10bb - Bump extensions/MassMessage from `bd6de25` to `7e532a9` (#4890) [22:20:23] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/MassMessage-7e532a9 - 13https://github.com/miraheze/mediawiki [22:20:23] [url] Page not found · GitHub · GitHub | github.com [22:20:25] [02mediawiki] 07RhinosF1 closed pull request 03#4889: Bump extensions/Disambiguator from `53e8268` to `8cb42d8` - 13https://github.com/miraheze/mediawiki/pull/4889 [22:20:25] [url] Page not found · GitHub · GitHub | github.com [22:20:26] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/a3e10bb3435b...a1a20013c719 [22:20:27] [url] Comparing a3e10bb3435b...a1a20013c719 · miraheze/mediawiki · GitHub | github.com [22:20:28] [02miraheze/mediawiki] 07dependabot[bot] 03a1a2001 - Bump extensions/Disambiguator from `53e8268` to `8cb42d8` (#4889) [22:20:29] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Disambiguator-8cb42d8 - 13https://github.com/miraheze/mediawiki [22:20:30] [url] Page not found · GitHub · GitHub | github.com [22:20:31] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Disambiguator-8cb42d8 [22:20:32] [02mediawiki] 07RhinosF1 closed pull request 03#4888: Bump extensions/PageTriage from `5b81b50` to `d9440b4` - 13https://github.com/miraheze/mediawiki/pull/4888 [22:20:33] [url] Page not found · GitHub · GitHub | github.com [22:20:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb [22:20:34] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/a1a20013c719...97420e5569a5 [22:20:35] [url] Comparing a1a20013c719...97420e5569a5 · miraheze/mediawiki · GitHub | github.com [22:20:35] [02miraheze/mediawiki] 07dependabot[bot] 0397420e5 - Bump extensions/PageTriage from `5b81b50` to `d9440b4` (#4888) [22:20:37] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/PageTriage-d9440b4 - 13https://github.com/miraheze/mediawiki [22:20:37] ... [22:20:38] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/PageTriage-d9440b4 [22:20:40] [02mediawiki] 07RhinosF1 closed pull request 03#4886: Bump extensions/Gadgets from `5404436` to `4979fe1` - 13https://github.com/miraheze/mediawiki/pull/4886 [22:20:40] [url] Page not found · GitHub · GitHub | github.com [22:20:41] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Gadgets-4979fe1 [22:20:43] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/97420e5569a5...014bea1ce1a7 [22:20:44] [url] Comparing 97420e5569a5...014bea1ce1a7 · miraheze/mediawiki · GitHub | github.com [22:20:44] [02miraheze/mediawiki] 07dependabot[bot] 03014bea1 - Bump extensions/Gadgets from `5404436` to `4979fe1` (#4886) [22:20:46] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Gadgets-4979fe1 - 13https://github.com/miraheze/mediawiki [22:20:46] [url] Page not found · GitHub · GitHub | github.com [22:20:47] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.12, 21.33, 19.96 [22:20:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb [22:20:59] [02mediawiki] 07RhinosF1 closed pull request 03#4884: Bump vendor from `65b8217` to `56c2e8a` - 13https://github.com/miraheze/mediawiki/pull/4884 [22:20:59] [url] Page not found · GitHub · GitHub | github.com [22:21:00] [02mediawiki] 07dependabot[bot] commented on pull request 03#4884: Bump vendor from `65b8217` to `56c2e8a` - 13https://github.com/miraheze/mediawiki/pull/4884#issuecomment-1011503796 [22:21:01] [url] Bump vendor from `65b8217` to `56c2e8a` by dependabot[bot] · Pull Request #4884 · miraheze/mediawiki · GitHub | github.com [22:21:02] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/vendor-56c2e8a [22:21:03] PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 11 packages available for upgrade (1 critical updates). [22:21:03] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/vendor-56c2e8a - 13https://github.com/miraheze/mediawiki [22:21:04] [url] Page not found · GitHub · GitHub | github.com [22:21:16] [02mediawiki] 07RhinosF1 closed pull request 03#4885: Bump extensions/TwoColConflict from `783002e` to `eff532d` - 13https://github.com/miraheze/mediawiki/pull/4885 [22:21:17] [url] Page not found · GitHub · GitHub | github.com [22:21:18] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/014bea1ce1a7...bc2507f25c80 [22:21:18] [url] Comparing 014bea1ce1a7...bc2507f25c80 · miraheze/mediawiki · GitHub | github.com [22:21:19] [02miraheze/mediawiki] 07dependabot[bot] 03bc2507f - Bump extensions/TwoColConflict from `783002e` to `eff532d` (#4885) [22:21:21] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/TwoColConflict-eff532d [22:21:22] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/TwoColConflict-eff532d - 13https://github.com/miraheze/mediawiki [22:21:22] ... [22:21:42] PROBLEM - mw112 APT on mw112 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:21:53] [02mediawiki] 07RhinosF1 closed pull request 03#4883: Bump skins/Vector from `f7e05a9` to `e826214` - 13https://github.com/miraheze/mediawiki/pull/4883 [22:21:53] [url] Page not found · GitHub · GitHub | github.com [22:21:54] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/bc2507f25c80...c94e6f4320a9 [22:21:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.09, 3.95, 3.84 [22:21:55] [url] Comparing bc2507f25c80...c94e6f4320a9 · miraheze/mediawiki · GitHub | github.com [22:21:56] [02miraheze/mediawiki] 07dependabot[bot] 03c94e6f4 - Bump skins/Vector from `f7e05a9` to `e826214` (#4883) [22:21:57] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/skins/Vector-e826214 - 13https://github.com/miraheze/mediawiki [22:21:58] [url] Page not found · GitHub · GitHub | github.com [22:21:59] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/skins/Vector-e826214 [22:22:03] [02mediawiki] 07RhinosF1 closed pull request 03#4882: Bump extensions/Kartographer from `819f44d` to `3ba56a9` - 13https://github.com/miraheze/mediawiki/pull/4882 [22:22:03] [url] Page not found · GitHub · GitHub | github.com [22:22:04] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/c94e6f4320a9...7d0b97c80f67 [22:22:05] [url] Comparing c94e6f4320a9...7d0b97c80f67 · miraheze/mediawiki · GitHub | github.com [22:22:05] PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [22:22:06] [02miraheze/mediawiki] 07dependabot[bot] 037d0b97c - Bump extensions/Kartographer from `819f44d` to `3ba56a9` (#4882) [22:22:07] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Kartographer-3ba56a9 - 13https://github.com/miraheze/mediawiki [22:22:08] [url] Page not found · GitHub · GitHub | github.com [22:22:09] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Kartographer-3ba56a9 [22:22:11] [02mediawiki] 07RhinosF1 closed pull request 03#4881: Bump extensions/Renameuser from `cbc7547` to `d112548` - 13https://github.com/miraheze/mediawiki/pull/4881 [22:22:11] [url] Page not found · GitHub · GitHub | github.com [22:22:12] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Renameuser-d112548 [22:22:14] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/7d0b97c80f67...691d2b245d0e [22:22:14] [url] Comparing 7d0b97c80f67...691d2b245d0e · miraheze/mediawiki · GitHub | github.com [22:22:15] [02miraheze/mediawiki] 07dependabot[bot] 03691d2b2 - Bump extensions/Renameuser from `cbc7547` to `d112548` (#4881) [22:22:17] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Renameuser-d112548 - 13https://github.com/miraheze/mediawiki [22:22:17] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:22:17] ... [22:22:19] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:22:19] [02mediawiki] 07RhinosF1 closed pull request 03#4879: Bump extensions/BetaFeatures from `a8d6bdb` to `2820d7f` - 13https://github.com/miraheze/mediawiki/pull/4879 [22:22:20] [url] Page not found · GitHub · GitHub | github.com [22:22:21] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/BetaFeatures-2820d7f [22:22:22] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/BetaFeatures-2820d7f - 13https://github.com/miraheze/mediawiki [22:22:22] ... [22:22:24] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/691d2b245d0e...1dbeffca4e46 [22:22:24] [url] Comparing 691d2b245d0e...1dbeffca4e46 · miraheze/mediawiki · GitHub | github.com [22:22:25] [02miraheze/mediawiki] 07dependabot[bot] 031dbeffc - Bump extensions/BetaFeatures from `a8d6bdb` to `2820d7f` (#4879) [22:22:26] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:22:28] [02mediawiki] 07RhinosF1 closed pull request 03#4876: Bump extensions/Graph from `676ba72` to `fc4eb2f` - 13https://github.com/miraheze/mediawiki/pull/4876 [22:22:28] [url] Page not found · GitHub · GitHub | github.com [22:22:29] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/1dbeffca4e46...dd8b1dabc04c [22:22:30] [url] Comparing 1dbeffca4e46...dd8b1dabc04c · miraheze/mediawiki · GitHub | github.com [22:22:31] [02miraheze/mediawiki] 07dependabot[bot] 03dd8b1da - Bump extensions/Graph from `676ba72` to `fc4eb2f` (#4876) [22:22:32] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Graph-fc4eb2f [22:22:34] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Graph-fc4eb2f - 13https://github.com/miraheze/mediawiki [22:22:34] [url] Page not found · GitHub · GitHub | github.com [22:22:34] PROBLEM - db11 APT on db11 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (1 critical updates). [22:22:36] [02mediawiki] 07RhinosF1 closed pull request 03#4874: Bump extensions/GlobalUsage from `7071682` to `87146f0` - 13https://github.com/miraheze/mediawiki/pull/4874 [22:22:37] [url] Page not found · GitHub · GitHub | github.com [22:22:38] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/GlobalUsage-87146f0 - 13https://github.com/miraheze/mediawiki [22:22:38] [url] Page not found · GitHub · GitHub | github.com [22:22:39] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/dd8b1dabc04c...4376385abe4e [22:22:40] [url] Comparing dd8b1dabc04c...4376385abe4e · miraheze/mediawiki · GitHub | github.com [22:22:41] [02miraheze/mediawiki] 07dependabot[bot] 034376385 - Bump extensions/GlobalUsage from `7071682` to `87146f0` (#4874) [22:22:42] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/GlobalUsage-87146f0 [22:22:44] [02mediawiki] 07RhinosF1 closed pull request 03#4873: Bump extensions/StandardDialogs from `c116646` to `f601fed` - 13https://github.com/miraheze/mediawiki/pull/4873 [22:22:44] [url] Page not found · GitHub · GitHub | github.com [22:22:45] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/4376385abe4e...52dc5539eea6 [22:22:46] [url] Comparing 4376385abe4e...52dc5539eea6 · miraheze/mediawiki · GitHub | github.com [22:22:47] [02miraheze/mediawiki] 07dependabot[bot] 0352dc553 - Bump extensions/StandardDialogs from `c116646` to `f601fed` (#4873) [22:22:48] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/StandardDialogs-f601fed - 13https://github.com/miraheze/mediawiki [22:22:49] ... [22:22:50] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/StandardDialogs-f601fed [22:22:51] [02mediawiki] 07RhinosF1 closed pull request 03#4871: Bump extensions/TimedMediaHandler from `ad5c67f` to `16976ec` - 13https://github.com/miraheze/mediawiki/pull/4871 [22:22:52] [url] Page not found · GitHub · GitHub | github.com [22:22:53] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/52dc5539eea6...8a845aa817d0 [22:22:54] [url] Comparing 52dc5539eea6...8a845aa817d0 · miraheze/mediawiki · GitHub | github.com [22:22:54] [02miraheze/mediawiki] 07dependabot[bot] 038a845aa - Bump extensions/TimedMediaHandler from `ad5c67f` to `16976ec` (#4871) [22:22:56] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/TimedMediaHandler-16976ec [22:22:57] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/TimedMediaHandler-16976ec - 13https://github.com/miraheze/mediawiki [22:22:58] [url] Page not found · GitHub · GitHub | github.com [22:22:59] PROBLEM - mw13 APT on mw13 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [22:23:02] [02mediawiki] 07RhinosF1 closed pull request 03#4870: Bump extensions/Video from `c5e1d1a` to `1713c47` - 13https://github.com/miraheze/mediawiki/pull/4870 [22:23:03] [url] Page not found · GitHub · GitHub | github.com [22:23:04] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Video-1713c47 [22:23:05] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Video-1713c47 - 13https://github.com/miraheze/mediawiki [22:23:06] [url] Page not found · GitHub · GitHub | github.com [22:23:07] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/8a845aa817d0...53420a36cdfc [22:23:08] [url] Comparing 8a845aa817d0...53420a36cdfc · miraheze/mediawiki · GitHub | github.com [22:23:08] [02miraheze/mediawiki] 07dependabot[bot] 0353420a3 - Bump extensions/Video from `c5e1d1a` to `1713c47` (#4870) [22:23:24] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:23:28] [02mediawiki] 07RhinosF1 closed pull request 03#4868: Bump extensions/MatomoAnalytics from `0424557` to `ba5a4c8` - 13https://github.com/miraheze/mediawiki/pull/4868 [22:23:28] [url] Page not found · GitHub · GitHub | github.com [22:23:29] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/53420a36cdfc...a88d470f91ae [22:23:30] [url] Comparing 53420a36cdfc...a88d470f91ae · miraheze/mediawiki · GitHub | github.com [22:23:31] [02miraheze/mediawiki] 07dependabot[bot] 03a88d470 - Bump extensions/MatomoAnalytics from `0424557` to `ba5a4c8` (#4868) [22:23:32] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/MatomoAnalytics-ba5a4c8 - 13https://github.com/miraheze/mediawiki [22:23:33] ... [22:23:34] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/MatomoAnalytics-ba5a4c8 [22:23:35] [02mediawiki] 07RhinosF1 closed pull request 03#4867: Bump extensions/Popups from `dc1a397` to `669cc18` - 13https://github.com/miraheze/mediawiki/pull/4867 [22:23:36] [url] Page not found · GitHub · GitHub | github.com [22:23:37] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Popups-669cc18 - 13https://github.com/miraheze/mediawiki [22:23:37] ... [22:23:38] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Popups-669cc18 [22:23:40] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/a88d470f91ae...69da8cb8f823 [22:23:40] [url] Comparing a88d470f91ae...69da8cb8f823 · miraheze/mediawiki · GitHub | github.com [22:23:41] [02miraheze/mediawiki] 07dependabot[bot] 0369da8cb - Bump extensions/Popups from `dc1a397` to `669cc18` (#4867) [22:23:43] [02mediawiki] 07RhinosF1 closed pull request 03#4866: Bump extensions/Babel from `9e55da4` to `5b01abf` - 13https://github.com/miraheze/mediawiki/pull/4866 [22:23:43] [url] Page not found · GitHub · GitHub | github.com [22:23:44] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Babel-5b01abf [22:23:46] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/69da8cb8f823...59fe32da54fc [22:23:46] [url] Comparing 69da8cb8f823...59fe32da54fc · miraheze/mediawiki · GitHub | github.com [22:23:47] [02miraheze/mediawiki] 07dependabot[bot] 0359fe32d - Bump extensions/Babel from `9e55da4` to `5b01abf` (#4866) [22:23:49] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Babel-5b01abf - 13https://github.com/miraheze/mediawiki [22:23:49] [url] Page not found · GitHub · GitHub | github.com [22:23:50] [02mediawiki] 07RhinosF1 closed pull request 03#4865: Bump skins/Nimbus from `7b47d15` to `8669188` - 13https://github.com/miraheze/mediawiki/pull/4865 [22:23:51] [url] Page not found · GitHub · GitHub | github.com [22:23:52] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/skins/Nimbus-8669188 [22:23:53] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/skins/Nimbus-8669188 - 13https://github.com/miraheze/mediawiki [22:23:54] [url] Page not found · GitHub · GitHub | github.com [22:23:55] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/59fe32da54fc...7fb06abb3a70 [22:23:55] [url] Comparing 59fe32da54fc...7fb06abb3a70 · miraheze/mediawiki · GitHub | github.com [22:23:56] [02miraheze/mediawiki] 07dependabot[bot] 037fb06ab - Bump skins/Nimbus from `7b47d15` to `8669188` (#4865) [22:24:01] PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [22:24:17] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:25:21] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:26:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.81, 7.45, 7.88 [22:26:48] PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:26:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:26:58] CosmicAlpha: deploying now [22:27:15] !log [rhinos@test3] starting deploy of {'world': True, 'l10n': True} to skip [22:27:29] !log [rhinos@test101] starting deploy of {'world': True, 'l10n': True} to skip [22:27:34] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.783 second response time [22:27:37] !log [rhinos@mwtask111] starting deploy of {'world': True, 'l10n': True} to scsvg [22:28:06] Thanks! [22:28:06] PROBLEM - cloud5 APT on cloud5 is CRITICAL: APT CRITICAL: 11 packages available for upgrade (1 critical updates). [22:28:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:28:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.43, 7.89, 8.00 [22:28:25] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.010 second response time [22:28:29] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.012 second response time [22:28:30] CosmicAlpha: will do prod last for safety [22:28:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.36, 19.70, 19.82 [22:28:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:28:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:28:44] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.275 second response time [22:29:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:29:20] PROBLEM - mw12 APT on mw12 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [22:29:26] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.458 second response time [22:29:43] CosmicAlpha: can you poke apt [22:29:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.77, 3.73 [22:30:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.30, 7.50, 7.85 [22:31:49] CosmicAlpha: you seen "fatal: exec 'rev-list': cd to 'view/lib/wikibase-data-values-value-view' failed: No such file or directory" before [22:31:53] only on SCSVG [22:31:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.78, 3.74 [22:32:40] nope test3 too [22:34:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.63, 7.74, 7.84 [22:35:45] !log [rhinos@mw11] starting deploy of {'world': True, 'l10n': True} to ovlon [22:35:49] !log rhinos@mw11:/srv/mediawiki-staging/w/extensions/Wikibase$ sudo -u www-data git submodule update --init [22:36:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:36:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:36:48] RhinosF1: mobile right now, can be on PC in about 30 minutes if necessary. [22:37:27] CosmicAlpha: think i fixed [22:37:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.17, 3.92, 3.80 [22:38:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:38:46] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.08, 20.88, 19.66 [22:39:29] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [22:39:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.99, 3.95, 3.83 [22:40:55] !log [rhinos@test101] finished deploy of {'world': True, 'l10n': True} to skip - SUCCESS in 805s [22:41:16] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:41:34] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:41:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:41:38] !log same command on rest of stages [22:41:54] !log [rhinos@test3] finished deploy of {'world': True, 'l10n': True} to skip - SUCCESS in 878s [22:42:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:42:16] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:42:23] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:43:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:43:53] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:44:20] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.571 second response time [22:44:27] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.607 second response time [22:44:33] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.89, 20.07, 19.77 [22:45:13] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:45:29] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.063 second response time [22:45:44] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.370 second response time [22:45:53] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.522 second response time [22:46:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:46:48] !log [rhinos@mwtask111] finished deploy of {'world': True, 'l10n': True} to scsvg - SUCCESS in 1150s [22:46:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:47:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.25, 7.26, 7.94 [22:49:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.05, 7.60, 7.98 [22:49:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.82, 3.75, 3.64 [22:50:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [22:50:53] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:50:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb [22:51:01] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:08] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:15] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:51:17] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.33, 3.40, 3.52 [22:52:50] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.008 second response time [22:52:57] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.042 second response time [22:53:07] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.209 second response time [22:53:14] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.679 second response time [22:53:18] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.313 second response time [22:54:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.06, 24.22, 20.98 [22:55:45] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:56:52] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:57:00] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:57:24] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:45] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:59:55] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.469 second response time [22:59:56] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.762 second response time [22:59:57] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 8.12, 4.56, 3.84 [23:00:01] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:02] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:02] !log [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg [23:00:03] !log [@mw11] starting deploy of {'l10nupdate': True} to ovlon [23:00:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.28, 23.59, 21.92 [23:00:45] !log [rhinos@mw11] finished deploy of {'world': True, 'l10n': True} to ovlon - SUCCESS in 1499s [23:00:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:59] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.658 second response time [23:01:06] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.190 second response time [23:01:36] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.293 second response time [23:01:53] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:01:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.01, 3.92, 3.70 [23:02:08] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.05, 22.81, 20.92 [23:02:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:34] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:02:42] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:03:01] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:03:06] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:03:14] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:03:18] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:03:23] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:03:54] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:04:03] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.01, 23.17, 21.27 [23:04:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:04:13] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:04:30] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.025 second response time [23:04:39] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:04:42] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:04:45] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:04:58] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:05:55] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.384 second response time [23:05:59] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.06, 22.57, 21.29 [23:06:17] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.082 second response time [23:06:38] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.071 second response time [23:06:45] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.852 second response time [23:06:46] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.371 second response time [23:06:47] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.320 second response time [23:06:49] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.395 second response time [23:06:54] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.608 second response time [23:07:02] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.743 second response time [23:07:06] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.424 second response time [23:07:22] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:07:26] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.868 second response time [23:07:27] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.133 second response time [23:07:34] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:07:37] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.728 second response time [23:07:50] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:07:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.71, 3.11, 3.39 [23:08:01] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:08:07] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw12 [23:08:32] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw12 [23:09:08] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.825 second response time [23:09:24] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.516 second response time [23:09:37] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.475 second response time [23:10:09] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 18 backends are healthy [23:10:32] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 18 backends are healthy [23:10:46] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:11:17] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:11:28] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:11:29] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:12:03] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:12:12] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.012 second response time [23:12:15] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:12:30] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:12:49] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.211 second response time [23:13:19] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.866 second response time [23:13:20] !log [@test101] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 798s [23:13:26] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.392 second response time [23:13:33] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.079 second response time [23:13:46] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:13:47] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:13:50] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:04] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:06] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:23] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:15:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:15:25] !log [@mwtask111] finished deploy of {'l10nupdate': True} to scsvg - SUCCESS in 922s [23:15:33] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:15:36] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.08, 19.62, 20.35 [23:15:52] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.768 second response time [23:16:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:16:06] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.495 second response time [23:16:09] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.687 second response time [23:16:11] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.401 second response time [23:16:17] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.232 second response time [23:16:20] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.187 second response time [23:16:22] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.120 second response time [23:16:23] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 1.399 second response time [23:16:38] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.352 second response time [23:17:30] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.130 second response time [23:17:53] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.314 second response time [23:17:56] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.423 second response time [23:20:15] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:20:32] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:20:41] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:20:43] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:21:01] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:21:20] [02puppet] 07RhinosF1 edited pull request 03#2241: MediaWiki: Switchover to new infra - 13https://github.com/miraheze/puppet/pull/2241 [23:21:20] [url] Page not found · GitHub · GitHub | github.com [23:22:02] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:22:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.71, 21.63, 21.14 [23:22:27] [02mw-config] 07RhinosF1 assigned pull request 03#4353: DC-Switch: Stop uploads - 13https://github.com/miraheze/mw-config/pull/4353 [23:22:27] ... [23:22:36] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:22:45] [02mw-config] 07RhinosF1 edited pull request 03#4353: [19:45] DC-Switch: Stop uploads - 13https://github.com/miraheze/mw-config/pull/4353 [23:22:45] ... [23:22:52] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:22:58] [02mw-config] 07RhinosF1 assigned pull request 03#4355: DC-Switch: All wikis read only - 13https://github.com/miraheze/mw-config/pull/4355 [23:22:58] [url] Page not found · GitHub · GitHub | github.com [23:23:08] [02mw-config] 07RhinosF1 edited pull request 03#4355: [22:45] DC-Switch: All wikis read only - 13https://github.com/miraheze/mw-config/pull/4355 [23:23:08] ... [23:23:09] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:23:26] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:23:27] [02mw-config] 07RhinosF1 edited pull request 03#4356: [22:53] DC-Switch: Make SCSVG RW - 13https://github.com/miraheze/mw-config/pull/4356 [23:23:27] [url] Page not found · GitHub · GitHub | github.com [23:23:29] [02mw-config] 07RhinosF1 assigned pull request 03#4356: [22:53] DC-Switch: Make SCSVG RW - 13https://github.com/miraheze/mw-config/pull/4356 [23:23:29] [url] Page not found · GitHub · GitHub | github.com [23:23:41] [02mw-config] 07RhinosF1 edited pull request 03#4366: [22:53] DC-Switch: Switch to mon111 for IRC RecentChanges feed - 13https://github.com/miraheze/mw-config/pull/4366 [23:23:42] [url] Page not found · GitHub · GitHub | github.com [23:23:43] [02mw-config] 07RhinosF1 assigned pull request 03#4366: [22:53] DC-Switch: Switch to mon111 for IRC RecentChanges feed - 13https://github.com/miraheze/mw-config/pull/4366 [23:23:54] [02mw-config] 07RhinosF1 assigned pull request 03#4354: DC-Switch: Disable wiki creation (trust Stews for rename) - 13https://github.com/miraheze/mw-config/pull/4354 [23:24:01] [02mw-config] 07RhinosF1 edited pull request 03#4354: [22:15] DC-Switch: Disable wiki creation (trust Stews for rename) - 13https://github.com/miraheze/mw-config/pull/4354 [23:24:21] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/1af0eb658fd7...912e8ee64f34 [23:24:21] [url] Comparing 1af0eb658fd7...912e8ee64f34 · miraheze/RemovePII · GitHub | github.com [23:24:22] [02miraheze/RemovePII] 07Universal-Omega 03912e8ee - Fix [23:24:24] [02RemovePII] 07Universal-Omega synchronize pull request 03#43: Create GeneratePII maintenance script - 13https://github.com/miraheze/RemovePII/pull/43 [23:24:24] [url] Page not found · GitHub · GitHub | github.com [23:24:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.50, 21.11, 21.01 [23:24:42] [02puppet] 07RhinosF1 edited pull request 03#2241: [22:50] MediaWiki: Switchover to new infra - 13https://github.com/miraheze/puppet/pull/2241 [23:24:43] ... [23:25:06] [02dns] 07RhinosF1 edited pull request 03#241: [22:38] betaheze: reduce TTL to 10 seconds - 13https://github.com/miraheze/dns/pull/241 [23:25:06] ... [23:25:25] [02dns] 07RhinosF1 edited pull request 03#238: [22:48] betaheze: use new test server - 13https://github.com/miraheze/dns/pull/238 [23:25:26] [url] Page not found · GitHub · GitHub | github.com [23:25:44] [02dns] 07RhinosF1 edited pull request 03#233: [22:58] mediawiki: failover deployment & mwtask - 13https://github.com/miraheze/dns/pull/233 [23:25:44] ... [23:26:52] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.538 second response time [23:29:12] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 3.927 second response time [23:29:19] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.037 second response time [23:29:20] miraheze/RemovePII - Universal-Omega the build passed. [23:29:37] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.122 second response time [23:30:32] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.370 second response time [23:30:42] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.773 second response time [23:30:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:30:55] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.328 second response time [23:31:04] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.315 second response time [23:31:11] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.427 second response time [23:31:18] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.006 second response time [23:31:40] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:34:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [23:35:13] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.40, 20.44, 19.82 [23:35:16] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:35:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:36:38] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 2196s [23:37:05] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:37:13] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 20.05, 20.31, 19.85 [23:37:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:37:35] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:38:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.24, 20.65, 20.54 [23:38:47] !log [@mw11] finished deploy of {'l10nupdate': True} to ovlon - SUCCESS in 2324s [23:39:07] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.692 second response time [23:39:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:39:26] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.255 second response time [23:39:35] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.732 second response time [23:40:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.55, 20.18, 20.36 [23:41:13] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.07, 21.36, 20.40 [23:41:33] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/912e8ee64f34...fffefb36c773 [23:41:34] [url] Comparing 912e8ee64f34...fffefb36c773 · miraheze/RemovePII · GitHub | github.com [23:41:35] [02miraheze/RemovePII] 07Universal-Omega 03fffefb3 - Update generatePII.php [23:41:36] [02RemovePII] 07Universal-Omega synchronize pull request 03#43: Create GeneratePII maintenance script - 13https://github.com/miraheze/RemovePII/pull/43 [23:41:37] [url] Page not found · GitHub · GitHub | github.com [23:41:47] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:42:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.97, 19.77, 20.19 [23:43:13] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:43:27] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:43:30] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:43:33] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:44:28] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:45:14] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.14, 19.65, 19.91 [23:45:33] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.062 second response time [23:45:59] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.355 second response time [23:46:22] miraheze/RemovePII - Universal-Omega the build passed. [23:47:17] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:48:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.20, 21.51, 20.82 [23:50:38] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.315 second response time [23:51:26] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.329 second response time [23:51:28] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.057 second response time [23:51:44] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.438 second response time [23:52:00] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.131 second response time [23:52:00] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:52:47] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:53:13] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.91, 20.54, 20.12 [23:54:04] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.820 second response time [23:54:53] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.282 second response time [23:55:13] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.40, 20.15, 20.03 [23:56:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.72, 19.72, 20.26 [23:56:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:56:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online