[00:01:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.43, 5.97, 5.01 [00:03:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.23, 7.27, 7.81 [00:03:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:03:34] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.91, 1.48, 1.04 [00:04:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:05:34] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.13, 1.32, 1.03 [00:08:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:10:50] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.29, 6.61, 7.82 [00:11:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.29, 7.50, 7.53 [00:12:38] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.57, 7.14, 7.04 [00:13:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.10, 7.29, 7.45 [00:14:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.70, 6.86, 7.90 [00:14:37] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.14, 7.03, 7.01 [00:16:47] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.92, 6.54, 7.71 [00:19:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.14, 5.76, 6.71 [00:20:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.84, 7.99, 7.34 [00:21:50] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 18.84, 21.03, 23.62 [00:21:55] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.07, 7.03, 7.95 [00:22:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.53, 7.11, 7.45 [00:22:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.32, 7.33, 7.18 [00:22:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.96, 5.03, 5.84 [00:24:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.60, 6.91, 7.31 [00:24:41] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.48, 5.41, 6.79 [00:24:49] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.40, 5.51, 6.63 [00:26:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.69, 8.14, 7.72 [00:28:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.11, 6.85, 7.30 [00:28:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.68, 6.25, 6.77 [00:28:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 2.42, 3.71, 5.04 [00:31:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.17, 3.54, 3.96 [00:35:16] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.29, 3.61, 3.87 [00:36:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.85, 6.19, 6.80 [00:37:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.34, 3.73 [00:37:40] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.48, 5.88, 6.67 [00:38:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.59, 4.80, 4.93 [00:40:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.06, 5.24, 5.07 [00:42:11] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.85, 7.13, 6.54 [00:44:08] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.63, 6.12, 6.23 [00:44:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:44:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.85, 5.77, 5.32 [00:45:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.42, 3.42, 3.56 [00:45:43] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.06, 21.96, 21.57 [00:46:34] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.50, 7.13, 6.90 [00:47:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.73, 3.19, 3.46 [00:47:42] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.21, 20.87, 21.22 [00:48:31] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.04, 7.82, 7.15 [00:48:50] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.18, 5.76, 5.40 [00:49:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.45, 3.52 [00:49:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:49:42] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.07, 22.59, 21.78 [00:49:58] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.77, 7.18, 6.56 [00:50:29] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.07, 7.61, 7.16 [00:50:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.11, 7.06, 6.84 [00:50:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb [00:51:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.03, 3.35, 3.48 [00:51:15] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:51:19] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.17, 7.18, 6.48 [00:51:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:51:41] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.26, 21.34, 21.41 [00:51:56] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.63, 6.98, 6.54 [00:52:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.24, 6.43, 6.63 [00:52:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.64, 6.96, 6.21 [00:52:42] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:52:50] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.60, 8.05, 6.67 [00:53:11] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.026 second response time [00:53:17] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.64, 7.64, 6.78 [00:53:53] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.31, 6.34, 6.36 [00:54:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.37, 6.04, 5.96 [00:54:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.39, 5.75, 5.58 [00:54:49] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.02, 6.61, 6.31 [00:55:17] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.94, 6.58, 6.49 [00:56:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:57:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.50, 4.15, 3.80 [00:57:39] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.88, 17.94, 19.90 [00:58:20] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.99, 6.17, 6.71 [00:58:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.77, 4.53, 5.10 [01:01:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.51, 3.40, 5.91 [01:02:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 12.25, 15.35, 22.73 [01:05:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.85, 6.27, 6.09 [01:06:09] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.85, 6.77, 6.73 [01:07:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.23, 3.95, 3.96 [01:07:33] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.54, 5.77, 5.92 [01:08:07] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.53, 6.59, 6.66 [01:09:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.52, 3.60, 4.99 [01:12:04] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.91, 7.17, 6.33 [01:13:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.52, 4.05, 3.97 [01:13:58] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.00, 7.50, 6.97 [01:14:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.47, 23.51, 22.75 [01:15:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.73, 3.85 [01:15:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 11.04, 6.75, 5.74 [01:15:53] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:15:56] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.53, 7.06, 6.90 [01:16:01] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.95, 6.24, 6.15 [01:16:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:16:36] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.29, 23.35, 20.93 [01:16:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.66, 5.77, 5.12 [01:17:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.66, 4.11, 4.01 [01:17:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.60, 6.83, 6.15 [01:17:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb [01:17:51] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 0.013 second response time [01:17:54] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.50, 8.01, 7.26 [01:18:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.82, 7.66, 6.81 [01:18:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.33, 5.83, 5.23 [01:19:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.56, 6.89, 6.33 [01:19:34] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.58, 6.70, 6.20 [01:19:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:19:52] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.05, 7.40, 7.14 [01:19:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.97, 7.19, 6.57 [01:20:34] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.25, 22.36, 21.17 [01:20:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.44, 22.28, 22.87 [01:21:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.28, 7.01, 6.42 [01:21:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:21:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.01, 5.39, 5.61 [01:22:34] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.86, 6.48, 6.50 [01:23:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.77, 6.93, 6.47 [01:24:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.26, 5.51, 5.20 [01:25:49] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.48, 4.28, 5.10 [01:27:21] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.72, 6.64, 6.48 [01:27:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.17, 6.87, 6.47 [01:27:52] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.54, 6.70, 6.69 [01:28:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.42, 4.97, 5.10 [01:28:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.33, 17.08, 20.02 [01:29:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.11, 5.97, 6.19 [01:31:40] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.73, 6.13, 6.70 [01:32:31] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.55, 19.10, 20.34 [01:49:29] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.05, 5.10, 4.83 [01:51:24] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.83, 5.14, 4.87 [01:53:19] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 4.70, 5.00, 4.85 [01:54:48] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.03, 2.87, 1.77 [01:58:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.96, 1.77, 1.56 [01:59:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.64, 6.19, 5.66 [02:00:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.90, 2.47, 1.90 [02:01:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.45, 6.39, 5.81 [02:02:22] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.45, 20.80, 19.78 [02:02:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.78, 1.86, 1.74 [02:04:21] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.73, 19.18, 19.31 [02:04:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.27, 3.58, 2.42 [02:08:50] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.51, 1.88, 1.98 [02:09:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.06, 3.51, 3.98 [02:10:22] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.70, 21.59, 20.29 [02:11:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.70, 3.99 [02:11:07] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 12.81, 8.53, 7.21 [02:11:19] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.87, 7.15, 6.03 [02:11:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.33, 6.50, 5.93 [02:11:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.33, 6.50, 5.93 [02:11:40] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 7.63, 6.19, 5.12 [02:11:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.22, 6.44, 5.56 [02:12:21] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 28.93, 24.18, 21.40 [02:12:51] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.77, 1.22, 1.68 [02:13:56] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.00, 5.81, 5.43 [02:15:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.26, 3.79, 3.98 [02:15:12] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.05, 6.72, 6.17 [02:15:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.87, 6.32, 6.05 [02:15:29] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 3.35, 5.47, 5.13 [02:16:20] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 16.30, 21.33, 21.03 [02:17:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.47, 7.34, 7.24 [02:17:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.89, 3.99 [02:19:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.79, 3.56, 3.86 [02:19:19] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 2.55, 4.27, 4.75 [02:20:18] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.15, 19.31, 20.22 [02:21:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.83, 4.25, 4.09 [02:22:54] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.27, 6.35, 6.79 [02:23:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.87, 3.65, 3.88 [02:37:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.63, 2.90, 3.31 [02:41:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.30, 3.36 [02:41:22] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.65, 4.98, 4.54 [02:43:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.65, 3.70, 3.49 [02:43:09] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.01, 21.14, 19.91 [02:44:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.97, 5.95, 5.55 [02:45:09] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.44, 19.80, 19.59 [02:45:11] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 4.29, 4.88, 4.60 [02:46:23] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.41, 6.91, 6.72 [02:46:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.60, 5.93, 5.59 [02:50:18] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.04, 6.36, 6.58 [02:56:04] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 31.26, 22.72, 20.52 [02:56:05] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.91, 6.14, 4.20 [02:56:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:56:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 7.24, 6.52, 5.48 [02:56:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.24, 22.49, 18.40 [02:57:07] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.92, 8.45, 7.37 [02:57:49] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.21, 7.71, 6.46 [02:58:00] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.66, 5.40, 4.16 [02:58:04] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.92, 22.12, 20.60 [02:58:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.84, 7.55, 6.52 [02:58:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.34, 20.67, 18.27 [02:59:05] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.08, 7.93, 7.31 [02:59:54] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.10, 4.17, 3.85 [03:00:33] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.31, 6.49, 6.27 [03:00:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.05, 4.86, 5.08 [03:00:55] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.86, 18.10, 17.61 [03:01:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.02, 3.83, 3.88 [03:01:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:01:46] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.01, 7.29, 6.51 [03:02:03] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.25, 19.83, 20.06 [03:03:44] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.16, 6.04, 6.14 [03:05:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.89, 3.87 [03:09:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.98, 3.66, 3.79 [03:12:36] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 8.00, 7.86, 6.78 [03:14:34] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.60, 6.78, 6.52 [03:15:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.07, 3.92, 3.82 [03:16:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.21, 7.76, 7.25 [03:18:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.47, 7.25, 7.13 [03:18:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.10, 5.86, 5.25 [03:19:05] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.42, 7.02, 6.45 [03:20:41] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 12.95, 9.43, 7.93 [03:20:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.97, 5.88, 5.32 [03:22:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 7.98, 6.37, 5.55 [03:22:57] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.34, 24.54, 21.99 [03:23:01] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.13, 7.79, 6.86 [03:24:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:24:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.23, 5.92, 5.49 [03:24:56] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.48, 23.19, 21.83 [03:24:58] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.75, 7.50, 6.88 [03:25:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.22, 7.52, 6.64 [03:25:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.51, 7.12, 6.40 [03:26:36] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.79, 7.94, 7.81 [03:26:55] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.70, 6.06, 6.42 [03:26:59] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.25, 6.73, 6.46 [03:27:02] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.30, 6.06, 6.09 [03:29:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:30:49] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.05, 7.34, 6.74 [03:32:46] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.99, 7.08, 6.73 [03:34:27] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.69, 7.86, 7.72 [03:34:43] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.61, 6.25, 6.47 [03:36:25] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.80, 7.37, 7.55 [03:36:52] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.03, 19.12, 20.37 [03:39:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.82, 3.63, 3.95 [03:42:18] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.27, 5.59, 6.67 [03:44:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.51, 4.29, 4.89 [03:47:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.52, 3.72 [03:48:09] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.57, 6.93, 6.89 [03:49:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.38, 3.64 [03:51:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.65, 4.21, 3.91 [03:51:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.22, 7.12, 6.29 [03:51:46] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.82, 21.09, 20.18 [03:53:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.91, 3.83 [03:53:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.00, 7.02, 6.35 [03:55:21] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.00, 6.04, 6.08 [03:55:39] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.31, 5.30, 5.01 [03:57:34] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.49, 4.54, 4.76 [03:57:44] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.30, 19.73, 20.06 [03:57:58] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.43, 6.12, 6.60 [04:01:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.62, 3.66 [04:03:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.35, 3.55 [04:05:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.35, 2.91, 3.36 [04:13:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.79, 6.20, 5.43 [04:15:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.69, 6.28, 5.55 [04:22:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.87, 4.82, 4.65 [04:24:50] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 10.32, 6.39, 5.22 [04:25:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.52, 3.37 [04:26:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.67, 5.88, 5.17 [04:28:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.21, 6.32, 5.44 [04:29:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.65, 3.28, 3.32 [04:30:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.01, 7.21, 5.90 [04:30:33] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.93, 7.42, 6.79 [04:32:31] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.94, 6.64, 6.59 [04:32:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.29, 6.22, 5.68 [04:33:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.45, 3.39 [04:34:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.25, 5.80, 5.52 [04:35:52] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.60, 3.20, 3.31 [04:36:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.74, 5.92, 5.58 [04:38:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.44, 5.47, 5.47 [04:39:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.36, 3.44, 3.41 [04:41:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.59, 3.46 [04:43:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.53, 3.45 [04:44:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.36, 5.10, 5.25 [04:45:47] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.04, 3.20, 3.33 [04:46:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.82, 5.20, 5.28 [04:47:11] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.89, 7.15, 6.75 [04:48:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.84, 4.60, 5.04 [04:49:09] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.43, 6.80, 6.66 [04:53:29] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.15, 20.76, 19.95 [04:53:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.87, 3.57 [04:53:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.84, 6.23, 5.71 [04:54:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.96, 5.50, 5.25 [04:55:58] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.56, 5.92, 5.66 [04:57:29] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.79, 19.63, 19.72 [04:58:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 7.26, 5.73, 5.35 [05:00:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.96, 5.51, 5.31 [05:02:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 7.21, 5.87, 5.45 [05:03:47] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.67, 7.44, 6.91 [05:04:18] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.56, 7.71, 6.49 [05:04:28] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.37, 22.31, 20.85 [05:04:47] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.07, 6.92, 6.23 [05:05:16] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.96, 6.78, 6.01 [05:05:45] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.87, 7.06, 6.86 [05:06:13] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.78, 6.67, 6.26 [05:06:27] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 18.65, 20.66, 20.42 [05:06:44] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.19, 6.32, 6.10 [05:06:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 3.86, 5.36, 5.40 [05:07:16] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.87, 5.71, 5.71 [05:07:42] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.35, 6.07, 6.52 [05:08:27] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.39, 19.21, 19.92 [05:08:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 2.69, 4.55, 5.10 [05:11:35] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.43, 3.04, 3.32 [05:15:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.96, 1.58, 1.18 [05:17:09] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.15, 1.39, 1.16 [05:21:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.13, 6.71, 6.51 [05:23:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.13, 3.34, 3.41 [05:23:40] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.14, 6.08, 6.30 [05:25:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.83, 3.21, 3.35 [05:37:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.95, 7.29, 5.70 [05:38:16] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.65, 19.12, 18.54 [05:38:29] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.15, 6.93, 6.59 [05:38:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.74, 5.80, 5.18 [05:39:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.30, 6.57, 5.63 [05:40:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.08, 7.42, 5.85 [05:41:13] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: No response from NTP server [05:42:24] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.92, 6.23, 6.39 [05:43:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.82, 7.09, 6.10 [05:44:14] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.64, 18.34, 18.54 [05:44:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.94, 6.80, 5.99 [05:45:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.51, 6.60, 6.02 [05:52:20] ns1 is down? [05:53:06] In theory it should fix itself [05:53:22] If not thats why we have ns2 [05:53:30] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.21, 6.92, 6.39 [05:53:32] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.002383261919 secs [05:53:40] Yep, it did! [05:55:28] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.32, 6.29, 6.21 [05:56:51] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.96, 5.04, 4.50 [05:58:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.47, 5.36, 4.66 [06:00:02] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.23, 8.03, 6.80 [06:00:04] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.33, 6.73, 5.89 [06:00:07] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.48, 22.75, 20.22 [06:00:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.47, 3.05 [06:01:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.45, 6.81, 6.40 [06:02:00] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.64, 6.96, 6.55 [06:02:02] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.98, 6.11, 5.76 [06:02:07] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 17.69, 20.72, 19.78 [06:02:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.23, 3.34, 3.05 [06:02:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.75, 5.26, 4.80 [06:03:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.69, 5.71, 6.05 [06:04:06] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.73, 19.49, 19.45 [06:05:04] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.96, 5.78, 3.61 [06:05:55] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.07, 7.40, 6.81 [06:07:04] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.09, 6.64, 4.20 [06:07:53] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.28, 6.90, 6.69 [06:08:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.47, 4.89, 4.86 [06:09:51] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.74, 6.79, 6.68 [06:13:45] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.48, 7.57, 6.99 [06:14:09] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 7.36, 8.09, 7.05 [06:16:07] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.18, 7.68, 7.01 [06:17:04] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.61, 7.52, 6.04 [06:18:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.88, 7.52, 6.31 [06:18:51] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.35, 5.03, 4.73 [06:19:04] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.42, 6.77, 5.94 [06:19:59] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.45, 22.06, 20.73 [06:20:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.42, 3.18 [06:20:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.63, 7.57, 6.48 [06:20:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.87, 4.45, 4.54 [06:22:00] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.65, 3.30, 3.17 [06:23:12] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 37.57, 22.24, 12.00 [06:23:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 3.04, 6.72, 7.20 [06:23:57] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.10, 6.07, 6.59 [06:23:57] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 13.22, 17.50, 19.25 [06:24:16] whoops [06:24:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.49, 5.81, 6.02 [06:24:55] tbh I'm going to clean up imported files later, hopefully in the end wiki won't take too much space [06:25:39] cleaning up wiki on fandom before migration isn't sufficient [06:26:12] maybe change some png to jpg [06:29:40] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.04, 5.82, 6.65 [06:33:04] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 2.11, 5.19, 7.54 [06:37:04] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 1.72, 3.36, 6.25 [06:38:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 13.94, 7.80, 5.30 [06:38:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.14, 16.92, 14.55 [06:38:57] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:39:00] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:39:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:39:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 8.00, 6.75, 6.45 [06:39:51] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.61, 21.95, 19.27 [06:40:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.58, 6.99, 6.01 [06:40:54] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.210 second response time [06:40:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.30, 16.45, 14.62 [06:40:56] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.041 second response time [06:41:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.67, 7.74, 6.83 [06:41:51] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.24, 21.78, 19.54 [06:42:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.67, 7.66, 6.38 [06:43:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.85, 8.00, 7.05 [06:44:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:44:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.04, 5.94, 5.91 [06:45:49] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.03, 19.42, 19.22 [06:46:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 3.85, 5.27, 5.17 [06:47:40] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.95, 5.93, 6.42 [06:48:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 4.28, 4.81, 5.00 [06:50:00] .tell RhinosF1 do we have a list of what remaining SQL patches need ran for the upgrade today? If not I can probably find them via which was ran on test3, but just wondering, so I can do that also. [06:50:00] CosmicAlpha: I'll pass that on when RhinosF1 is around. [07:00:00] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JMhd4 [07:00:02] [02miraheze/mediawiki] 07Reception123 0328b64e0 - Update DarkMode [07:31:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.52, 4.85, 4.33 [07:35:54] Hi RhinosF1 [07:36:52] PROBLEM - sarovia.graalmilitary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sarovia.graalmilitary.com' expires in 15 day(s) (Thu 23 Dec 2021 07:34:12 GMT +0000). [07:37:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.89, 4.23, 4.30 [07:38:14] Hi CosmicAlpha [07:38:14] [tell] RhinosF1: 2021-12-07 - 06:50:00GMT tell RhinosF1 do we have a list of what remaining SQL patches need ran for the upgrade today? If not I can probably find them via which was ran on test3, but just wondering, so I can do that also. [07:40:43] CosmicAlpha: I was gonna compare the list later I ran on SAL with what's in 137updates.sql [07:40:51] So feel free to [07:41:06] PROBLEM - lakehub.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lakehub.ga' expires in 15 day(s) (Thu 23 Dec 2021 07:32:57 GMT +0000). [07:43:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.80, 6.42, 5.11 [07:45:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.53, 5.38, 4.87 [07:46:43] RhinosF1: OK, I'm going to go to sleep now, I'll be back at around ~1500 or 1600 to finalise preparations for the upgrade at 1700. So I'll compare what you already ran in SAL, and what's in 137updates.sql then, and what's left needs ran I guess? [07:47:00] Ye [07:47:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.73, 4.70, 4.68 [07:48:09] RhinosF1: everything you did on test3 is logged right? (Just double verifying, I really want this to go smoothly for once, so don't want to miss something) [07:54:32] CosmicAlpha: yes everything was logged [07:56:04] RhinosF1: great! Thanks! [08:00:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JMjJ8 [08:00:32] [02miraheze/mw-config] 07Universal-Omega 0345510ff - Switch to post-1.37 upgrade sitenotice [08:00:33] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [08:00:35] [02mw-config] 07Universal-Omega opened pull request 03#4264: Switch to post-1.37 upgrade sitenotice - 13https://git.io/JMjJ4 [08:00:57] [02mw-config] 07Universal-Omega assigned pull request 03#4264: Switch to post-1.37 upgrade sitenotice - 13https://git.io/JMjJ4 [08:01:36] miraheze/mw-config - Universal-Omega the build passed. [08:22:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.18, 7.55, 5.46 [08:22:28] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.94, 6.43, 5.25 [08:22:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.61, 6.94, 5.12 [08:22:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.75, 7.27, 5.11 [08:22:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.49, 22.55, 19.36 [08:24:07] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.63, 6.96, 5.50 [08:24:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.60, 6.35, 5.13 [08:24:49] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.54, 6.17, 4.97 [08:24:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.88, 19.86, 18.79 [08:26:06] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.05, 6.62, 5.55 [08:26:16] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.49, 5.48, 5.15 [08:30:05] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.24, 5.32, 5.13 [08:32:00] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.30, 7.04, 6.06 [08:32:00] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.94, 5.30, 5.15 [08:33:54] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.07, 4.80, 4.98 [08:33:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.86, 6.32, 5.92 [08:43:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.65, 5.29, 4.91 [08:45:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.11, 4.43, 4.64 [08:59:24] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.01, 5.99, 5.21 [09:01:18] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.32, 4.99, 4.93 [09:22:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.36, 18.30, 15.58 [09:23:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.68, 5.41, 4.61 [09:24:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.62, 18.09, 15.82 [09:25:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.39, 5.00, 4.55 [09:33:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.74, 5.55, 4.90 [09:35:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.70, 5.01, 4.80 [09:49:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 14.84, 9.31, 6.21 [09:50:12] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 53.27, 32.89, 22.31 [09:50:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.67, 6.19, 5.23 [09:51:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:51:03] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.24, 6.64, 4.89 [09:51:45] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.83, 6.45, 4.90 [09:52:13] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.60, 7.45, 5.65 [09:52:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.41, 7.12, 5.15 [09:53:43] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.97, 6.65, 5.14 [09:54:11] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.92, 7.53, 5.92 [09:54:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.12, 6.17, 5.04 [09:54:35] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.06, 6.48, 5.59 [09:54:53] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.20, 6.15, 5.14 [09:55:40] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.45, 5.41, 4.86 [09:56:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:56:10] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.32, 5.99, 5.55 [09:56:14] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.006 second response time [09:56:45] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [09:56:58] PROBLEM - cp20 Varnish Backends on cp20 is WARNING: No backends detected. If this is an error, see readme.txt [09:57:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb [09:57:54] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb [09:59:43] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 14.97, 21.04, 22.08 [10:03:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.23, 4.69, 5.70 [10:05:25] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.05, 15.94, 19.50 [10:07:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.04, 3.57, 5.04 [10:09:27] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.99, 4.05, 2.14 [10:11:11] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.90, 19.79, 19.81 [10:11:51] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.75, 7.20, 6.19 [10:13:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:13:05] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 38.83, 25.89, 21.99 [10:15:16] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.16, 1.94, 1.75 [10:17:14] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.07, 1.67, 1.68 [10:17:51] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.44, 6.59, 5.51 [10:18:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:19:49] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.03, 7.34, 5.91 [10:20:05] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.12, 7.92, 6.37 [10:20:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.41, 6.57, 5.40 [10:21:09] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.88, 6.83, 5.92 [10:21:27] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.32, 7.65, 6.13 [10:21:48] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.59, 6.63, 5.83 [10:22:03] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.83, 7.01, 6.22 [10:22:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.32, 6.33, 5.45 [10:23:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.11, 6.58, 5.94 [10:25:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.20, 6.26, 5.93 [10:25:58] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.71, 6.58, 6.25 [10:29:03] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.35, 7.41, 6.40 [10:29:07] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.58, 8.02, 6.69 [10:29:40] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.58, 6.96, 6.18 [10:29:52] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.94, 6.99, 6.54 [10:30:03] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.61, 6.60, 5.55 [10:31:01] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.45, 6.27, 6.10 [10:31:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.14, 6.80, 6.39 [10:31:38] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.81, 5.85, 5.87 [10:31:49] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.73, 6.78, 6.53 [10:32:01] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.29, 5.69, 5.34 [10:35:43] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.80, 7.10, 6.73 [10:36:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.99, 1.82, 1.07 [10:38:31] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.40, 1.30, 0.97 [10:39:40] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.29, 6.16, 6.43 [10:44:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 14.27, 20.17, 23.30 [10:48:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.90, 22.99, 23.57 [10:51:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.85, 6.53, 6.19 [10:52:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.89, 21.99, 23.20 [10:53:40] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.37, 5.90, 5.99 [10:54:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.97, 22.94, 23.36 [11:00:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.29, 22.28, 23.17 [11:01:16] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.94, 7.91, 6.45 [11:05:16] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.48, 6.00, 6.03 [11:14:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.98, 23.84, 22.92 [11:16:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.84, 22.52, 22.52 [11:18:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 38.10, 26.55, 23.87 [11:20:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.83, 23.45, 23.06 [11:22:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.22, 25.16, 23.71 [11:24:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.10, 23.71, 23.33 [11:26:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.86, 25.38, 23.94 [11:36:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.61, 21.85, 23.36 [11:42:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.99, 24.66, 23.81 [11:46:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.35, 23.76, 23.83 [11:48:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.31, 25.98, 24.59 [11:50:46] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.38, 6.89, 6.23 [11:51:29] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.92, 7.43, 6.04 [11:51:41] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.68, 7.07, 6.14 [11:52:44] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.24, 6.56, 6.19 [11:53:05] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.06, 7.90, 6.82 [11:53:24] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.85, 7.24, 6.13 [11:53:40] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.33, 6.62, 6.10 [11:56:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.89, 6.94, 6.44 [11:57:02] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.60, 6.51, 6.53 [11:57:13] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.96, 6.07, 5.95 [11:58:38] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.39, 6.64, 6.39 [12:01:31] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.05, 9.56, 7.49 [12:02:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:02:31] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.81, 7.28, 6.79 [12:02:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 7.06, 8.23, 7.32 [12:02:59] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 7.04, 8.20, 6.96 [12:04:29] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.81, 6.46, 6.55 [12:04:54] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.29, 6.93, 6.62 [12:04:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.53, 7.30, 7.06 [12:05:28] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.13, 7.58, 7.16 [12:07:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:08:22] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.79, 8.23, 7.27 [12:08:34] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.56, 6.84, 6.08 [12:08:48] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.56, 7.53, 6.32 [12:08:49] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.62, 6.47, 6.56 [12:09:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.02, 7.12, 7.03 [12:10:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.92, 6.62, 6.10 [12:10:46] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.22, 6.83, 6.19 [12:11:23] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.36, 6.37, 6.76 [12:12:43] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.63, 6.46, 6.14 [12:13:02] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.59, 8.01, 7.31 [12:14:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.89, 7.33, 7.21 [12:14:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.58, 6.97, 6.77 [12:15:01] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.18, 7.55, 7.23 [12:16:49] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.12, 6.10, 6.50 [12:18:11] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.04, 7.26, 7.17 [12:18:58] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.92, 6.10, 6.76 [12:22:06] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.73, 7.10, 7.17 [12:28:00] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.57, 5.85, 6.63 [12:28:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.33, 21.04, 23.32 [12:32:59] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.14, 22.26, 23.22 [12:34:46] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.49, 6.86, 6.39 [12:35:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.20, 7.00, 6.38 [12:36:45] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.71, 6.57, 6.35 [12:37:34] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.91, 6.41, 6.23 [12:38:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.29, 23.89, 23.89 [12:40:41] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.51, 6.91, 6.61 [12:40:53] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.52, 6.84, 6.48 [12:40:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 22.72, 24.19, 24.06 [12:41:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.91, 7.38, 6.41 [12:41:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.79, 7.81, 7.28 [12:42:40] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.67, 6.11, 6.35 [12:42:52] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.18, 6.18, 6.27 [12:42:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.28, 23.03, 23.63 [12:43:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.42, 6.78, 6.32 [12:48:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.33, 6.84, 6.55 [12:48:47] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.77, 8.11, 6.98 [12:48:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.43, 24.25, 23.84 [12:49:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.26, 7.76, 6.68 [12:49:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 14.86, 9.56, 7.93 [12:50:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.40, 8.03, 7.06 [12:51:04] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.11, 7.52, 6.76 [12:52:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.02, 6.56, 6.64 [12:52:44] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.71, 6.97, 6.85 [12:52:59] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.75, 6.49, 6.46 [12:54:04] ok so, everything is pretty slow right now [12:54:15] and I'm occasionally getting 503s [12:54:43] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.30, 8.29, 7.36 [12:55:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.95, 7.39, 6.74 [12:55:57] mw10/11/12 are backed up [12:56:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:56:01] Reception123: [12:56:04] !sre [12:56:13] * RhinosF1 no access [12:56:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.46, 7.19, 6.32 [12:56:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.96, 6.95, 6.85 [12:56:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.99, 7.45, 6.99 [12:56:54] Never mind [12:57:01] It's recovered [12:58:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.97, 8.03, 7.22 [12:59:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.12, 6.09, 6.39 [12:59:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.53, 7.90, 7.93 [13:00:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.83, 5.92, 6.02 [13:00:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.66, 7.51, 7.11 [13:00:49] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.47, 6.61, 6.78 [13:01:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:04:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.25, 5.91, 6.60 [13:04:36] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.41, 7.31, 7.59 [13:09:40] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.21, 5.67, 6.69 [13:10:31] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.88, 5.66, 6.77 [13:10:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.95, 21.49, 23.96 [13:12:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.47, 24.04, 24.56 [13:20:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.37, 7.11, 6.59 [13:22:12] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.67, 8.04, 6.76 [13:22:21] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.43, 7.04, 6.85 [13:22:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.77, 6.95, 6.60 [13:24:24] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.44, 7.36, 6.91 [13:24:59] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.12, 7.99, 6.52 [13:26:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.55, 7.83, 6.94 [13:26:18] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.87, 6.45, 6.66 [13:26:22] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.57, 7.43, 7.01 [13:26:55] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.88, 7.49, 6.49 [13:27:56] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.44, 6.53, 6.57 [13:28:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.00, 6.43, 6.60 [13:29:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.47, 3.67, 3.19 [13:30:49] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.07, 8.14, 6.96 [13:30:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.07, 22.06, 23.97 [13:31:07] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.59, 3.29, 3.11 [13:32:46] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.98, 7.79, 6.98 [13:34:43] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.59, 7.95, 7.13 [13:36:11] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.24, 7.34, 7.09 [13:36:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.02, 7.25, 6.79 [13:36:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 31.14, 23.71, 23.76 [13:37:07] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.31, 7.66, 6.95 [13:38:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.98, 7.49, 6.97 [13:38:37] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.57, 7.41, 7.12 [13:38:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.08, 22.77, 23.41 [13:39:05] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.70, 6.83, 6.72 [13:40:34] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.93, 8.07, 7.38 [13:40:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.52, 7.95, 7.19 [13:40:56] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 32.52, 25.72, 24.37 [13:41:04] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.80, 8.03, 7.20 [13:41:25] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.99, 7.39, 6.82 [13:41:43] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.32, 8.16, 6.89 [13:43:20] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.64, 9.01, 7.50 [13:44:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.07, 7.47, 7.28 [13:45:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.35, 7.89, 7.15 [13:47:10] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.78, 7.61, 7.25 [13:49:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.51, 3.30 [13:49:05] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.39, 9.08, 7.82 [13:49:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.52, 5.81, 6.48 [13:50:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.03, 5.87, 6.65 [13:53:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.57, 4.12, 3.56 [13:54:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.71, 7.68, 7.22 [13:55:53] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.35, 1.79, 1.37 [13:56:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.45, 7.67, 7.96 [13:57:49] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.86, 1.78, 1.42 [13:58:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.39, 6.50, 6.89 [13:58:38] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.96, 7.34, 7.91 [13:59:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 4.00, 3.73 [13:59:45] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.38, 1.63, 1.40 [14:00:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.21, 6.23, 6.75 [14:01:43] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.02, 7.43, 7.91 [14:02:47] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.14, 7.38, 7.95 [14:05:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.39, 3.92, 3.77 [14:05:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.63, 5.03, 5.88 [14:06:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.27, 6.13, 6.97 [14:06:44] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.28, 7.69, 7.88 [14:06:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.86, 7.06, 7.33 [14:07:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.85, 3.86, 3.77 [14:07:48] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 11.80, 7.22, 6.56 [14:08:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.90, 6.39, 6.98 [14:08:43] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.73, 6.96, 7.60 [14:08:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.83, 6.34, 7.02 [14:10:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.62, 7.44, 7.31 [14:10:41] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.50, 7.61, 7.77 [14:10:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.47, 7.17, 7.25 [14:11:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.55, 8.43, 8.06 [14:12:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.02, 7.37, 7.31 [14:12:40] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.78, 7.76, 7.81 [14:12:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.81, 7.79, 7.49 [14:13:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.01, 7.55, 7.79 [14:16:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.38, 8.40, 7.64 [14:16:37] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 13.55, 9.82, 8.53 [14:16:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.56, 7.55, 7.37 [14:17:15] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.42, 7.09, 6.86 [14:17:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.09, 8.53, 8.06 [14:18:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:21:42] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.77, 7.38, 6.59 [14:23:05] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 12.00, 9.95, 8.11 [14:23:40] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.97, 7.58, 6.79 [14:25:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.86, 3.16, 3.38 [14:28:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:28:56] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.97, 7.47, 7.63 [14:29:32] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.07, 7.26, 6.91 [14:29:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.58, 7.17, 7.85 [14:30:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.89, 7.53, 7.84 [14:30:53] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.39, 8.00, 7.79 [14:31:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.12, 8.20, 8.16 [14:32:50] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.24, 7.88, 7.91 [14:32:50] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.99, 7.16, 7.51 [14:34:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.76, 6.60, 7.80 [14:34:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.24, 6.47, 7.39 [14:35:25] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.11, 7.57, 7.24 [14:35:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.24, 6.89, 7.66 [14:42:34] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.04, 5.83, 6.70 [14:42:49] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.85, 5.81, 6.70 [14:43:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.50, 6.29, 6.71 [14:47:40] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.93, 5.78, 6.73 [14:48:13] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.59, 6.21, 7.58 [14:48:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.49, 6.04, 6.74 [14:50:01] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.27, 3.75, 3.29 [14:53:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.94, 3.83, 3.43 [14:55:59] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.70, 3.39, 3.32 [14:56:32] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.28, 7.14, 6.98 [14:56:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.50, 6.84, 6.88 [14:58:11] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.32, 7.29, 7.37 [14:58:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.79, 1.65, 1.25 [14:58:29] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.33, 6.66, 6.84 [14:58:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.92, 7.32, 7.03 [15:00:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.40, 1.48, 1.24 [15:00:10] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.11, 7.47, 7.45 [15:00:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.33, 6.89, 6.89 [15:02:25] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.70, 6.37, 6.70 [15:02:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.37, 6.10, 6.61 [15:06:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.05, 7.26, 6.93 [15:08:16] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.89, 6.78, 6.64 [15:09:51] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.21, 1.99, 1.58 [15:10:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.86, 6.64, 6.74 [15:11:47] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.84, 1.62, 1.49 [15:12:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.03, 5.94, 6.47 [15:13:59] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.41, 6.09, 6.74 [15:14:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.06, 22.34, 23.65 [15:18:05] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.05, 6.09, 6.57 [15:20:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 31.79, 25.82, 24.47 [15:21:51] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.85, 8.80, 7.57 [15:22:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.92, 8.85, 7.33 [15:22:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.98, 3.52, 3.26 [15:22:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.66, 8.47, 7.12 [15:23:01] .op [15:23:01] Attempting to OP... [15:23:35] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.20, 7.02, 6.40 [15:23:52] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.15, 7.43, 7.23 [15:23:56] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.52, 7.63, 7.06 [15:24:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.22, 7.60, 7.07 [15:24:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.30, 3.18, 3.15 [15:24:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.56, 8.00, 7.10 [15:25:32] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.17, 6.50, 6.29 [15:25:54] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.00, 7.08, 6.93 [15:27:52] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.69, 7.31, 7.01 [15:31:47] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.67, 7.60, 7.23 [15:32:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.29, 7.14, 6.95 [15:32:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.81, 7.76, 7.25 [15:33:44] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.08, 7.80, 7.35 [15:33:45] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.53, 8.19, 7.48 [15:34:41] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.70, 7.33, 6.66 [15:34:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.69, 6.73, 6.94 [15:36:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.73, 7.99, 7.38 [15:37:41] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.99, 7.52, 7.32 [15:38:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.38, 7.91, 7.00 [15:38:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.84, 7.86, 7.40 [15:39:40] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.90, 7.94, 7.49 [15:40:33] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.44, 7.62, 7.01 [15:40:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.95, 8.15, 7.56 [15:42:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.60, 7.81, 7.53 [15:43:58] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.73, 7.29, 6.64 [15:44:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.88, 7.64, 7.56 [15:45:55] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.76, 8.16, 6.98 [15:46:26] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.58, 8.31, 7.46 [15:46:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.79, 8.49, 7.87 [15:46:50] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.78, 9.40, 8.15 [15:48:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:50:11] RhinosF1: https://phabricator.miraheze.org/T7696#169542 should be everything we still need to run [15:50:12] [url] ⚓ T7696 Upgrade to MediaWiki 1.37.0 | phabricator.miraheze.org [15:50:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.24, 7.77, 7.49 [15:51:18] Also, Reception123: ^ [15:51:30] ack, thanks! [15:51:41] Im around if extra eyes are needed [15:52:17] CosmicAlpha: just the 3 rename indexes need to be done during upgrade [15:52:18] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.70, 7.98, 7.59 [15:52:28] the DROPs can be done after in RW [15:52:41] 17:37 MirahezeLSBot: [paladox@test3] sudo -u www-data php /srv/mediawiki/w/extensions/CommentStreams/maintenance/MigrateToAbstractSchema.php --wiki test3wiki i'm not sure on [15:53:12] i will handle the logging/flaggedrevs/image scripts [15:53:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:54:15] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.51, 7.44, 7.43 [15:54:27] CosmicAlpha: you will need to do python3 /usr/local/bin/mwupgradetool because i didn't add the python3 at the top [15:55:22] it's mostly automated now, i'm nervous given i wrote a lot of the script [15:55:31] Lol [15:55:58] in theory one command, merge a few PRs and spam enter [15:56:11] and a lot of waiting [15:56:13] PROBLEM - ping6 on cp31 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 78.13 ms [15:56:13] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.71, 9.30, 8.13 [15:56:15] Delete ~/ for good measure [15:56:33] it does delete w [15:56:39] well the staging w [15:56:40] Close enough [15:56:45] easier than git [15:57:58] my hope is it will take ~90 minutes [15:58:11] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.93, 7.94, 7.77 [15:58:13] RECOVERY - ping6 on cp31 is OK: PING OK - Packet loss = 0%, RTA = 76.34 ms [15:58:41] We've put down 3 hours though to avert any disappointments in case it takes longer / something goes wrong [15:58:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.47, 7.73, 7.96 [15:58:53] Though hopefully next time we'll be able to have a better estimate after the first time using the script on prod [15:59:21] Reception123: well that depends on if we manage multiversion [15:59:32] with multiversion it would be 5 minutes RO [15:59:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.73, 7.60, 7.65 [15:59:57] our other gain would be to do all servers at once rather than 1 by 1 [16:00:12] it's about 40 minutes going all at once [16:00:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.63, 8.14, 8.09 [16:02:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.44, 7.08, 7.70 [16:03:27] RhinosF1: https://phabricator.miraheze.org/transactions/detail/PHID-XACT-TASK-gxadxdbxp23wugf/ that look right for mwscript? sql.php will work with mwscript, right? [16:03:35] [url] Change Details | phabricator.miraheze.org [16:03:50] yeah [16:03:55] CosmicAlpha: yep [16:04:15] sysadmins should be able to use htop to get some idea of progress [16:04:24] i can let you know the master PID [16:04:30] when it's set off [16:04:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.12, 7.55, 7.80 [16:06:57] RhinosF1: what would happen if I put that all in one bash script, would it work or cause issues? I also need to create patch files for the actual SQL commands so they can be ran easier I think, or maybe sql.php with `--query` would work? As that's a bit easier to me if so. [16:07:13] CosmicAlpha: single bash should work [16:07:21] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.43, 5.54, 6.68 [16:07:22] or put everything in 1 sql file [16:07:59] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.94, 7.26, 7.33 [16:08:00] Just bc it should work doesnt mean it will esp with sql involved [16:08:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.70, 7.76, 7.84 [16:09:56] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.89, 7.37, 7.36 [16:12:02] RhinosF1: yeah I think I'll just run individual, I'd rather not risk it with sql. [16:13:08] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.76, 6.94, 6.99 [16:14:23] ok [16:15:05] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.85, 6.40, 6.77 [16:15:19] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.68, 6.96, 7.75 [16:15:49] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.95, 7.81, 7.56 [16:16:14] Now that we got SQL added to task, update should be good to go when time. I really think we are much more prepared this time. Inevitably we will have some issues post-upgrade with extensions that we missed, etc... though I think we are pretty good for upgrade itself and hope everything goes according to plan! [16:17:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.70, 7.30, 7.90 [16:17:46] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.35, 7.26, 7.39 [16:18:11] RhinosF1: what would happen if the sql.php foreachwikiindblist reaches test3wiki, which is already ran on, would it skip it or cause script to fatal, I can't remember.... [16:18:33] Idk why it would fatal [16:18:35] CosmicAlpha: it should if the sql is wrote properly just carry on [16:19:17] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.80, 7.33, 7.68 [16:19:55] @Zppix: I mean if the sql fails I wasn't sure if the entire sql.php would, some of the scripts don't use IF EXISTS, etc... so I was wondering if it would cause sql.php to exit in those instances, as sql itself would fail. Also RhinosF1 [16:20:31] CosmicAlpha: not sure but it won't matter on test3 [16:20:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.16, 8.51, 7.82 [16:21:37] RhinosF1: oh true, lol yeah it won't cause mwscript/foreachwikiindblist to exit only the wiki it's currently ran on. I guess I wasn't thinking right about that... [16:22:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.34, 7.21, 8.00 [16:22:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.69, 7.71, 7.59 [16:23:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.81, 4.31, 3.61 [16:25:17] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.72, 7.52, 7.81 [16:27:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.33, 4.00, 3.66 [16:27:33] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.80, 5.94, 6.74 [16:28:07] [02mw-config] 07Universal-Omega commented on pull request 03#4264: Switch to post-1.37 upgrade sitenotice - 13https://git.io/JDeQC [16:29:13] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.85, 4.12, 3.75 [16:29:17] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.57, 8.40, 8.06 [16:29:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.85, 7.82, 7.77 [16:30:24] [02mw-config] 07AgentIsai commented on pull request 03#4264: Switch to post-1.37 upgrade sitenotice - 13https://git.io/JDeQb [16:30:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.02, 7.84, 7.85 [16:31:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 1.93, 3.31, 3.50 [16:31:27] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.61, 6.91, 6.98 [16:31:45] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:32:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:32:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.07, 7.47, 7.24 [16:33:24] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.14, 8.15, 7.45 [16:33:39] CosmicAlpha: looks like we're down :( [16:33:40] 502 for me [16:33:43] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.016 second response time [16:34:12] Reception123: it's just high load [16:34:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.09, 6.49, 6.47 [16:34:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.22, 7.45, 7.27 [16:35:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.87, 2.73, 3.23 [16:35:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.47, 7.58, 7.33 [16:35:50] No issues here [16:36:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.44, 6.71, 6.57 [16:36:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.57, 8.05, 7.50 [16:37:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.32, 8.43, 7.68 [16:37:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:38:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.47, 7.59, 7.41 [16:39:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.59, 3.46 [16:40:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.52, 8.20, 7.64 [16:41:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.68, 3.51 [16:42:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.21, 8.72, 7.42 [16:43:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:45:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.86, 3.19, 3.34 [16:49:21] Reception123, CosmicAlpha: less than 10 minutes [16:50:03] yup [16:52:56] Reception123: do you want to depool mw11 [16:53:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.63, 3.40 [16:53:06] i can remove from jobrunner [16:53:24] RhinosF1: will do at 58 [16:53:28] then it's all CosmicAlpha 's [16:53:35] well most [16:53:45] ok [16:55:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.33, 3.15, 3.25 [16:56:59] !log stop jobrunner & jobchron on mw11 in prep for 1.37 upgrade [16:57:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:57:11] !log only jobrunner, no chron [16:57:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:58:22] Reception123: now 58 [16:58:51] !log depool mw11 [16:59:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:00:01] PROBLEM - mw11 JobRunner Service on mw11 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [17:00:07] CosmicAlpha: feel free to start [17:00:29] Let the upgrades commence! [17:00:38] Good luck! [17:01:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.39, 3.32 [17:01:25] RhinosF1: script not working. Might end up doing manual if it does not. [17:01:54] CosmicAlpha: what is it saying [17:02:00] https://www.irccloud.com/pastebin/J9LNLU1C/ [17:02:02] Reception123: doesn't look depooled [17:02:27] CosmicAlpha: use the full python3 /usr/local/bin/mwupgradetool [17:03:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:03:02] i did say you'll need python3 at start [17:03:12] RECOVERY - wiki.villagecollaborative.net - reverse DNS on sslhost is OK: SSL OK - wiki.villagecollaborative.net reverse DNS resolves to cp21.miraheze.org - CNAME OK [17:03:19] RhinosF1: I did it with puppet3 with the usual [17:05:00] [02mw-config] 07Universal-Omega deleted branch 03Reception123-patch-2 - 13https://git.io/vbvb3 [17:05:01] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDeAy [17:05:03] [02miraheze/mw-config] 07Reception123 03d177df6 - set all wikis to readonly (MW 1.37 upgrade) (#4245) [17:05:04] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Reception123-patch-2 [17:05:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.96, 3.26, 3.28 [17:05:06] [02mw-config] 07Universal-Omega closed pull request 03#4245: set all wikis to readonly (MW 1.37 upgrade) - 13https://git.io/JMMPs [17:05:16] !log restart varnish on cp20, seems to crashed? [17:06:07] miraheze/mw-config - Universal-Omega the build passed. [17:06:12] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:22] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3126 bytes in 7.607 second response time [17:06:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:06:40] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw8 mw10 [17:06:41] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:06:47] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 10% [17:06:52] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 9.15, 5.71, 3.62 [17:06:58] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw10 [17:06:58] People are still editing though. [17:07:08] should stick a shebang on that script [17:07:11] !log [@mw11] starting deploy of {'config': True} to all [17:07:14] we only just merged the read only script [17:07:24] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 28.01, 24.75, 20.95 [17:07:28] !log depool mw11 [17:07:29] Voidwalker: yes that was my fault but i only saw about half an hour ago [17:08:18] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 21168 bytes in 8.474 second response time [17:08:34] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 82s [17:08:38] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 9 backends are healthy [17:08:41] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 1.710 second response time [17:08:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.68, 5.91, 3.97 [17:08:57] should be stopped [17:09:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.82, 3.51 [17:09:07] Alright. [17:09:16] Reception123: puppet branch [17:09:24] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.40, 23.23, 20.90 [17:09:29] CosmicAlpha: doing now [17:09:54] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:09:55] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JDexK [17:09:56] [02miraheze/puppet] 07RhinosF1 034fd408f - mediawiki: update to 1.37 (#2141) [17:10:08] [02puppet] 07Reception123 closed pull request 03#2141: mediawiki: update to 1.37 - 13https://git.io/JMMP9 [17:10:16] CosmicAlpha: done and pulled on puppet3 [17:10:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 2.93, 4.90, 3.84 [17:10:55] i see puppet running [17:11:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.74, 3.51 [17:11:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:11:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:11:59] that was htop slow [17:12:05] w is being cleared on 11 [17:12:20] CosmicAlpha: please ping before you start canary deploy [17:12:46] RhinosF1: will do [17:12:47] [02puppet] 07DarkMatterMan4500 commented on pull request 03#2141: mediawiki: update to 1.37 - 13https://git.io/JDepe [17:13:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.68, 3.38, 3.41 [17:13:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 1.60, 4.41, 7.61 [17:13:23] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.53, 19.37, 19.85 [17:13:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:13:40] @DarkMatterMan4500 if we could refrain from redundant comments, so sre can focus [17:13:44] [02puppet] 07DarkMatterMan4500 deleted a comment on pull request 03#2141: mediawiki: update to 1.37 - 13https://git.io/JDepe [17:14:00] Sorry about that. [17:14:11] looks to be pulling on 11 [17:14:22] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki core] [17:15:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.69, 3.29, 3.38 [17:16:25] yep. it is pulling on mw11. [17:16:25] it's on the variables extension [17:16:51] PID is 21134 if anyone with access wants to follow along [17:17:16] it's now checking everything out [17:17:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 2.96, 3.61, 6.53 [17:18:27] now it's doing npm + composr [17:19:17] 6267 is the main tool [17:19:31] puppet looks to have stopped [17:19:49] RhinosF1: Canary deploy is next [17:20:05] CosmicAlpha: cool, if it says REL1_37, press enter [17:20:33] deploy-mediawiki: error: unrecognized arguments: --gitinfo [17:20:37] RhinosF1: ^ [17:20:46] I wish SRE a big success in the upgrade, Amen. [17:20:47] urgh [17:20:52] you took that outt [17:21:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.06, 3.83, 5.66 [17:22:02] [02puppet] 07Universal-Omega opened pull request 03#2147: remove --gitinfo - 13https://git.io/JDeh6 [17:22:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.76, 6.10, 7.62 [17:22:49] RhinosF1: I will need to restart script. I can comment stuff like clone again out though right? [17:23:04] CosmicAlpha: python3 /home/rhinos/mwupgrade.py should be fixed and resume from where you were [17:23:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.15, 3.54, 3.44 [17:23:30] it'll go from checking the branch [17:23:30] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True, 'force': True, 'ignoretime': True} to skip [17:23:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:24:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:24:43] [02puppet] 07paladox closed pull request 03#2147: remove --gitinfo - 13https://git.io/JDeh6 [17:24:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDejT [17:24:46] [02miraheze/puppet] 07Universal-Omega 03ca98f82 - remove --gitinfo (#2147) [17:24:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.73, 21.28, 23.66 [17:25:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.45, 4.06, 3.63 [17:26:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.45, 6.94, 7.57 [17:26:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.83, 23.50, 24.25 [17:27:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.84, 3.60 [17:28:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.25, 6.42, 7.29 [17:28:38] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw11 [17:29:03] [no req] ExtensionDependencyError: Maps is not compatible with the current PHP version 7.3.31), it requires: >= 7.4. [17:29:14] urgh [17:29:21] CosmicAlpha: erm [17:29:22] I guess we should have considered that but it was hard to test [17:29:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:29:32] If that stops us then we need to disable Maps temporarily unfortunately [17:29:44] it's still running afaics [17:30:03] CosmicAlpha: did it cause any fatal or can we just disable once deployed [17:30:30] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:30:34] CosmicAlpha: downgrade Maps i guess. [17:30:39] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 9 backends are healthy [17:30:40] RhinosF1: lookes to have fataled mergeMessageFileList [17:30:43] unless it requires something specific. [17:30:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.23, 4.78, 3.99 [17:30:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.99, 22.85, 23.79 [17:31:00] paladox: the 7.3 version is 1.37 incompatible. [17:31:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.33, 3.89, 3.66 [17:31:09] oh ok [17:31:10] if it fatals then we should just remove for now [17:31:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.22, 3.99, 5.00 [17:32:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JDveq [17:32:30] [02miraheze/mw-config] 07Universal-Omega 03426d384 - disable Maps temporarily [17:32:32] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [17:32:33] !log upgrade grafana [17:32:33] [02mw-config] 07Universal-Omega opened pull request 03#4265: disable Maps temporarily - 13https://git.io/JDveY [17:32:44] !log [universalomega@mw11] finished deploy of {'world': True, 'l10n': True, 'force': True, 'ignoretime': True} to skip - FAIL: [0, 0, 0, 0, 65280, 0] in 554s [17:32:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 4.20, 4.54, 3.99 [17:33:35] miraheze/mw-config - Universal-Omega the build passed. [17:33:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JDveu [17:33:58] [02miraheze/mw-config] 07Universal-Omega 032ed2af4 - Update LocalSettings.php [17:34:00] [02mw-config] 07Universal-Omega synchronize pull request 03#4265: disable Maps temporarily - 13https://git.io/JDveY [17:34:28] CosmicAlpha: hold an i'll rerun then ping you [17:34:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.58, 5.83, 6.78 [17:34:59] miraheze/mw-config - Universal-Omega the build passed. [17:35:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.18, 3.27, 3.48 [17:35:09] CosmicAlpha: can i merge your config [17:35:37] mw11 seems to be up and on 1.37 RhinosF1, also config should be OK to merge yeah, might want to also restrict it in MWE so users don't get confused though. [17:36:04] CosmicAlpha: let's do that after [17:36:13] [02mw-config] 07RhinosF1 closed pull request 03#4265: disable Maps temporarily - 13https://git.io/JDveY [17:36:14] [02mw-config] 07RhinosF1 deleted branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [17:36:16] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDveh [17:36:17] [02miraheze/mw-config] 07Universal-Omega 03a7e9c8c - disable Maps temporarily (#4265) [17:36:19] [02miraheze/mw-config] 07RhinosF1 deleted branch 03Universal-Omega-patch-4 [17:36:47] i18n is more user visible [17:37:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.78, 3.07, 3.37 [17:37:13] !log [@mw11] starting deploy of {'config': True} to all [17:37:21] CosmicAlpha: i'll ping you when you can press rollout [17:37:23] miraheze/mw-config - RhinosF1 the build passed. [17:37:29] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 16s [17:38:14] RhinosF1: may need to remove maps from extension-list in config also now that I think about it. [17:38:49] [02puppet] 07RhinosF1 opened pull request 03#2148: composer: disable maps - 13https://git.io/JDvvm [17:39:02] CosmicAlpha: send pr [17:39:09] paladox, Reception123: pls merge puppet [17:39:28] and pull [17:39:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JDvvW [17:39:31] [02miraheze/mw-config] 07Universal-Omega 03d6664e5 - Remove maps from extension-list [17:39:33] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [17:39:34] [02puppet] 07Reception123 closed pull request 03#2148: composer: disable maps - 13https://git.io/JDvvm [17:39:36] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDvv8 [17:39:37] [02miraheze/puppet] 07RhinosF1 03f9ef831 - composer: disable maps (#2148) [17:39:38] done and done [17:39:39] [02mw-config] 07Universal-Omega opened pull request 03#4266: Remove maps from extension-list - 13https://git.io/JDvv4 [17:39:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.12, 6.58, 7.82 [17:39:55] [02mw-config] 07RhinosF1 closed pull request 03#4266: Remove maps from extension-list - 13https://git.io/JDvv4 [17:39:57] [02miraheze/mw-config] 07RhinosF1 deleted branch 03Universal-Omega-patch-4 [17:39:58] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDvvz [17:40:00] [02miraheze/mw-config] 07Universal-Omega 031ed2feb - Remove maps from extension-list (#4266) [17:40:02] [02mw-config] 07RhinosF1 deleted branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [17:40:40] miraheze/mw-config - Universal-Omega the build passed. [17:40:50] !log [@mw11] starting deploy of {'config': True} to all [17:40:56] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 6s [17:41:00] miraheze/mw-config - RhinosF1 the build passed. [17:41:25] !log [rhinos@mw11] starting deploy of {'l10n': True} to skip [17:42:21] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:43:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.49, 7.97, 8.08 [17:44:11] !log [rhinos@mw11] finished deploy of {'l10n': True} to skip - SUCCESS in 166s [17:44:22] RhinosF1: am I good to continue then? [17:44:25] CosmicAlpha: proceed [17:44:48] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True, 'force': True, 'ignoretime': True} to mw8 [17:45:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4245-Reception123-patch-2 [+0/-0/±1] 13https://git.io/JDvfC [17:45:48] [02miraheze/mw-config] 07Universal-Omega 032ac9444 - Revert "set all wikis to readonly (MW 1.37 upgrade) (#4245)" [17:45:49] [02mw-config] 07Universal-Omega created branch 03revert-4245-Reception123-patch-2 - 13https://git.io/vbvb3 [17:45:51] [02mw-config] 07Universal-Omega opened pull request 03#4267: Revert "set all wikis to readonly (MW 1.37 upgrade)" - 13https://git.io/JDvfW [17:46:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.67, 7.21, 6.93 [17:46:45] paladox: why are things not logging? [17:46:54] read only mode [17:46:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.60, 23.31, 22.79 [17:46:57] oh [17:46:58] miraheze/mw-config - Universal-Omega the build passed. [17:47:09] heh, I never realise that either :P [17:47:15] just use !log still so we have a record here [17:47:37] yeah... I will manually add them in the end probably. [17:48:02] * RhinosF1 going mobile on and off - we should be okay now [17:48:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.87, 22.22, 22.44 [17:50:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.95, 6.45, 6.72 [17:50:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.06, 22.98, 22.70 [17:51:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.85, 7.57, 7.90 [17:51:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb [17:52:55] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:53:02] so i can repool mw11 RhinosF1 / CosmicAlpha / Reception123 ? [17:53:24] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.92, 5.59, 5.07 [17:53:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:53:50] paladox: would wait till all servers are done so no load on mw11 during, but just my thoughts. [17:53:58] ok [17:54:06] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw8 [17:54:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.99, 7.86, 7.27 [17:54:55] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:55:46] !log [universalomega@mw11] finished deploy of {'world': True, 'l10n': True, 'force': True, 'ignoretime': True} to mw8 - SUCCESS in 658s [17:55:47] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True, 'force': True, 'ignoretime': True} to mw9 [17:55:59] paladox: what CosmicAlpha said, the whole point of the depool is to ensure no load so it runs faster given 75% of the work is done on 11 [17:56:06] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [17:56:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:57:13] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.11, 5.82, 5.33 [17:57:24] Based on current timings, 50 minutes left [17:57:35] + sqls etc [17:58:21] 50 minutes of actual deploying [18:00:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:01:01] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.88, 4.67, 4.96 [18:01:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.17, 8.46, 8.12 [18:01:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.30, 3.20 [18:02:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.06, 22.87, 23.59 [18:03:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.63, 3.15, 3.16 [18:04:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.92, 8.49, 7.60 [18:04:38] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw9 [18:05:37] ok : [RESOLVED] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:06:38] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 9 backends are healthy [18:06:52] !log [universalomega@mw11] finished deploy of {'world': True, 'l10n': True, 'force': True, 'ignoretime': True} to mw9 - SUCCESS in 665s [18:06:54] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True, 'force': True, 'ignoretime': True} to mw10 [18:06:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.04, 24.25, 23.93 [18:07:08] That's 3/7 [18:07:42] alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:08:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.74, 23.42, 23.66 [18:10:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.42, 7.38, 7.43 [18:12:17] Exceptions are expected [18:12:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.89, 8.13, 7.69 [18:12:42] ok : [RESOLVED] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:13:20] It's just while it's deploying to a server [18:13:36] Because we can't guarantee order of files will avoid every case [18:14:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.90, 7.50, 7.53 [18:16:07] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 878 bytes in 0.074 second response time [18:16:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:16:38] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw10 [18:17:03] [02mw-config] 07Universal-Omega assigned pull request 03#4267: Revert "set all wikis to readonly (MW 1.37 upgrade)" - 13https://git.io/JDvfW [18:17:54] !log [universalomega@mw11] finished deploy of {'world': True, 'l10n': True, 'force': True, 'ignoretime': True} to mw10 - SUCCESS in 660s [18:18:08] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True, 'force': True, 'ignoretime': True} to mw12 [18:18:14] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 21684 bytes in 6.605 second response time [18:18:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.53, 6.81, 7.09 [18:18:38] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 9 backends are healthy [18:20:19] ~35 minutes left [18:21:19] RhinosF1: CosmicAlpha https://phabricator.miraheze.org/T8394 hmm [18:21:20] [url] ⚓ T8394 Fatal miraheze data base error. | phabricator.miraheze.org [18:21:33] have SQLs been ran yet? [18:21:37] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:21:58] Reception123: it should be fixed when I run sqls right after the deploys finish, I think. [18:22:30] ah ok, so they're at the end [18:22:35] yes [18:24:59] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.73, 22.60, 19.27 [18:26:37] ok : [RESOLVED] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:26:59] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 14.26, 19.67, 18.63 [18:27:42] alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:28:38] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw12 [18:28:41] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw12 [18:28:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.53, 21.86, 22.18 [18:29:19] !log [universalomega@mw11] finished deploy of {'world': True, 'l10n': True, 'force': True, 'ignoretime': True} to mw12 - SUCCESS in 671s [18:29:23] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True, 'force': True, 'ignoretime': True} to mw13 [18:29:38] Final user facing server [18:30:38] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 9 backends are healthy [18:30:39] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 9 backends are healthy [18:30:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.63, 21.96, 22.21 [18:35:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.02, 3.42, 3.18 [18:37:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.79, 4.30, 3.51 [18:37:41] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.23, 6.54, 7.60 [18:37:42] ok : [RESOLVED] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:38:38] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.88, 7.55, 7.94 [18:39:05] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 34.94, 23.01, 19.42 [18:39:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:39:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 16.58, 9.45, 8.48 [18:39:55] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [18:40:24] !log [universalomega@mw11] finished deploy of {'world': True, 'l10n': True, 'force': True, 'ignoretime': True} to mw13 - SUCCESS in 660s [18:40:32] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [18:40:34] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.02, 8.64, 8.28 [18:40:37] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True, 'force': True, 'ignoretime': True} to mwtask1 [18:41:55] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:42:26] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:42:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.14, 7.49, 7.89 [18:43:04] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 17.21, 22.67, 20.43 [18:44:07] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:44:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.68, 8.06, 8.05 [18:45:03] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 23.27, 24.05, 21.26 [18:46:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.37, 7.89, 7.99 [18:47:03] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.72, 22.50, 20.99 [18:47:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.39, 3.47, 3.59 [18:49:07] ok : [RESOLVED] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:49:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.79, 2.94, 3.38 [18:50:18] !log [universalomega@mw11] finished deploy of {'world': True, 'l10n': True, 'force': True, 'ignoretime': True} to mwtask1 - SUCCESS in 580s [18:50:29] [02mw-config] 07Universal-Omega closed pull request 03#4267: Revert "set all wikis to readonly (MW 1.37 upgrade)" - 13https://git.io/JDvfW [18:50:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDvmh [18:50:32] [02miraheze/mw-config] 07Universal-Omega 031dc65fc - Revert "set all wikis to readonly (MW 1.37 upgrade) (#4245)" (#4267) [18:50:33] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-4245-Reception123-patch-2 [18:50:35] [02mw-config] 07Universal-Omega deleted branch 03revert-4245-Reception123-patch-2 - 13https://git.io/vbvb3 [18:50:38] [02mw-config] 07Universal-Omega closed pull request 03#4264: Switch to post-1.37 upgrade sitenotice - 13https://git.io/JMjJ4 [18:50:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDvYJ [18:50:41] [02miraheze/mw-config] 07Universal-Omega 030ce3315 - Switch to post-1.37 upgrade sitenotice (#4264) [18:50:42] alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:50:43] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [18:50:44] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [18:50:53] !log [universalomega@mw11] starting deploy of {'pull': 'config', 'config': True} to all [18:51:02] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.82, 20.03, 20.31 [18:51:13] !log [universalomega@mw11] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 19s [18:51:28] PROBLEM - mw9 APT on mw9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:51:37] miraheze/mw-config - Universal-Omega the build passed. [18:51:43] miraheze/mw-config - Universal-Omega the build passed. [18:51:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:52:12] paladox, reception123 mw11 can be repooled [18:52:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.44, 7.80, 7.82 [18:52:47] CosmicAlpha: will do [18:52:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.13, 22.98, 21.62 [18:53:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:53:29] RECOVERY - mw9 APT on mw9 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:53:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [18:54:07] Official RO time is 1 hour, 44 minutes and 2 seconds [18:54:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.61, 7.67, 7.76 [18:54:37] !log repool mw11 [18:54:38] !log (START) sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/maintenance/archives/patch-change_tag-rename-indexes.sql [18:54:39] done CosmicAlpha ^ [18:54:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.65, 21.11, 21.10 [18:54:58] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 9 backends are healthy [18:54:59] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.79, 22.43, 21.22 [18:55:39] Congratulations and Well Done everyone! [18:55:42] ok : [RESOLVED] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:55:56] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:56:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.69, 8.28, 7.94 [18:56:37] !log mw11: sudo service jobrunner start [18:56:59] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 31.22, 24.94, 22.25 [18:57:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.19, 5.80, 3.20 [18:58:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.97, 3.95, 3.61 [18:59:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:59:40] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.67, 5.99, 4.48 [19:00:08] [02mw-config] 07Naleksuh opened pull request 03#4268: site venue - 13https://git.io/JDvOV [19:00:30] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.17, 3.61, 3.53 [19:00:39] Thanks Sario [19:00:43] Well done CosmicAlpha [19:01:11] miraheze/mw-config - Naleksuh the build passed. [19:02:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.53, 4.62, 3.95 [19:02:55] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.36, 22.16, 21.44 [19:03:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 51.195.220.68/cpweb [19:04:42] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.88, 7.17, 5.08 [19:04:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.90, 23.19, 21.97 [19:05:45] RhinosF1: did you run 137updates.sql on all wikis before the upgrade? it is not in SAL is why I am asking. [19:05:59] CosmicAlpha: yes [19:06:29] [02mw-config] 07Universal-Omega closed pull request 03#4261: Drop 137updates.sql - 13https://git.io/JMdGY [19:06:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JDv3D [19:06:32] [02miraheze/mw-config] 07Universal-Omega 03f041dec - Drop 137updates.sql (#4261) [19:06:34] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 - 13https://git.io/vbvb3 [19:06:35] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 [19:06:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.55, 23.92, 22.39 [19:07:37] miraheze/mw-config - Universal-Omega the build passed. [19:07:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:07:59] !log [universalomega@mw11] starting deploy of {'pull': 'config', 'config': True, 'extensionlist': True} to all [19:08:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:56] !log [universalomega@mw11] finished deploy of {'pull': 'config', 'config': True, 'extensionlist': True} to all - SUCCESS in 56s [19:09:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:10:41] !log update default branch for miraheze/mediawiki to REL1_37 [19:10:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 24.00, 23.85, 22.77 [19:10:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:13:00] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.16, 25.06, 23.28 [19:13:03] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.77, 6.00, 5.89 [19:14:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:14:58] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 7.53, 6.68, 6.15 [19:17:00] !log [reception@mw11] starting deploy of {'world': True, 'l10n': True} to skip [19:17:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:22:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:23:36] !log [reception@mw11] finished deploy of {'world': True, 'l10n': True} to skip - SUCCESS in 394s [19:23:50] !log sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/maintenance/archives/patch-page-rename-name_title-index.sql [19:23:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [19:24:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:24:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.70, 3.64, 4.00 [19:24:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:57] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:26:19] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.87, 4.04 [19:26:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.67, 22.85, 23.89 [19:27:16] [02mw-config] 07Universal-Omega closed pull request 03#4263: Fix config naming for 1.37 - 13https://git.io/JMbfl [19:27:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JDvns [19:27:19] [02miraheze/mw-config] 07Universal-Omega 03d6fdb28 - Fix config naming for 1.37 (#4263) [19:27:20] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [19:27:22] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [19:27:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:28:10] !log [universalomega@mw11] starting deploy of {'pull': 'config', 'config': True} to all [19:28:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:28:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.69, 3.95 [19:28:19] miraheze/mw-config - Universal-Omega the build passed. [19:28:21] !log [universalomega@mw11] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [19:28:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:28:31] !log UPDATE mw_settings SET s_settings = REPLACE(s_settings,'wgRelatedArticlesFooterWhitelistedSkins', 'wgRelatedArticlesFooterAllowedSkins'); [19:28:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:30:14] !log [universalomega@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (START) [19:30:20] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.43, 4.05, 4.03 [19:30:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:31:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.26, 4.40, 5.58 [19:32:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.15, 7.18, 7.89 [19:33:17] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.31, 7.16, 7.81 [19:35:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.26, 5.18, 5.55 [19:36:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:36:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.12, 7.06, 7.91 [19:36:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.36, 22.70, 22.71 [19:37:17] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.62, 7.96, 7.98 [19:38:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.99, 7.47, 7.68 [19:38:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.10, 7.94, 8.13 [19:38:57] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.08, 23.09, 22.87 [19:38:58] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:39:07] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:39:22] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:40:01] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:40:24] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:40:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.72, 7.39, 7.62 [19:40:56] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15869 bytes in 1.250 second response time [19:41:06] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 1.543 second response time [19:41:20] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 2.048 second response time [19:42:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.55, 3.51, 3.90 [19:42:18] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:42:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JDvWG [19:42:28] [02miraheze/mw-config] 07Universal-Omega 031938d5c - Disable NumberedHeadings [19:42:29] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [19:42:31] [02mw-config] 07Universal-Omega opened pull request 03#4269: Disable NumberedHeadings - 13https://git.io/JDvWn [19:42:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.99, 23.81, 23.14 [19:43:03] [02mw-config] 07Universal-Omega closed pull request 03#4269: Disable NumberedHeadings - 13https://git.io/JDvWn [19:43:05] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDvW8 [19:43:06] [02miraheze/mw-config] 07Universal-Omega 03a906d25 - Disable NumberedHeadings (#4269) [19:43:08] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [19:43:09] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [19:43:23] !log [universalomega@mw11] starting deploy of {'pull': 'config', 'config': True} to all [19:43:30] miraheze/mw-config - Universal-Omega the build passed. [19:43:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:43:53] !log [universalomega@mw11] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 30s [19:44:06] miraheze/mw-config - Universal-Omega the build passed. [19:44:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:44:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.18, 6.85, 7.70 [19:44:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.93, 21.55, 22.40 [19:46:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:46:13] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.66, 3.85 [19:46:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.88, 7.23, 7.35 [19:48:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.28, 21.89, 22.22 [19:48:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [19:49:55] !log sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/maintenance/archives/patch-revision-rename-index.sql [19:50:02] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:50:23] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.78, 8.22, 7.87 [19:50:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:55] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.83, 22.71, 22.50 [19:51:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.59, 5.75, 5.89 [19:52:03] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:52:25] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 21672 bytes in 2.981 second response time [19:52:49] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:51] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:54:00] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.755 second response time [19:54:02] !log [universalomega@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (END - exit=0) [19:54:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:54:46] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 21672 bytes in 1.021 second response time [19:54:48] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15869 bytes in 0.324 second response time [19:54:54] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw10 [19:55:48] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.80, 5.48, 5.69 [19:56:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.74, 23.30, 22.71 [19:56:55] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [19:57:24] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.63, 1.81, 1.13 [19:58:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.96, 3.76, 3.97 [19:58:25] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:58:52] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03revert-4172-dependabot/submodules/REL1_37/skins/chameleon-de81cab [+0/-0/±1] 13https://git.io/JDv4l [19:58:54] [02miraheze/mediawiki] 07Universal-Omega 0338ac788 - Revert "Bump skins/chameleon from `a923c46` to `de81cab` (#4172)" [19:58:55] [02mediawiki] 07Universal-Omega created branch 03revert-4172-dependabot/submodules/REL1_37/skins/chameleon-de81cab - 13https://git.io/vbL5b [19:58:57] [02mediawiki] 07Universal-Omega opened pull request 03#4416: Revert "Bump skins/chameleon from `a923c46` to `de81cab`" - 13https://git.io/JDv44 [19:59:07] [02mediawiki] 07Universal-Omega closed pull request 03#4416: Revert "Bump skins/chameleon from `a923c46` to `de81cab`" - 13https://git.io/JDv44 [19:59:08] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JDv4B [19:59:10] [02miraheze/mediawiki] 07Universal-Omega 039fac990 - Revert "Bump skins/chameleon from `a923c46` to `de81cab` (#4172)" (#4416) [19:59:11] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03revert-4172-dependabot/submodules/REL1_37/skins/chameleon-de81cab [19:59:13] [02mediawiki] 07Universal-Omega deleted branch 03revert-4172-dependabot/submodules/REL1_37/skins/chameleon-de81cab - 13https://git.io/vbL5b [19:59:24] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.69, 1.36, 1.04 [19:59:41] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:59:45] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:00:27] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 5.872 second response time [20:00:43] !log [universalomega@mw11] starting deploy of {'world': True} to all [20:02:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:20] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:03:36] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:03:54] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:03:56] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:04:04] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:05:22] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 21672 bytes in 2.739 second response time [20:05:35] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.783 second response time [20:05:55] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 4.850 second response time [20:06:01] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 21673 bytes in 5.802 second response time [20:08:04] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.10, 3.00, 3.39 [20:08:14] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 4.520 second response time [20:09:32] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:10:19] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:10:29] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:12:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.26, 3.41, 3.49 [20:12:23] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 8.503 second response time [20:12:30] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 2.636 second response time [20:12:31] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 21672 bytes in 2.896 second response time [20:12:38] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 2.457 second response time [20:13:20] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:13:35] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15910 bytes in 0.039 second response time [20:14:25] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:14:33] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:14:51] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:14:52] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:16:26] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 4.458 second response time [20:16:37] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 4.793 second response time [20:16:56] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 7.841 second response time [20:16:57] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 21672 bytes in 8.842 second response time [20:17:36] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 7.163 second response time [20:20:00] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.50, 3.47 [20:20:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:21:01] !log mw10: sudo service php7.3-fpm restart [20:21:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.57, 3.51 [20:22:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:22:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 3.23, 4.73, 5.97 [20:23:23] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:23:55] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:23:59] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.74, 2.94, 3.29 [20:24:01] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.209 second response time [20:24:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:24:06] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:25:58] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 7.808 second response time [20:26:18] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:26:19] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:27:27] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 1.220 second response time [20:28:09] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 21670 bytes in 5.625 second response time [20:28:11] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 2.740 second response time [20:28:20] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 4.726 second response time [20:28:20] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 4.063 second response time [20:29:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:30:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:30:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.67, 5.30, 5.65 [20:33:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:33:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.41, 3.38 [20:34:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:35:14] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:35:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.57, 3.79, 3.52 [20:37:15] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.335 second response time [20:37:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:37:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.70, 3.51 [20:38:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:38:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 3.99, 5.28, 5.74 [20:39:40] Could a meta admin please add news to the main page about the 1.37 upgrade? [20:41:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.88, 3.33, 3.40 [20:44:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.47, 4.01, 5.00 [20:45:20] !log ALTER TABLE revision ALTER rev_timestamp DROP DEFAULT; on all wikis [20:45:34] MacFan4000: on it [20:45:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:02] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [20:48:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.22, 5.45, 5.40 [20:50:44] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.70, 4.29, 3.72 [20:50:53] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 8.162 second response time [20:51:19] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:52:02] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:52:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.92, 5.69, 5.50 [20:52:55] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 3.242 second response time [20:52:56] !log [universalomega@mw11] finished deploy of {'world': True} to all - SUCCESS in 3132s [20:53:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:53:19] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 21672 bytes in 0.906 second response time [20:54:32] RhinosF1: is flaggedimages table safe to drop? [20:54:47] CosmicAlpha: should be [20:54:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 8.56, 6.46, 5.78 [20:55:01] Why did you empty pruneRevData from the list [20:55:42] I thought that was unrelated to what you were doing on test3 for 1.37, since it was not a new script? [20:55:46] is it needed? [20:58:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.86, 3.47, 1.93 [20:59:14] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.455 second response time [20:59:14] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 3.233 second response time [20:59:15] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:59:23] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:59:48] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:59:54] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:00:02] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:00:04] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:00:09] CosmicAlpha: it'll save us db space [21:00:20] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 9.674 second response time [21:00:22] It's supposed to store less in 1.37 I believe [21:00:33] ok, I guess we can add it back to list then. [21:00:53] I'll run that, the log removals and image metadata later [21:00:58] In about an hour or so [21:01:13] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 1.271 second response time [21:01:14] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.015 second response time [21:01:14] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15875 bytes in 0.345 second response time [21:01:20] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.047 second response time [21:01:48] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 21672 bytes in 0.709 second response time [21:01:58] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15909 bytes in 0.096 second response time [21:02:01] RhinosF1: I was just about to run log removals, and dropping flaggedimages. want me to wait then? [21:02:02] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:02:16] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15909 bytes in 0.027 second response time [21:02:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.21, 1.74, 1.58 [21:04:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.35, 1.31, 1.43 [21:06:02] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:07:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:08:02] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:09:23] [02mw-config] 07Universal-Omega closed pull request 03#4260: Update SQL for CommentStreams for 1.37 - 13https://git.io/JMdst [21:09:25] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 - 13https://git.io/vbvb3 [21:09:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JDvwh [21:09:28] [02miraheze/mw-config] 07Universal-Omega 0301772cb - Update SQL for CommentStreams for 1.37 (#4260) [21:09:29] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 [21:09:56] !log [universalomega@mw11] starting deploy of {'pull': 'config', 'config': True} to all [21:10:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:10:24] !log [universalomega@mw11] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 28s [21:10:31] miraheze/mw-config - Universal-Omega the build passed. [21:10:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:10:47] !log [universalomega@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/commentstreams.json /srv/mediawiki/w/extensions/CommentStreams/maintenance/MigrateToAbstractSchema.php (START) [21:10:54] !log [universalomega@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/commentstreams.json /srv/mediawiki/w/extensions/CommentStreams/maintenance/MigrateToAbstractSchema.php (END - exit=0) [21:10:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:11:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:11:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:14:17] !log update mathoid on mw* and mwtask1 [21:14:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:14:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.83, 3.99 [21:15:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:16:37] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.55, 4.53, 4.20 [21:19:52] I might end up making a phab task for Structured Discussions, it's not been playing nicely for me and other users on the reception wiki side of things [21:21:36] Is SQL stuff still running? https://phabricator.miraheze.org/T8396#169645 looks to be an issue with Moderation [21:21:37] [url] ⚓ T8396 Error after upgrading miraheze to 1.37.0 | phabricator.miraheze.org [21:23:53] CosmicAlpha: up to you [21:24:00] I'm happy to [21:25:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:26:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.90, 3.71, 3.95 [21:34:27] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 17.46, 20.40, 23.57 [21:34:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 3.84, 4.84, 5.68 [21:36:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.09, 5.13, 5.66 [21:37:44] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JDvi3 [21:37:46] [02miraheze/mw-config] 07RhinosF1 033c054a9 - Experimental: update for 1.37 [21:37:47] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-2 - 13https://git.io/vbvb3 [21:37:49] [02mw-config] 07RhinosF1 opened pull request 03#4270: Experimental: update for 1.37 - 13https://git.io/JDvis [21:38:08] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JDvi8 [21:38:10] [02miraheze/mw-config] 07github-actions 0392a9faa - CI: lint code to MediaWiki standards [21:38:14] [02mw-config] 07github-actions[bot] synchronize pull request 03#4270: Experimental: update for 1.37 - 13https://git.io/JDvis [21:38:25] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 28.58, 23.78, 24.04 [21:38:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.22, 2.80, 3.38 [21:38:34] Voidwalker: SQL should be done, though maybe missed something with Moderation. Will look. [21:38:49] miraheze/mw-config - RhinosF1 the build passed. [21:38:54] yeah, Error 1054: Unknown column 'mod_type' in 'where clause' doesn't look right [21:39:07] ah, ok, looking [21:39:19] also, something's off with that CI: lint code [21:39:37] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JDvio [21:39:39] [02miraheze/mw-config] 07RhinosF1 037f47bc4 - Update Defines.php [21:39:40] [02mw-config] 07RhinosF1 synchronize pull request 03#4270: Experimental: update for 1.37 - 13https://git.io/JDvis [21:40:07] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JDviX [21:40:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:40:08] [02miraheze/mw-config] 07github-actions 036977b83 - CI: lint code to MediaWiki standards [21:40:10] [02mw-config] 07github-actions[bot] synchronize pull request 03#4270: Experimental: update for 1.37 - 13https://git.io/JDvis [21:40:18] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JDvi1 [21:40:20] [02miraheze/mw-config] 07RhinosF1 031f09b3e - Remove test3 files hack [21:40:21] [02mw-config] 07RhinosF1 synchronize pull request 03#4270: Experimental: update for 1.37 - 13https://git.io/JDvis [21:40:42] [02mw-config] 07RhinosF1 edited pull request 03#4270: Experimental: update for 1.37 - 13https://git.io/JDvis [21:40:46] miraheze/mw-config - RhinosF1 the build passed. [21:41:20] miraheze/mw-config - RhinosF1 the build passed. [21:41:41] Voidwalker: I'll manually fix it [21:41:54] CosmicAlpha: my oatch is needed before the image metadata script [21:42:14] I'm gonna run it on experimental for sanity first [21:42:19] And let it settle for a day [21:42:24] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.39, 23.35, 23.87 [21:42:32] Voidwalker: hmm, moderation did not have updates to sql this time, I do not know why it would be happening. [21:42:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.61, 5.81, 5.92 [21:44:23] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 27.74, 24.10, 24.01 [21:45:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:46:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:46:23] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.07, 23.14, 23.69 [21:47:35] yeah, no SQL updates, but does look like extension update [21:48:36] could be the field was never used until https://github.com/edwardspec/mediawiki-moderation/commit/cc0091d9e3004563149848bdc8b05ab18e082c00 [21:48:37] [url] (cleanup) Remove B/C code for supporting obsolete (pre-1.35) DB schema · edwardspec/mediawiki-moderation@cc0091d · GitHub | github.com [21:49:00] or some other weirdness that made it not apply until now [21:49:26] oh, so maybe the patch does need then? https://github.com/edwardspec/mediawiki-moderation/blob/master/sql/patch-moderation-mod_type.sql [21:49:27] [url] mediawiki-moderation/patch-moderation-mod_type.sql at master · edwardspec/mediawiki-moderation · GitHub | github.com [21:49:45] yeah, but it should have been needed much sooner [21:50:03] yeah thats what I was thinking we would not need to now. [21:50:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.26, 3.21 [21:51:04] might be worth running a check on all moderation wikis to see if they have the right columns in all their tables [21:51:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:51:34] We really need the schema drift tool [21:52:03] yeah [21:52:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.81, 3.18, 3.20 [21:53:43] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-6 [+0/-0/±1] 13https://git.io/JDvXx [21:53:45] [02miraheze/mw-config] 07RhinosF1 03a9f4d80 - CreateWiki: Core is now fully abstract schema [21:53:46] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-6 - 13https://git.io/vbvb3 [21:53:48] [02mw-config] 07RhinosF1 opened pull request 03#4271: CreateWiki: Core is now fully abstract schema - 13https://git.io/JDvXp [21:53:57] CosmicAlpha: also ^ [21:54:14] Voidwalker: Amir created one [21:54:20] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.70, 22.78, 22.97 [21:54:47] miraheze/mw-config - RhinosF1 the build passed. [21:54:49] It only works with abstract schema though [21:54:51] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.90, 5.54, 5.52 [21:55:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:56:20] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.66, 23.13, 23.08 [21:56:32] Voidwalker: https://github.com/Ladsgroup/db-analyzor-tools/blob/master/db_drift_checker.py [21:56:32] [url] db-analyzor-tools/db_drift_checker.py at master · Ladsgroup/db-analyzor-tools · GitHub | github.com [21:56:39] [02mw-config] 07Universal-Omega closed pull request 03#4271: CreateWiki: Core is now fully abstract schema - 13https://git.io/JDvXp [21:56:41] [02mw-config] 07Universal-Omega deleted branch 03RhinosF1-patch-6 - 13https://git.io/vbvb3 [21:56:42] [02miraheze/mw-config] 07Universal-Omega deleted branch 03RhinosF1-patch-6 [21:56:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDv1R [21:56:45] [02miraheze/mw-config] 07RhinosF1 033947564 - CreateWiki: Core is now fully abstract schema (#4271) [21:56:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.93, 5.61, 5.57 [21:57:44] miraheze/mw-config - Universal-Omega the build passed. [22:00:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:02:11] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JDvMZ [22:02:12] [02miraheze/CreateWiki] 07Universal-Omega 0323cc28d - Update workflow (#265) [22:02:19] [02CreateWiki] 07Universal-Omega closed pull request 03#265: Update workflow - 13https://git.io/JMbSw [22:02:20] [02CreateWiki] 07Universal-Omega deleted branch 03update-ci - 13https://git.io/vpJTL [22:02:22] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03update-ci [22:02:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.51, 4.51, 5.05 [22:03:07] [02miraheze/RemovePII] 07Universal-Omega deleted branch 03REL1_36 [22:03:17] [02RemovePII] 07Universal-Omega deleted branch 03REL1_36 - 13https://git.io/Jsfpj [22:07:56] miraheze/CreateWiki - Universal-Omega the build passed. [22:08:16] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.91, 21.66, 21.65 [22:08:49] Can one of u see general on discord [22:09:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.42, 3.28 [22:09:55] In a minute [22:10:15] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.96, 21.64, 21.65 [22:11:37] !log [@mw11] starting deploy of {'config': True} to all [22:11:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:11:55] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 17s [22:12:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:13:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 3.17, 3.23 [22:15:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:15:24] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.82, 5.39, 5.10 [22:15:34] [02mw-config] 07RhinosF1 closed pull request 03#4252: MassMessage: enable global on test3 - 13https://git.io/JMSRF [22:15:35] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-5 [22:15:37] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-5 - 13https://git.io/vbvb3 [22:15:38] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDvDE [22:15:40] [02miraheze/mw-config] 07RhinosF1 03412b43e - MassMessage: enable global on test3 (#4252) [22:15:44] [02mw-config] 07RhinosF1 closed pull request 03#4251: LS: fail CentralNotice over on test3 - 13https://git.io/JMSRr [22:15:45] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-4 - 13https://git.io/vbvb3 [22:15:47] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-4 [22:15:48] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDvD2 [22:15:50] [02miraheze/mw-config] 07RhinosF1 0342137c8 - LS: fail CentralNotice over on test3 (#4251) [22:15:57] [02mw-config] 07RhinosF1 closed pull request 03#4250: test3: shift GUP & GCSS/JS - 13https://git.io/JMSt0 [22:15:58] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDvDa [22:16:00] [02miraheze/mw-config] 07RhinosF1 03fa8f3ac - test3: shift GUP & GCSS/JS (#4250) [22:16:01] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-3 - 13https://git.io/vbvb3 [22:16:03] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-3 [22:16:13] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 28.51, 23.73, 22.34 [22:16:17] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:16:44] miraheze/mw-config - RhinosF1 the build passed. [22:16:52] miraheze/mw-config - RhinosF1 the build passed. [22:17:02] miraheze/mw-config - RhinosF1 the build passed. [22:17:18] PROBLEM - mw11 Puppet on mw11 is WARNING: WARNING: Puppet is currently disabled, message: merging all the changes, last run 5 minutes ago with 0 failures [22:17:19] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.02, 4.40, 4.77 [22:17:47] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:17:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.13, 4.27, 5.67 [22:17:52] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:17:53] mw11 is me [22:18:05] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [22:18:44] paladox: around [22:18:55] semi [22:19:11] paladox: think i found cause [22:19:18] for what [22:19:23] https://www.irccloud.com/pastebin/0tn0esHO/ [22:19:31] paladox: that's you right ^ [22:19:45] yes [22:19:51] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 1.779 second response time [22:20:08] paladox: ok, didn't expect it at first [22:20:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.88, 21.38, 23.77 [22:21:02] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.53, 1.85, 1.21 [22:21:09] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.94, 5.46, 5.13 [22:21:13] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:21:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:21:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.16, 3.29, 4.94 [22:21:59] !log [@mw11] starting deploy of {'config': True} to all [22:22:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:22:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:22:15] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 15s [22:22:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:23:00] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.43, 1.61, 1.20 [22:23:04] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 4.11, 4.98, 5.00 [22:23:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.57, 3.28 [22:25:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.39, 3.11, 3.14 [22:25:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:26:10] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.27, 22.97, 23.19 [22:26:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.07, 23.91, 23.78 [22:27:48] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.40, 5.41, 5.30 [22:28:09] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.29, 23.58, 23.36 [22:28:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:28:50] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 9.37, 6.76, 5.65 [22:29:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.26, 3.54, 3.33 [22:29:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.93, 5.96, 5.55 [22:30:09] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 18.73, 22.35, 22.98 [22:30:36] !log cleanup old flaggedrevs logs from all wikis [22:30:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:30:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.11, 5.46, 5.30 [22:30:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.84, 22.37, 23.34 [22:31:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.26, 5.86, 5.54 [22:32:12] !log also prune all revision data [22:32:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:33:15] CosmicAlpha: doing flagged images now [22:33:32] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JDvSc [22:33:34] [02miraheze/mediawiki] 07Universal-Omega 034bd1f64 - Update ManageWiki [22:33:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:34:10] RhinosF1: OK, thanks! [22:34:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 4.19, 4.70, 5.03 [22:35:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.07, 3.97, 3.54 [22:35:54] !log drop flaggedimages everywhere [22:36:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:36:11] !log [universalomega@mw11] starting deploy of {'world': True} to all [22:36:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:36:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 32.56, 24.69, 23.73 [22:38:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.45, 5.74, 5.37 [22:39:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.04, 3.85, 3.61 [22:39:10] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDv9m [22:39:11] [02miraheze/mw-config] 07RhinosF1 03061ab79 - LS: disable global interwiki on test3 [22:40:06] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.61, 24.30, 23.61 [22:40:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:40:18] miraheze/mw-config - RhinosF1 the build passed. [22:40:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.39, 5.75, 5.43 [22:41:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.94, 4.80, 3.99 [22:41:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.57, 5.85, 5.84 [22:42:50] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.60, 6.20, 5.64 [22:42:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.10, 22.79, 23.41 [22:43:27] !log [@mw11] starting deploy of {'config': True} to all [22:44:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:44:07] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 39s [22:44:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:44:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:45:48] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.65, 5.89, 5.83 [22:46:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:46:37] !log [universalomega@mw11] finished deploy of {'world': True} to all - SUCCESS in 625s [22:46:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:47:00] Fatal error on gratisdatawiki when viewing RC Feed, but doesn't on meta [22:47:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.94, 5.93, 5.86 [22:48:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.60, 24.65, 23.81 [22:49:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.68, 6.54, 6.10 [22:50:35] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:40] hmm, don't think I can confirm, do you have a full error message? [22:51:09] PROBLEM - ping4 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 123.27 ms [22:52:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:52:37] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:52:52] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:52:55] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:54:10] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [22:54:16] !log [@test3] starting deploy of {'config': True} to skip [22:54:17] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:54:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:54:37] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.334 second response time [22:54:44] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 21673 bytes in 0.270 second response time [22:54:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.17, 5.77, 5.94 [22:54:51] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.503 second response time [22:54:52] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.394 second response time [22:54:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:55:08] RECOVERY - ping4 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 112.53 ms [22:56:04] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:56:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:57:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.07, 3.53, 3.98 [22:57:28] I also can not save edits too [22:59:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.47, 3.76, 4.00 [23:00:01] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:02] !log [@mw11] starting deploy of {'l10nupdate': True} to all [23:00:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 8.11, 5.90, 5.83 [23:02:00] ugochimobi: Call to undefined method Wikibase\Repo\WikibaseRepo::getDefaultInstance() [23:02:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [23:02:10] you had a task from 1.36 right? [23:02:44] that was some content model stuff [23:03:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.80, 3.95 [23:03:51] ugochimobi: is the issue new with 1.37 or old? [23:04:04] new issue of course [23:04:30] CosmicAlpha: another undefined method ^ [23:04:34] it was after the upgrade [23:05:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.85, 3.94 [23:06:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:06:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 3.13, 5.05, 5.59 [23:06:53] PROBLEM - test3 APT on test3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:07:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.48, 3.72, 3.88 [23:08:51] RECOVERY - test3 APT on test3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:08:52] PROBLEM - test3 Current Load on test3 is CRITICAL: CRITICAL - load average: 5.77, 3.71, 1.95 [23:09:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.57, 4.11, 5.80 [23:09:53] Pywikibot tells me this `internal_api_error_Error: [62f416c04dd2d03cfe705ceb] Caught exception of type Error` [23:10:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:10:23] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:10:25] both on famedatawiki [23:10:52] RECOVERY - test3 Current Load on test3 is OK: OK - load average: 0.94, 2.58, 1.75 [23:13:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.18, 4.95, 5.71 [23:14:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb [23:14:10] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [23:14:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 2.91, 4.05, 4.90 [23:17:03] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:17:05] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:17:23] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:17:39] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:18:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.57, 5.98, 5.54 [23:19:03] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 21673 bytes in 1.030 second response time [23:19:06] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 0.352 second response time [23:20:47] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:20:52] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:21:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.54, 3.60, 3.54 [23:21:06] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:21:12] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:21:13] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:21:40] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 8.742 second response time [23:21:50] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 5.937 second response time [23:21:56] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:21:58] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:22:43] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 21673 bytes in 1.172 second response time [23:22:48] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15909 bytes in 0.305 second response time [23:22:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 3.86, 5.73, 5.61 [23:23:00] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15915 bytes in 0.008 second response time [23:23:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.98, 3.50, 3.50 [23:23:08] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.034 second response time [23:23:12] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15909 bytes in 0.369 second response time [23:23:54] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 21671 bytes in 0.594 second response time [23:24:50] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 8.50, 6.56, 5.90 [23:25:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.46, 3.97, 3.65 [23:26:06] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 6.008 second response time [23:26:18] @RhinosF1 I added the wikis to this task [23:26:20] .task T8396 [23:26:27] https://phabricator.miraheze.org/T8396 - Error after upgrading miraheze to 1.37.0, authored by Velerwiki, assigned to None, Priority: Unbreak Now!, Status: Open [23:27:33] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:28:50] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.18, 5.85, 5.78 [23:28:56] PROBLEM - test3 APT on test3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:29:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.94, 3.74, 3.66 [23:29:08] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:29:36] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 7.298 second response time [23:30:04] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:30:50] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 10.72, 7.55, 6.39 [23:30:52] RECOVERY - test3 APT on test3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:31:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.42, 4.42, 3.92 [23:31:15] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 4.386 second response time [23:31:43] !log [@mw11] finished deploy of {'l10nupdate': True} to all - FAIL: [35072, 0, 0, 0, 0, 0, 0, 0, 0] in 1901s [23:31:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:32:03] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 21673 bytes in 0.320 second response time [23:32:14] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:35:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.15, 3.44, 3.66 [23:37:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:40:26] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 2424s [23:40:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:41:07] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.81, 2.71, 3.30 [23:43:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [23:43:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:44:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CreateWiki-23cc28d [+0/-0/±1] 13https://git.io/JDvAp [23:44:38] [02miraheze/mediawiki] 07dependabot[bot] 03f1db99c - Bump extensions/CreateWiki from `9a3be23` to `23cc28d` [23:44:39] [02mediawiki] 07dependabot[bot] opened pull request 03#4417: Bump extensions/CreateWiki from `9a3be23` to `23cc28d` - 13https://git.io/JDvAh [23:44:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CreateWiki-23cc28d - 13https://git.io/vbL5b [23:44:42] [02mediawiki] 07dependabot[bot] labeled pull request 03#4417: Bump extensions/CreateWiki from `9a3be23` to `23cc28d` - 13https://git.io/JDvAh [23:44:44] [02mediawiki] 07dependabot[bot] labeled pull request 03#4417: Bump extensions/CreateWiki from `9a3be23` to `23cc28d` - 13https://git.io/JDvAh [23:44:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/IncidentReporting-583cede [+0/-0/±1] 13https://git.io/JDvxe [23:44:47] [02miraheze/mediawiki] 07dependabot[bot] 03bc74259 - Bump extensions/IncidentReporting from `344c2a5` to `583cede` [23:44:48] [02mediawiki] 07dependabot[bot] opened pull request 03#4418: Bump extensions/IncidentReporting from `344c2a5` to `583cede` - 13https://git.io/JDvxv [23:44:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/IncidentReporting-583cede - 13https://git.io/vbL5b [23:44:51] [02mediawiki] 07dependabot[bot] labeled pull request 03#4418: Bump extensions/IncidentReporting from `344c2a5` to `583cede` - 13https://git.io/JDvxv [23:44:53] [02mediawiki] 07dependabot[bot] labeled pull request 03#4418: Bump extensions/IncidentReporting from `344c2a5` to `583cede` - 13https://git.io/JDvxv [23:44:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/QuizGame-e40a484 [+0/-0/±1] 13https://git.io/JDvxf [23:44:56] [02miraheze/mediawiki] 07dependabot[bot] 03c4ff132 - Bump extensions/QuizGame from `9b681b9` to `e40a484` [23:44:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/QuizGame-e40a484 - 13https://git.io/vbL5b [23:44:59] [02mediawiki] 07dependabot[bot] labeled pull request 03#4419: Bump extensions/QuizGame from `9b681b9` to `e40a484` - 13https://git.io/JDvxJ [23:45:00] [02mediawiki] 07dependabot[bot] labeled pull request 03#4419: Bump extensions/QuizGame from `9b681b9` to `e40a484` - 13https://git.io/JDvxJ [23:45:02] [02mediawiki] 07dependabot[bot] opened pull request 03#4419: Bump extensions/QuizGame from `9b681b9` to `e40a484` - 13https://git.io/JDvxJ [23:45:03] [02mediawiki] 07dependabot[bot] commented on pull request 03#4413: Bump extensions/QuizGame from `9b681b9` to `6be59b1` - 13https://git.io/JDvxU [23:45:05] [02mediawiki] 07dependabot[bot] closed pull request 03#4413: Bump extensions/QuizGame from `9b681b9` to `6be59b1` - 13https://git.io/JMb6d [23:45:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.62, 3.47, 3.50 [23:45:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikiDiscover-83eeb61 [+0/-0/±1] 13https://git.io/JDvxT [23:45:08] [02miraheze/mediawiki] 07dependabot[bot] 03c36a711 - Bump extensions/WikiDiscover from `d6aac79` to `83eeb61` [23:45:09] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/QuizGame-6be59b1 [23:45:11] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/QuizGame-6be59b1 - 13https://git.io/vbL5b [23:45:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikiDiscover-83eeb61 - 13https://git.io/vbL5b [23:45:14] [02mediawiki] 07dependabot[bot] labeled pull request 03#4420: Bump extensions/WikiDiscover from `d6aac79` to `83eeb61` - 13https://git.io/JDvxk [23:45:15] [02mediawiki] 07dependabot[bot] opened pull request 03#4420: Bump extensions/WikiDiscover from `d6aac79` to `83eeb61` - 13https://git.io/JDvxk [23:45:17] [02mediawiki] 07dependabot[bot] commented on pull request 03#4337: Bump extensions/WikiDiscover from `d6aac79` to `e04acca` - 13https://git.io/JDvxI [23:45:17] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:45:18] [02mediawiki] 07dependabot[bot] labeled pull request 03#4420: Bump extensions/WikiDiscover from `d6aac79` to `83eeb61` - 13https://git.io/JDvxk [23:45:20] [02mediawiki] 07dependabot[bot] closed pull request 03#4337: Bump extensions/WikiDiscover from `d6aac79` to `e04acca` - 13https://git.io/JM4te [23:45:21] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/WikiDiscover-e04acca [23:45:23] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/WikiDiscover-e04acca - 13https://git.io/vbL5b [23:45:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Evelution-ed1a35f [+0/-0/±1] 13https://git.io/JDvxm [23:45:26] [02miraheze/mediawiki] 07dependabot[bot] 033b20e30 - Bump skins/Evelution from `6d2b3bf` to `ed1a35f` [23:45:27] [02mediawiki] 07dependabot[bot] opened pull request 03#4421: Bump skins/Evelution from `6d2b3bf` to `ed1a35f` - 13https://git.io/JDvxY [23:45:29] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Evelution-ed1a35f - 13https://git.io/vbL5b [23:45:30] [02mediawiki] 07dependabot[bot] labeled pull request 03#4421: Bump skins/Evelution from `6d2b3bf` to `ed1a35f` - 13https://git.io/JDvxY [23:45:32] [02mediawiki] 07dependabot[bot] labeled pull request 03#4421: Bump skins/Evelution from `6d2b3bf` to `ed1a35f` - 13https://git.io/JDvxY [23:45:33] [02mediawiki] 07dependabot[bot] commented on pull request 03#4406: Bump skins/Evelution from `6d2b3bf` to `7975a1e` - 13https://git.io/JDvxO [23:45:35] [02mediawiki] 07dependabot[bot] closed pull request 03#4406: Bump skins/Evelution from `6d2b3bf` to `7975a1e` - 13https://git.io/JMb62 [23:45:36] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/skins/Evelution-7975a1e - 13https://git.io/vbL5b [23:45:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/YouTube-3cde500 [+0/-0/±1] 13https://git.io/JDvx3 [23:45:39] [02miraheze/mediawiki] 07dependabot[bot] 03d4e1eea - Bump extensions/YouTube from `cad9390` to `3cde500` [23:45:41] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/skins/Evelution-7975a1e [23:45:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/YouTube-3cde500 - 13https://git.io/vbL5b [23:45:44] [02mediawiki] 07dependabot[bot] opened pull request 03#4422: Bump extensions/YouTube from `cad9390` to `3cde500` - 13https://git.io/JDvxG [23:45:45] [02mediawiki] 07dependabot[bot] labeled pull request 03#4422: Bump extensions/YouTube from `cad9390` to `3cde500` - 13https://git.io/JDvxG [23:45:47] [02mediawiki] 07dependabot[bot] labeled pull request 03#4422: Bump extensions/YouTube from `cad9390` to `3cde500` - 13https://git.io/JDvxG [23:45:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:45:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Cargo-d9b9ae0 [+0/-0/±1] 13https://git.io/JDvxZ [23:45:50] [02miraheze/mediawiki] 07dependabot[bot] 031b9a0b2 - Bump extensions/Cargo from `ee250d4` to `d9b9ae0` [23:45:51] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Cargo-d9b9ae0 - 13https://git.io/vbL5b [23:45:53] [02mediawiki] 07dependabot[bot] opened pull request 03#4423: Bump extensions/Cargo from `ee250d4` to `d9b9ae0` - 13https://git.io/JDvxn [23:45:54] [02mediawiki] 07dependabot[bot] labeled pull request 03#4423: Bump extensions/Cargo from `ee250d4` to `d9b9ae0` - 13https://git.io/JDvxn [23:45:56] [02mediawiki] 07dependabot[bot] labeled pull request 03#4423: Bump extensions/Cargo from `ee250d4` to `d9b9ae0` - 13https://git.io/JDvxn [23:45:57] [02mediawiki] 07dependabot[bot] commented on pull request 03#4348: Bump extensions/YouTube from `cad9390` to `e9da06b` - 13https://git.io/JDvxW [23:45:59] [02mediawiki] 07dependabot[bot] closed pull request 03#4348: Bump extensions/YouTube from `cad9390` to `e9da06b` - 13https://git.io/JM4tz [23:46:00] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/YouTube-e9da06b - 13https://git.io/vbL5b [23:46:02] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/YouTube-e9da06b [23:46:03] [02mediawiki] 07dependabot[bot] commented on pull request 03#4407: Bump extensions/Cargo from `ee250d4` to `d9c03c0` - 13https://git.io/JDvx8 [23:46:05] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Translate-4729925 [+0/-0/±1] 13https://git.io/JDvx4 [23:46:06] [02miraheze/mediawiki] 07dependabot[bot] 0391bd4ec - Bump extensions/Translate from `b615834` to `4729925` [23:46:08] [02mediawiki] 07dependabot[bot] opened pull request 03#4424: Bump extensions/Translate from `b615834` to `4729925` - 13https://git.io/JDvxB [23:46:09] [02mediawiki] 07dependabot[bot] closed pull request 03#4407: Bump extensions/Cargo from `ee250d4` to `d9c03c0` - 13https://git.io/JMb6w [23:46:11] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Translate-4729925 - 13https://git.io/vbL5b [23:46:12] [02mediawiki] 07dependabot[bot] labeled pull request 03#4424: Bump extensions/Translate from `b615834` to `4729925` - 13https://git.io/JDvxB [23:46:14] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Cargo-d9c03c0 - 13https://git.io/vbL5b [23:46:15] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Cargo-d9c03c0 [23:46:17] [02mediawiki] 07dependabot[bot] labeled pull request 03#4424: Bump extensions/Translate from `b615834` to `4729925` - 13https://git.io/JDvxB [23:46:18] [02mediawiki] 07dependabot[bot] commented on pull request 03#4412: Bump extensions/Translate from `b615834` to `6a05b9d` - 13https://git.io/JDvx0 [23:46:20] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PDFEmbed-18560b0 [+0/-0/±1] 13https://git.io/JDvxE [23:46:21] [02miraheze/mediawiki] 07dependabot[bot] 032bb9fc0 - Bump extensions/PDFEmbed from `97ec549` to `18560b0` [23:46:23] [02mediawiki] 07dependabot[bot] closed pull request 03#4412: Bump extensions/Translate from `b615834` to `6a05b9d` - 13https://git.io/JMb69 [23:46:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PDFEmbed-18560b0 - 13https://git.io/vbL5b [23:46:26] [02mediawiki] 07dependabot[bot] opened pull request 03#4425: Bump extensions/PDFEmbed from `97ec549` to `18560b0` - 13https://git.io/JDvxu [23:46:27] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Translate-6a05b9d - 13https://git.io/vbL5b [23:46:29] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Translate-6a05b9d [23:46:30] [02mediawiki] 07dependabot[bot] labeled pull request 03#4425: Bump extensions/PDFEmbed from `97ec549` to `18560b0` - 13https://git.io/JDvxu [23:46:32] [02mediawiki] 07dependabot[bot] labeled pull request 03#4425: Bump extensions/PDFEmbed from `97ec549` to `18560b0` - 13https://git.io/JDvxu [23:46:33] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MatomoAnalytics-a4ee20b [+0/-0/±1] 13https://git.io/JDvxz [23:46:35] [02miraheze/mediawiki] 07dependabot[bot] 03c97a630 - Bump extensions/MatomoAnalytics from `9528cc3` to `a4ee20b` [23:46:36] [02mediawiki] 07dependabot[bot] closed pull request 03#4362: Bump extensions/PDFEmbed from `97ec549` to `cb3a8ea` - 13https://git.io/JM4qv [23:46:38] [02mediawiki] 07dependabot[bot] commented on pull request 03#4362: Bump extensions/PDFEmbed from `97ec549` to `cb3a8ea` - 13https://git.io/JDvxg [23:46:39] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MatomoAnalytics-a4ee20b - 13https://git.io/vbL5b [23:46:41] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/PDFEmbed-cb3a8ea - 13https://git.io/vbL5b [23:46:42] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/PDFEmbed-cb3a8ea [23:46:44] [02mediawiki] 07dependabot[bot] opened pull request 03#4426: Bump extensions/MatomoAnalytics from `9528cc3` to `a4ee20b` - 13https://git.io/JDvx2 [23:46:45] [02mediawiki] 07dependabot[bot] labeled pull request 03#4426: Bump extensions/MatomoAnalytics from `9528cc3` to `a4ee20b` - 13https://git.io/JDvx2 [23:46:47] [02mediawiki] 07dependabot[bot] labeled pull request 03#4426: Bump extensions/MatomoAnalytics from `9528cc3` to `a4ee20b` - 13https://git.io/JDvx2 [23:46:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-3afdaf9 [+0/-0/±1] 13https://git.io/JDvxV [23:46:50] [02miraheze/mediawiki] 07dependabot[bot] 03d21d4ef - Bump extensions/MirahezeMagic from `378ce3c` to `3afdaf9` [23:46:51] [02mediawiki] 07dependabot[bot] opened pull request 03#4427: Bump extensions/MirahezeMagic from `378ce3c` to `3afdaf9` - 13https://git.io/JDvxw [23:46:53] [02mediawiki] 07dependabot[bot] labeled pull request 03#4427: Bump extensions/MirahezeMagic from `378ce3c` to `3afdaf9` - 13https://git.io/JDvxw [23:46:54] [02mediawiki] 07dependabot[bot] commented on pull request 03#4401: Bump extensions/MatomoAnalytics from `9528cc3` to `a38aab2` - 13https://git.io/JDvxr [23:46:56] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-3afdaf9 - 13https://git.io/vbL5b [23:46:57] [02mediawiki] 07dependabot[bot] closed pull request 03#4401: Bump extensions/MatomoAnalytics from `9528cc3` to `a38aab2` - 13https://git.io/JMb6G [23:46:59] [02mediawiki] 07dependabot[bot] labeled pull request 03#4427: Bump extensions/MirahezeMagic from `378ce3c` to `3afdaf9` - 13https://git.io/JDvxw [23:47:00] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/MatomoAnalytics-a38aab2 [23:47:02] [02mediawiki] 07dependabot[bot] commented on pull request 03#4403: Bump extensions/MirahezeMagic from `378ce3c` to `6298756` - 13https://git.io/JDvxo [23:47:03] [02mediawiki] 07dependabot[bot] closed pull request 03#4403: Bump extensions/MirahezeMagic from `378ce3c` to `6298756` - 13https://git.io/JMb64 [23:47:05] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/MatomoAnalytics-a38aab2 - 13https://git.io/vbL5b [23:47:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.63, 3.21, 3.40 [23:47:06] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-6298756 - 13https://git.io/vbL5b [23:47:08] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-6298756 [23:47:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Moderation-956c332 [+0/-0/±1] 13https://git.io/JDvxK [23:47:11] [02miraheze/mediawiki] 07dependabot[bot] 0381db4d7 - Bump extensions/Moderation from `916fa2d` to `956c332` [23:47:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/RatePage-1182c92 [+0/-0/±1] 13https://git.io/JDvx6 [23:47:14] [02miraheze/mediawiki] 07dependabot[bot] 03dd7bd5a - Bump extensions/RatePage from `b774f5c` to `1182c92` [23:47:15] [02mediawiki] 07dependabot[bot] labeled pull request 03#4428: Bump extensions/Moderation from `916fa2d` to `956c332` - 13https://git.io/JDvxi [23:47:17] [02mediawiki] 07dependabot[bot] labeled pull request 03#4428: Bump extensions/Moderation from `916fa2d` to `956c332` - 13https://git.io/JDvxi [23:47:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/RatePage-1182c92 - 13https://git.io/vbL5b [23:47:20] [02mediawiki] 07dependabot[bot] labeled pull request 03#4429: Bump extensions/RatePage from `b774f5c` to `1182c92` - 13https://git.io/JDvxP [23:47:21] [02mediawiki] 07dependabot[bot] opened pull request 03#4428: Bump extensions/Moderation from `916fa2d` to `956c332` - 13https://git.io/JDvxi [23:47:23] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Moderation-956c332 - 13https://git.io/vbL5b [23:47:24] [02mediawiki] 07dependabot[bot] labeled pull request 03#4429: Bump extensions/RatePage from `b774f5c` to `1182c92` - 13https://git.io/JDvxP [23:47:26] [02mediawiki] 07dependabot[bot] opened pull request 03#4429: Bump extensions/RatePage from `b774f5c` to `1182c92` - 13https://git.io/JDvxP [23:47:27] [02mediawiki] 07dependabot[bot] commented on pull request 03#4386: Bump extensions/Moderation from `916fa2d` to `e0428ae` - 13https://git.io/JDvxX [23:47:29] [02mediawiki] 07dependabot[bot] closed pull request 03#4386: Bump extensions/Moderation from `916fa2d` to `e0428ae` - 13https://git.io/JMbKz [23:47:30] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Moderation-e0428ae [23:47:32] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Moderation-e0428ae - 13https://git.io/vbL5b [23:47:33] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageSchemas-58064fe [+0/-0/±1] 13https://git.io/JDvxM [23:47:35] [02miraheze/mediawiki] 07dependabot[bot] 034c5fcfd - Bump extensions/PageSchemas from `23c8865` to `58064fe` [23:47:36] [02mediawiki] 07dependabot[bot] opened pull request 03#4430: Bump extensions/PageSchemas from `23c8865` to `58064fe` - 13https://git.io/JDvxD [23:47:38] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageSchemas-58064fe - 13https://git.io/vbL5b [23:47:39] [02mediawiki] 07dependabot[bot] labeled pull request 03#4430: Bump extensions/PageSchemas from `23c8865` to `58064fe` - 13https://git.io/JDvxD [23:47:41] [02mediawiki] 07dependabot[bot] labeled pull request 03#4430: Bump extensions/PageSchemas from `23c8865` to `58064fe` - 13https://git.io/JDvxD [23:47:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikiForum-e76b1cf [+0/-0/±1] 13https://git.io/JDvxy [23:47:44] [02miraheze/mediawiki] 07dependabot[bot] 03683f225 - Bump extensions/WikiForum from `ba358e1` to `e76b1cf` [23:47:45] [02mediawiki] 07dependabot[bot] commented on pull request 03#4404: Bump extensions/PageSchemas from `23c8865` to `f28147a` - 13https://git.io/JDvxS [23:47:47] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikiForum-e76b1cf - 13https://git.io/vbL5b [23:47:48] [02mediawiki] 07dependabot[bot] labeled pull request 03#4431: Bump extensions/WikiForum from `ba358e1` to `e76b1cf` - 13https://git.io/JDvx9 [23:47:50] [02mediawiki] 07dependabot[bot] labeled pull request 03#4431: Bump extensions/WikiForum from `ba358e1` to `e76b1cf` - 13https://git.io/JDvx9 [23:47:51] [02mediawiki] 07dependabot[bot] closed pull request 03#4404: Bump extensions/PageSchemas from `23c8865` to `f28147a` - 13https://git.io/JMb60 [23:47:53] [02mediawiki] 07dependabot[bot] opened pull request 03#4431: Bump extensions/WikiForum from `ba358e1` to `e76b1cf` - 13https://git.io/JDvx9 [23:47:54] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/PageSchemas-f28147a [23:47:56] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/PageSchemas-f28147a - 13https://git.io/vbL5b [23:47:57] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Comments-d472224 [+0/-0/±1] 13https://git.io/JDvxH [23:47:59] [02miraheze/mediawiki] 07dependabot[bot] 0374e0304 - Bump extensions/Comments from `821b75a` to `d472224` [23:48:00] [02mediawiki] 07dependabot[bot] opened pull request 03#4432: Bump extensions/Comments from `821b75a` to `d472224` - 13https://git.io/JDvxQ [23:48:02] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Comments-d472224 - 13https://git.io/vbL5b [23:48:03] [02mediawiki] 07dependabot[bot] labeled pull request 03#4432: Bump extensions/Comments from `821b75a` to `d472224` - 13https://git.io/JDvxQ [23:48:05] [02mediawiki] 07dependabot[bot] labeled pull request 03#4432: Bump extensions/Comments from `821b75a` to `d472224` - 13https://git.io/JDvxQ [23:48:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/cldr-2941387 [+0/-0/±1] 13https://git.io/JDvxd [23:48:08] [02miraheze/mediawiki] 07dependabot[bot] 03918f59f - Bump extensions/cldr from `937150a` to `2941387` [23:48:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/cldr-2941387 - 13https://git.io/vbL5b [23:48:11] [02mediawiki] 07dependabot[bot] opened pull request 03#4433: Bump extensions/cldr from `937150a` to `2941387` - 13https://git.io/JDvxF [23:48:12] [02mediawiki] 07dependabot[bot] labeled pull request 03#4433: Bump extensions/cldr from `937150a` to `2941387` - 13https://git.io/JDvxF [23:48:14] [02mediawiki] 07dependabot[bot] labeled pull request 03#4433: Bump extensions/cldr from `937150a` to `2941387` - 13https://git.io/JDvxF [23:48:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageForms-7d124a4 [+0/-0/±1] 13https://git.io/JDvxb [23:48:17] [02miraheze/mediawiki] 07dependabot[bot] 0355f8b7f - Bump extensions/PageForms from `5d178f8` to `7d124a4` [23:48:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageForms-7d124a4 - 13https://git.io/vbL5b [23:48:20] [02mediawiki] 07dependabot[bot] commented on pull request 03#4387: Bump extensions/cldr from `937150a` to `f915358` - 13https://git.io/JDvxN [23:48:21] [02mediawiki] 07dependabot[bot] closed pull request 03#4387: Bump extensions/cldr from `937150a` to `f915358` - 13https://git.io/JMbKa [23:48:23] [02mediawiki] 07dependabot[bot] labeled pull request 03#4434: Bump extensions/PageForms from `5d178f8` to `7d124a4` - 13https://git.io/JDvxA [23:48:24] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/cldr-f915358 - 13https://git.io/vbL5b [23:48:26] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/cldr-f915358 [23:48:27] [02mediawiki] 07dependabot[bot] opened pull request 03#4434: Bump extensions/PageForms from `5d178f8` to `7d124a4` - 13https://git.io/JDvxA [23:48:29] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-0cf64b5 [+0/-0/±1] 13https://git.io/JDvxx [23:48:30] [02miraheze/mediawiki] 07dependabot[bot] 03a150ba0 - Bump extensions/UniversalLanguageSelector from `126936f` to `0cf64b5` [23:48:32] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-0cf64b5 - 13https://git.io/vbL5b [23:48:33] [02mediawiki] 07dependabot[bot] opened pull request 03#4435: Bump extensions/UniversalLanguageSelector from `126936f` to `0cf64b5` - 13https://git.io/JDvxp [23:48:35] [02mediawiki] 07dependabot[bot] commented on pull request 03#4402: Bump extensions/PageForms from `5d178f8` to `a4551cb` - 13https://git.io/JDvxh [23:48:36] [02mediawiki] 07dependabot[bot] labeled pull request 03#4434: Bump extensions/PageForms from `5d178f8` to `7d124a4` - 13https://git.io/JDvxA [23:48:38] [02mediawiki] 07dependabot[bot] labeled pull request 03#4435: Bump extensions/UniversalLanguageSelector from `126936f` to `0cf64b5` - 13https://git.io/JDvxp [23:48:39] [02mediawiki] 07dependabot[bot] closed pull request 03#4402: Bump extensions/PageForms from `5d178f8` to `a4551cb` - 13https://git.io/JMb6C [23:48:41] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/PageForms-a4551cb [23:48:42] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/PageForms-a4551cb - 13https://git.io/vbL5b [23:48:44] [02mediawiki] 07dependabot[bot] labeled pull request 03#4435: Bump extensions/UniversalLanguageSelector from `126936f` to `0cf64b5` - 13https://git.io/JDvxp [23:48:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DataDump-aa33858 [+0/-0/±1] 13https://git.io/JDvxj [23:48:47] [02miraheze/mediawiki] 07dependabot[bot] 0309eaeb9 - Bump extensions/DataDump from `87f48f8` to `aa33858` [23:48:48] [02mediawiki] 07dependabot[bot] commented on pull request 03#4397: Bump extensions/UniversalLanguageSelector from `126936f` to `1158e2b` - 13https://git.io/JDvpe [23:48:50] [02mediawiki] 07dependabot[bot] opened pull request 03#4436: Bump extensions/DataDump from `87f48f8` to `aa33858` - 13https://git.io/JDvpv [23:48:51] [02mediawiki] 07dependabot[bot] closed pull request 03#4397: Bump extensions/UniversalLanguageSelector from `126936f` to `1158e2b` - 13https://git.io/JMbKj [23:48:53] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DataDump-aa33858 - 13https://git.io/vbL5b [23:48:54] [02mediawiki] 07dependabot[bot] labeled pull request 03#4436: Bump extensions/DataDump from `87f48f8` to `aa33858` - 13https://git.io/JDvpv [23:48:56] [02mediawiki] 07dependabot[bot] labeled pull request 03#4436: Bump extensions/DataDump from `87f48f8` to `aa33858` - 13https://git.io/JDvpv [23:48:57] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-1158e2b [23:48:59] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-1158e2b - 13https://git.io/vbL5b [23:49:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/RottenLinks-c400776 [+0/-0/±1] 13https://git.io/JDvpU [23:49:02] [02miraheze/mediawiki] 07dependabot[bot] 03bc0da5c - Bump extensions/RottenLinks from `895c528` to `c400776` [23:49:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/RottenLinks-c400776 - 13https://git.io/vbL5b [23:49:05] [02mediawiki] 07dependabot[bot] opened pull request 03#4437: Bump extensions/RottenLinks from `895c528` to `c400776` - 13https://git.io/JDvpT [23:49:06] [02mediawiki] 07dependabot[bot] closed pull request 03#4410: Bump extensions/DataDump from `87f48f8` to `b5bb055` - 13https://git.io/JMb61 [23:49:08] [02mediawiki] 07dependabot[bot] commented on pull request 03#4410: Bump extensions/DataDump from `87f48f8` to `b5bb055` - 13https://git.io/JDvpk [23:49:09] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/DataDump-b5bb055 - 13https://git.io/vbL5b [23:49:11] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/DataDump-b5bb055 [23:49:12] [02mediawiki] 07dependabot[bot] labeled pull request 03#4437: Bump extensions/RottenLinks from `895c528` to `c400776` - 13https://git.io/JDvpT [23:49:14] [02mediawiki] 07dependabot[bot] labeled pull request 03#4437: Bump extensions/RottenLinks from `895c528` to `c400776` - 13https://git.io/JDvpT [23:49:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SpriteSheet-b7a1c91 [+0/-0/±1] 13https://git.io/JDvpt [23:49:17] [02miraheze/mediawiki] 07dependabot[bot] 0363ab75a - Bump extensions/SpriteSheet from `d62639c` to `b7a1c91` [23:49:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SpriteSheet-b7a1c91 - 13https://git.io/vbL5b [23:49:20] [02mediawiki] 07dependabot[bot] labeled pull request 03#4438: Bump extensions/SpriteSheet from `d62639c` to `b7a1c91` - 13https://git.io/JDvpq [23:49:21] [02mediawiki] 07dependabot[bot] opened pull request 03#4438: Bump extensions/SpriteSheet from `d62639c` to `b7a1c91` - 13https://git.io/JDvpq [23:49:23] [02mediawiki] 07dependabot[bot] labeled pull request 03#4438: Bump extensions/SpriteSheet from `d62639c` to `b7a1c91` - 13https://git.io/JDvpq [23:49:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalNewFiles-788cef8 [+0/-0/±1] 13https://git.io/JDvpm [23:49:26] [02miraheze/mediawiki] 07dependabot[bot] 03604025b - Bump extensions/GlobalNewFiles from `2ee8275` to `788cef8` [23:49:27] [02mediawiki] 07dependabot[bot] opened pull request 03#4439: Bump extensions/GlobalNewFiles from `2ee8275` to `788cef8` - 13https://git.io/JDvpY [23:49:29] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalNewFiles-788cef8 - 13https://git.io/vbL5b [23:49:30] [02mediawiki] 07dependabot[bot] labeled pull request 03#4439: Bump extensions/GlobalNewFiles from `2ee8275` to `788cef8` - 13https://git.io/JDvpY [23:49:32] [02mediawiki] 07dependabot[bot] labeled pull request 03#4439: Bump extensions/GlobalNewFiles from `2ee8275` to `788cef8` - 13https://git.io/JDvpY [23:49:33] [02mediawiki] 07dependabot[bot] commented on pull request 03#4396: Bump extensions/GlobalNewFiles from `2ee8275` to `b940b1c` - 13https://git.io/JDvpO [23:49:35] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/GlobalNewFiles-b940b1c - 13https://git.io/vbL5b [23:49:36] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/GlobalNewFiles-b940b1c [23:49:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Cosmos-244d95a [+0/-0/±1] 13https://git.io/JDvps [23:49:39] [02miraheze/mediawiki] 07dependabot[bot] 038ce0f31 - Bump skins/Cosmos from `f66fddf` to `244d95a` [23:49:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Cosmos-244d95a - 13https://git.io/vbL5b [23:49:42] [02mediawiki] 07dependabot[bot] opened pull request 03#4440: Bump skins/Cosmos from `f66fddf` to `244d95a` - 13https://git.io/JDvpG [23:49:44] [02mediawiki] 07dependabot[bot] labeled pull request 03#4440: Bump skins/Cosmos from `f66fddf` to `244d95a` - 13https://git.io/JDvpG [23:49:45] [02mediawiki] 07dependabot[bot] labeled pull request 03#4440: Bump skins/Cosmos from `f66fddf` to `244d95a` - 13https://git.io/JDvpG [23:49:47] [02mediawiki] 07dependabot[bot] closed pull request 03#4396: Bump extensions/GlobalNewFiles from `2ee8275` to `b940b1c` - 13https://git.io/JMbKx [23:49:48] [02mediawiki] 07dependabot[bot] commented on pull request 03#4394: Bump skins/Cosmos from `f66fddf` to `f5731e8` - 13https://git.io/JDvpZ [23:49:50] [02mediawiki] 07dependabot[bot] closed pull request 03#4394: Bump skins/Cosmos from `f66fddf` to `f5731e8` - 13https://git.io/JMbKd [23:49:51] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/skins/Cosmos-f5731e8 [23:49:53] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/skins/Cosmos-f5731e8 - 13https://git.io/vbL5b [23:50:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:51:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.80, 3.62 [23:54:02] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-1/±0] 13https://git.io/JDvhn [23:54:03] [02miraheze/mediawiki] 07Universal-Omega 0354a077c - Delete labeler.yml [23:54:05] [02mediawiki] 07Universal-Omega synchronize pull request 03#4415: Add labeler workflow - 13https://git.io/JMNQA [23:54:23] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JDvhc [23:54:24] [02miraheze/mediawiki] 07Universal-Omega 03527d900 - Update label.yml [23:54:26] [02mediawiki] 07Universal-Omega synchronize pull request 03#4415: Add labeler workflow - 13https://git.io/JMNQA [23:54:53] RhinosF1: ^ should be done now. [23:55:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.69, 3.66, 3.57 [23:55:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:55:48] @CosmicAlpha I still have `[677e3ac25e43c3100a57b084] 2021-12-07 23:55:19: Fatal exception of type "Error"` [23:56:45] CosmicAlpha: merge [23:57:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 3.14, 3.38 [23:57:15] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JDvhK [23:57:16] [02miraheze/mediawiki] 07dependabot[bot] 0367ba711 - Bump extensions/WikibaseLocalMedia from `82b7cf0` to `c925f24` (#4399) [23:57:18] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/WikibaseLocalMedia-c925f24 [23:57:19] [02mediawiki] 07Universal-Omega closed pull request 03#4399: Bump extensions/WikibaseLocalMedia from `82b7cf0` to `c925f24` - 13https://git.io/JMb6L [23:57:21] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/WikibaseLocalMedia-c925f24 - 13https://git.io/vbL5b [23:57:33] !log [universalomega@mw11] starting deploy of {'world': True} to all [23:57:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:58:10] ^ will fix Call to undefined method Wikibase\Repo\WikibaseRepo::getDefaultInstance() [23:58:56] CosmicAlpha: shall I merge your label or [23:58:59] PR? [23:59:00] RhinosF1: will do, also FYI the workflow will only work for dependabot branch format since it parses it to get the submodule name. So maybe we need a check to only run if user is dependabot? [23:59:11] We know [23:59:17] [02mediawiki] 07RhinosF1 closed pull request 03#4415: Add labeler workflow - 13https://git.io/JMNQA [23:59:19] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+1/-0/±0] 13https://git.io/JDvhd [23:59:20] [02miraheze/mediawiki] 07Universal-Omega 03a43d86d - Add labeler workflow (#4415) [23:59:22] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03Universal-Omega-patch-2 [23:59:28] [02mediawiki] 07RhinosF1 deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbL5b [23:59:34] It will fail otherwise. (causing the red x) [23:59:56] [02mediawiki] 07RhinosF1 commented on pull request 03#4440: Bump skins/Cosmos from `f66fddf` to `244d95a` - 13https://git.io/JDvhp [23:59:58] [02mediawiki] 07dependabot[bot] edited pull request 03#4440: Bump skins/Cosmos from `f66fddf` to `244d95a` - 13https://git.io/JDvpG