[00:00:22] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.69, 4.34, 6.52 [00:00:26] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 2.74, 4.23, 6.29 [00:03:11] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.43, 4.49, 6.48 [00:04:10] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:08:03] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.82, 21.37, 18.28 [00:10:13] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.85, 19.86, 18.09 [00:21:28] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 31.39, 26.10, 21.46 [00:23:07] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.26, 5.86, 4.80 [00:25:23] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.35, 5.50, 4.83 [00:28:08] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.69, 23.18, 21.95 [00:32:44] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:38:59] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.74, 18.08, 20.22 [00:44:09] !log install gdb on mw8 [00:44:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:47:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.30, 3.09 [00:51:36] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.66, 3.28 [00:53:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.02, 3.34, 3.21 [00:59:12] might mw8 being repooled be causing issues with login? I keep being kicked off every time my login request is processed via mw8 [00:59:26] wasn't having these issues before an hour ago [01:05:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 3.86, 3.52 [01:08:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.88, 3.58 [01:12:22] !log depool mw8 again. [01:12:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:15:38] PROBLEM - cp14 Varnish Backends on cp14 is CRITICAL: 1 backends are down. mw8 [01:15:44] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw8 [01:15:50] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw8 [01:16:01] PROBLEM - cp3 Stunnel Http for mw8 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.785 second response time [01:16:02] PROBLEM - cp14 Stunnel Http for mw8 on cp14 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.003 second response time [01:16:23] PROBLEM - cp13 Stunnel Http for mw8 on cp13 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.003 second response time [01:16:29] PROBLEM - cp12 Stunnel Http for mw8 on cp12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.235 second response time [01:16:46] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw8 [01:17:01] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.33, 3.91, 3.66 [01:17:34] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.004 second response time [01:18:24] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.46, 6.04, 5.10 [01:19:15] ty [01:19:33] PROBLEM - mw8 php-fpm on mw8 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.4' [01:23:07] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.53, 7.29, 5.84 [01:26:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.14, 3.94, 3.88 [01:27:37] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.05, 7.84, 6.49 [01:31:01] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.90, 3.86 [01:31:56] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.04, 7.55, 6.67 [01:33:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.90, 3.79, 3.82 [01:33:27] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 19997 bytes in 0.133 second response time [01:33:28] RECOVERY - mw8 php-fpm on mw8 is OK: PROCS OK: 33 processes with command name 'php-fpm7.4' [01:34:07] RECOVERY - cp13 Stunnel Http for mw8 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15697 bytes in 0.005 second response time [01:34:22] RECOVERY - cp3 Stunnel Http for mw8 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15705 bytes in 1.022 second response time [01:34:25] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.15, 6.62, 6.05 [01:34:29] RECOVERY - cp14 Stunnel Http for mw8 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15705 bytes in 0.004 second response time [01:34:35] RECOVERY - cp12 Stunnel Http for mw8 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15697 bytes in 0.306 second response time [01:35:39] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.49, 4.37, 4.06 [01:36:20] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.04, 7.68, 6.36 [01:36:39] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.31, 7.65, 6.54 [01:37:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.36, 3.92, 3.93 [01:38:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.82, 6.89, 6.25 [01:38:37] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.06, 7.81, 7.24 [01:38:48] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.35, 6.04, 6.09 [01:40:43] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.19, 6.63, 6.23 [01:49:45] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.23, 7.84, 7.34 [01:58:36] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.39, 7.88, 7.65 [02:00:50] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.59, 8.87, 8.07 [02:05:13] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.28, 3.24, 3.38 [02:05:16] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.06, 7.83, 7.82 [02:07:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.52, 7.19, 6.65 [02:07:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.12, 9.34, 8.42 [02:07:51] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.77, 19.19, 18.12 [02:09:17] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.42, 6.43, 6.46 [02:10:05] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.26, 17.80, 17.74 [02:18:43] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.32, 7.66, 7.96 [02:20:24] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.84, 7.63, 6.90 [02:20:38] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.26, 7.69, 7.01 [02:22:38] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.69, 7.21, 6.84 [02:23:20] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.46, 7.66, 7.82 [02:24:58] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 13.15, 8.95, 7.49 [02:25:11] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 12.82, 9.43, 7.79 [02:27:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.04, 3.78, 3.52 [02:27:47] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.17, 7.74, 7.91 [02:31:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.71, 3.10, 3.30 [02:31:47] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.31, 7.48, 7.46 [02:34:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.07, 7.50, 7.45 [02:34:34] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.30, 8.37, 8.01 [02:36:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.22, 3.28 [02:38:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.48, 3.02, 3.20 [02:40:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.04, 7.80, 7.88 [02:41:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.69, 7.36, 7.85 [02:42:52] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.26, 7.67, 7.76 [02:49:11] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.19, 5.71, 6.80 [02:52:13] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.19, 5.81, 6.70 [03:01:48] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.62, 6.70, 6.85 [03:04:48] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.83, 7.52, 7.29 [03:06:21] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.68, 9.13, 7.77 [03:07:49] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.90, 8.69, 7.41 [03:10:02] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.88, 7.76, 7.26 [03:10:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.11, 3.92, 3.47 [03:11:31] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.27, 7.60, 7.53 [03:12:16] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.11, 8.21, 7.48 [03:12:43] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.94, 3.54 [03:13:00] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.58, 7.87, 7.71 [03:14:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.02, 7.09, 7.18 [03:19:34] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.50, 4.04, 3.76 [03:22:25] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.95, 5.76, 6.68 [03:23:15] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.18, 5.95, 6.65 [03:24:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.13, 3.76, 3.75 [03:26:16] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.38, 5.70, 6.63 [03:31:07] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.65, 7.06, 7.07 [03:33:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.44, 2.96, 3.40 [03:33:17] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.07, 7.47, 7.22 [03:34:39] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.52, 7.21, 6.72 [03:38:18] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.52, 3.61, 3.55 [03:39:25] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.81, 7.15, 6.65 [03:40:18] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.29, 7.96, 7.78 [03:41:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.04, 6.88, 6.88 [03:43:39] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.51, 6.31, 6.66 [03:43:45] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.23, 6.22, 6.43 [03:47:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.10, 3.73, 3.76 [03:51:16] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.20, 5.90, 6.80 [03:56:17] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.45, 2.88, 3.33 [04:08:35] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.62, 6.41, 6.07 [04:10:40] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.43, 5.67, 5.84 [04:33:56] That doesn't make a whole lot of sense to me, sounds like there's a lot of funny business with the upgraded servers. [05:05:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.10, 6.33, 5.31 [05:06:34] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.59, 7.08, 5.92 [05:07:43] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.99, 5.99, 5.31 [05:10:56] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.02, 6.37, 5.95 [05:15:31] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.56, 8.42, 6.89 [05:17:34] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.35, 7.97, 6.92 [05:17:43] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.74, 7.37, 6.17 [05:19:47] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 7.79, 8.27, 7.19 [05:19:56] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.62, 7.07, 6.22 [05:24:03] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.08, 7.88, 6.74 [05:24:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.47, 7.71, 7.24 [05:28:22] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.08, 7.74, 7.01 [05:30:47] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.12, 7.14, 6.63 [05:32:45] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.04, 8.47, 7.42 [05:32:47] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.77, 8.13, 7.52 [05:34:59] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.75, 7.54, 7.38 [05:35:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.48, 3.30 [05:37:28] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.56, 7.90, 7.27 [05:39:27] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.16, 7.94, 7.76 [05:40:03] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.66, 3.90, 3.49 [05:41:25] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.15, 7.72, 7.38 [05:43:37] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.96, 7.61, 7.37 [05:43:48] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.58, 7.85, 7.72 [05:44:00] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.14, 8.09, 7.46 [05:44:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.25, 3.64, 3.49 [05:44:36] !log disabled 2FA for a user [05:44:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:44:54] paladox: if we don't figure out what's going on with mw8 do you think it would be a terrible idea to go back to buster? [05:46:00] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.64, 7.60, 7.64 [05:46:13] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.76, 7.18, 7.20 [05:46:46] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.99, 3.33, 3.39 [05:50:40] !log MariaDB [tscwiki]> INSERT INTO slot_roles (role_id, role_name) VALUES (1, 'main'); [05:50:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:52:21] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.84, 6.08, 6.73 [05:52:47] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.55, 7.25, 7.11 [05:56:59] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.16, 7.59, 7.39 [05:59:24] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.06, 7.70, 7.38 [06:01:21] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.17, 7.52, 7.48 [06:03:36] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.30, 7.90, 7.60 [06:03:50] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.70, 8.11, 7.61 [06:03:52] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.20, 8.05, 7.17 [06:05:57] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.27, 7.67, 7.56 [06:06:06] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.81, 7.85, 7.57 [06:12:31] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.32, 7.01, 7.23 [06:12:33] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.12, 5.95, 6.80 [06:23:19] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.23, 6.10, 6.65 [06:23:21] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.19, 6.20, 6.72 [06:28:07] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.43, 7.18, 6.99 [06:28:11] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.25, 7.96, 7.22 [06:28:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.27, 7.29, 6.99 [06:30:18] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.64, 7.43, 7.11 [06:32:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.58, 7.78, 7.28 [06:33:02] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.55, 8.17, 7.39 [06:34:04] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:39:09] !log reception@mwtask1:/srv/mediawiki/w/maintenance$ sudo -u www-data php sql.php /srv/mediawiki/w/extensions/DiscussionTools/sql/mysql/discussiontools_subscription.sql --wiki polcompballwiki [06:39:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:39:39] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.99, 7.79, 7.58 [06:41:14] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.55, 7.87, 7.80 [06:43:25] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.09, 7.81, 7.77 [06:43:36] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.52, 7.33, 7.84 [06:45:34] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.67, 7.37, 7.61 [06:45:49] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.26, 7.60, 7.86 [06:46:10] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.47, 7.72, 7.57 [06:47:59] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.99, 7.33, 7.73 [06:48:20] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.65, 7.78, 7.64 [06:56:12] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.44, 5.74, 6.77 [07:03:33] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.34, 5.88, 6.66 [07:09:30] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.84, 6.25, 6.72 [07:12:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.93, 3.40, 3.07 [07:14:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.33, 3.05, 2.99 [07:19:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.97, 7.70, 7.09 [07:21:53] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.58, 7.79, 7.22 [07:23:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.12, 6.71, 6.15 [07:24:06] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.19, 8.06, 7.38 [07:27:53] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.71, 6.31, 6.12 [07:28:19] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.10, 7.39, 7.33 [07:39:05] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.29, 5.36, 6.47 [08:56:27] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.16, 6.59, 5.71 [08:58:32] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.51, 6.26, 5.71 [10:36:58] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.19, 7.08, 5.48 [10:39:13] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.88, 7.38, 5.81 [10:42:46] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.06, 6.94, 5.40 [10:44:52] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.75, 6.10, 5.29 [10:47:43] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.19, 6.30, 6.09 [10:54:23] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.90, 6.35, 6.04 [10:56:27] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.02, 6.19, 6.03 [11:33:36] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.68, 6.57, 5.69 [11:33:41] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.35, 6.87, 6.18 [11:35:00] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.37, 7.26, 6.15 [11:35:48] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.64, 8.25, 6.44 [11:35:54] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.57, 8.11, 6.72 [11:38:01] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.59, 7.54, 6.44 [11:38:07] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.35, 7.89, 6.87 [11:39:19] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.64, 6.98, 6.36 [11:42:05] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.56, 6.14, 6.14 [11:42:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.11, 6.55, 6.57 [11:43:31] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.73, 5.64, 5.99 [12:24:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.73, 3.34, 2.84 [12:24:35] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.49, 7.06, 6.40 [12:26:14] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.85, 3.14, 2.83 [12:26:45] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.63, 8.13, 6.88 [12:27:06] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.51, 7.27, 6.23 [12:31:11] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.01, 7.59, 7.01 [12:31:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.12, 7.31, 6.60 [12:34:18] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.93, 6.79, 6.26 [12:35:34] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.43, 7.70, 7.15 [12:36:34] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.54, 6.53, 6.26 [12:37:47] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.49, 6.99, 6.97 [12:38:01] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.75, 6.79, 6.65 [12:41:59] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.05, 6.15, 6.63 [12:50:51] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.96, 6.26, 6.33 [12:55:11] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.10, 6.46, 6.43 [13:03:32] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.72, 6.63, 6.10 [13:05:46] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.44, 6.56, 6.15 [13:20:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 12.39, 8.80, 7.31 [13:20:54] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.17, 8.04, 6.82 [13:24:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.89, 7.24, 6.72 [13:26:44] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.43, 8.35, 7.19 [13:31:53] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.96, 7.70, 7.52 [13:36:20] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.08, 8.41, 7.79 [13:38:28] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.65, 7.65, 7.59 [13:40:41] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.49, 8.32, 7.83 [13:53:07] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.28, 4.35, 3.60 [14:00:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.88, 3.72 [14:02:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.66, 4.22, 3.87 [14:05:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.88, 3.80 [14:10:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2001:41d0:801:2000::58af/cpweb [14:15:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::58af/cpweb [14:17:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.73, 2.98, 3.30 [14:18:17] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:18:18] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:30:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.51, 3.94, 3.63 [14:32:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.78, 3.52, 3.52 [14:34:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.29, 3.02, 3.33 [14:41:05] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.79, 6.16, 7.82 [14:41:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.72, 3.54 [14:42:50] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.11, 6.83, 7.88 [14:45:02] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.73, 8.19, 8.28 [14:46:19] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.79, 3.19, 3.36 [14:47:12] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.10, 6.99, 7.83 [14:51:33] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.62, 6.70, 7.93 [14:58:05] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.34, 7.26, 7.71 [15:00:14] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.03, 7.11, 7.23 [15:00:18] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.36, 6.70, 7.44 [15:01:13] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.32, 3.79, 3.41 [15:02:24] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.64, 7.08, 7.18 [15:02:51] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.54, 5.91, 6.72 [15:04:40] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.74, 7.41, 7.27 [15:05:48] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.48, 3.38, 3.37 [15:06:54] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.13, 7.83, 7.48 [15:06:57] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.54, 7.85, 7.62 [15:07:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.99, 6.98, 6.99 [15:09:07] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 7.65, 8.04, 7.62 [15:09:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.97, 7.98, 7.37 [15:11:51] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.61, 7.71, 7.37 [15:13:26] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.77, 7.32, 7.45 [15:14:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.59, 7.81, 7.43 [15:14:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.85, 3.70, 3.46 [15:15:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.69, 7.74, 7.79 [15:16:12] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.89, 7.64, 7.43 [15:17:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.51, 3.42 [15:20:32] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.77, 7.52, 7.39 [15:22:46] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.74, 7.12, 7.29 [15:24:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.90, 7.42, 7.21 [15:24:43] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 13.74, 9.29, 8.11 [15:25:10] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 14.26, 9.95, 8.30 [15:28:10] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.117 second response time [15:28:13] PROBLEM - cp3 Stunnel Http for mw10 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.780 second response time [15:28:55] PROBLEM - cp3 Stunnel Http for mw13 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:29:01] PROBLEM - cp12 Stunnel Http for mw10 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:29:02] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2001:41d0:801:2000::58af/cpweb, 2001:41d0:800:170b::5/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [15:30:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [15:30:59] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20010 bytes in 5.223 second response time [15:31:01] RECOVERY - cp3 Stunnel Http for mw10 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15708 bytes in 4.540 second response time [15:31:39] RECOVERY - cp3 Stunnel Http for mw13 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15716 bytes in 1.286 second response time [15:31:40] RECOVERY - cp12 Stunnel Http for mw10 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15708 bytes in 0.903 second response time [15:31:41] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:32:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.39, 3.40 [15:32:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:34:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.08, 3.35, 3.40 [15:49:39] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 3.41, 3.33 [15:51:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.47, 3.36 [15:55:56] Reception123: did you see any reason for the session issues [15:56:04] You should be able to filter in graylog [16:01:04] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.84, 3.19, 3.28 [16:05:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.06, 3.56, 3.48 [16:14:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.14, 2.91, 3.23 [16:33:50] while mw8 was pooled, I was unable to get login. As soon as I did, I got kicked off [16:34:06] A bit busy today so I'll try to look a bit later but if not will do tomorrow. [16:38:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.34, 3.70, 3.23 [16:39:21] Reception123: ok, am in a tech event now [16:43:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.44, 3.23 [16:45:27] Ooh my q got asked [16:45:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.28, 3.06, 3.12 [17:04:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 3.82, 3.49 [17:06:30] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.25, 3.49, 3.41 [17:08:46] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.39, 3.07, 3.26 [17:10:50] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.90, 6.66, 7.71 [17:13:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.23, 6.50, 7.73 [17:15:34] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.65, 6.09, 7.59 [17:17:13] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.70, 5.07, 6.72 [17:19:51] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.00, 4.68, 6.49 [17:21:54] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.44, 4.96, 6.55 [17:35:32] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JuAYV [17:35:34] [02miraheze/mediawiki] 07paladox 03d5db44f - Remove deprecated CURLPIPE_HTTP1 [17:35:35] [02mediawiki] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbL5b [17:35:37] [02mediawiki] 07paladox opened pull request 03#2992: Remove deprecated CURLPIPE_HTTP1 - 13https://git.io/JuAYo [17:36:09] paladox: please don't forget .37 [17:37:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.42, 3.18 [17:37:56] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JuAYQ [17:37:58] [02miraheze/mediawiki] 07paladox 0340608a0 - Update MultiHttpClient.php [17:37:59] [02mediawiki] 07paladox synchronize pull request 03#2992: Remove deprecated CURLPIPE_HTTP1 - 13https://git.io/JuAYo [17:38:04] RhinosF1: i did the commit for the mw team to decide [17:38:12] so will leave it to you to decide how to proceed [17:38:38] paladox: I trust you. If it works I can merge in ~30 minutes. [17:38:50] https://graylog.miraheze.org/search?q=mediawiki_host%3Amw8&rangetype=relative&from=86400 lots of messages [17:38:51] I can simply do a patch file for 1.37 [17:38:52] ok [17:38:55] Yeah I saw earlier [17:39:09] [02mediawiki] 07paladox closed pull request 03#2992: Remove deprecated CURLPIPE_HTTP1 - 13https://git.io/JuAYo [17:39:11] [02miraheze/mediawiki] 07paladox pushed 033 commits to 03REL1_36 [+0/-0/±3] 13https://git.io/JuAOv [17:39:12] [02miraheze/mediawiki] 07paladox 03d3a5b2b - Merge pull request #2992 from miraheze/paladox-patch-1 [17:39:14] [02mediawiki] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbL5b [17:39:15] [02miraheze/mediawiki] 07paladox deleted branch 03paladox-patch-1 [17:39:29] I'm on a call while at least 7 [17:39:50] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.04, 3.25, 3.14 [17:46:51] !log temporarily pool mw8 [17:47:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:47:51] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy [17:48:19] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 9 backends are healthy [17:48:24] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 9 backends are healthy [17:48:35] RECOVERY - cp14 Varnish Backends on cp14 is OK: All 9 backends are healthy [17:56:06] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JuAsD [17:56:07] [02miraheze/mediawiki] 07paladox 03a06639b - Update DataTransfer [18:05:49] [02miraheze/mediawiki] 07RhinosF1 pushed 032 commits to 03REL1_37 [+1/-0/±4] 13https://git.io/JuAn6 [18:05:51] [02miraheze/mediawiki] 07RhinosF1 0307b368c - Update MultiHttpClient.php [18:05:52] [02miraheze/mediawiki] 07RhinosF1 03909399b - Merge branch 'REL1_37' of https://github.com/miraheze/mediawiki into REL1_37 [18:05:53] [url] GitHub - miraheze/mediawiki: The collaborative editing software that runs Wikipedia. | github.com [18:06:03] paladox: ^ [18:06:18] thanks [18:07:58] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [18:13:53] majavah: you break CA a lot right? [18:14:01] correct [18:14:01] I'm seeing a lot of errors from 7.4 [18:14:08] yes? [18:14:29] majavah: Session "[50]CentralAuthSessionProvider<-:305:Agent Isai>": Metadata merge failed: MediaWiki\Session\MetadataMergeException: Key "CentralAuthSource" changed in /srv/mediawiki/w/includes/session/SessionProvider.php:247 [18:14:36] Key "CentralAuthSource" changed [18:14:38] uhhhh [18:14:48] Failed to load session, unpersisting [18:15:12] Session "" requested with mismatched UserID and UserName cookies. [18:15:22] mediawiki_mismatch_cookie_username [18:15:22] Agent+Isai [18:15:22] mediawiki_mismatch_userid [18:15:22] 155 [18:15:22] mediawiki_mismatch_username [18:15:22] Agent Isai [18:15:25] majavah: ^ [18:15:26] which branch? REL1_36? [18:15:35] majavah: ye [18:15:45] let me get commit id [18:16:09] b4b8fa35 [18:17:01] (that is latest) [18:17:36] ohh fun, this definitely has a ton of changes between REL1_36 and master [18:17:48] can you dump the full stack trace somewhere? [18:18:07] majavah: for which of the 3 errors i gave [18:18:15] all of them [18:20:49] PROBLEM - mw8 Puppet on mw8 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 0 failures [18:21:54] majavah: https://phabricator.miraheze.org/P445 [18:21:55] [url] Login | phabricator.miraheze.org [18:25:45] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+1/-0/±2] 13https://git.io/JuAlr [18:25:47] [02miraheze/mediawiki] 07Universal-Omega 030c335b1 - Install Evelution [18:27:20] oh that reminds me [18:27:34] CosmicAlpha: want to deploy the update for the portableinfobox ext? [18:27:38] (to your fork) [18:27:50]