[00:03:59] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.89, 20.85, 18.37 [00:04:41] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 14.94, 9.67, 7.46 [00:05:46] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 16.14, 10.73, 7.67 [00:05:59] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.44, 24.01, 19.80 [00:12:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.68, 7.86, 7.70 [00:15:30] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.26, 7.57, 7.58 [00:17:57] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.42, 22.04, 21.61 [00:19:58] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 31.14, 25.27, 22.80 [00:20:27] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.67, 8.54, 7.91 [00:21:57] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.23, 22.80, 22.20 [00:22:40] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.72, 1.54, 1.44 [00:24:25] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.18, 7.77, 7.74 [00:24:36] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.01, 1.39, 1.40 [00:28:21] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.29, 8.43, 8.01 [00:29:18] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.17, 7.18, 7.20 [00:31:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.91, 7.09, 7.17 [00:31:58] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.49, 18.27, 20.37 [00:32:18] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.53, 7.65, 7.81 [00:33:11] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.43, 7.53, 7.29 [00:34:18] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.22, 9.75, 8.55 [00:35:59] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.42, 21.10, 21.24 [00:37:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.48, 7.96, 7.58 [00:38:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.70, 7.53, 7.97 [00:39:59] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.06, 17.87, 19.91 [00:40:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.85, 8.09, 7.68 [00:42:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.16, 8.15, 8.04 [00:44:54] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.93, 7.53, 7.58 [00:46:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.98, 7.32, 7.77 [00:50:53] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.56, 7.74, 7.48 [00:52:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.31, 7.60, 7.46 [00:54:47] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.83, 1.60, 1.38 [00:56:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.47, 5.49, 6.68 [00:56:42] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.40, 1.53, 1.38 [00:59:57] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.45, 2.95 [01:00:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.04, 5.86, 6.67 [01:01:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.96, 3.65, 3.09 [01:04:29] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.20, 1.75, 1.52 [01:05:48] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.02, 3.24, 3.04 [01:06:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.12, 7.30, 6.77 [01:06:24] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.92, 1.46, 1.44 [01:06:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.89, 7.58, 7.10 [01:08:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.66, 6.45, 6.51 [01:08:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.81, 6.72, 6.83 [01:10:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.04, 6.13, 6.60 [01:34:51] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.20, 6.78, 6.03 [01:36:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.09, 6.14, 5.90 [01:36:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.85, 3.36, 3.05 [01:38:52] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.85, 3.00, 2.94 [01:44:18] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 15.27, 10.08, 7.36 [01:44:51] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.90, 8.57, 6.90 [01:45:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.36, 4.23, 3.03 [01:47:54] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.62, 4.01, 3.10 [01:48:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.10, 7.71, 6.92 [01:50:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.30, 8.28, 7.25 [01:52:14] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.29, 1.86, 1.48 [01:54:14] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.72, 1.88, 1.54 [01:56:14] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.17, 2.01, 1.63 [01:58:06] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.75, 7.20, 6.11 [01:58:14] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.43, 1.76, 1.58 [02:00:14] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.45, 1.57, 1.53 [02:06:06] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.85, 6.69, 6.38 [02:12:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.02, 7.01, 7.85 [02:18:53] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.63, 7.84, 7.80 [02:20:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.27, 7.31, 7.62 [02:21:26] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 42% [02:23:27] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 2% [02:24:13] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.03, 1.75, 1.47 [02:26:13] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.14, 1.54, 1.43 [02:28:51] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.09, 7.50, 7.49 [02:29:03] PROBLEM - wikipariksha.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikipariksha.com could not be found [02:29:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.32, 3.05 [02:30:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.86, 7.62, 7.57 [02:33:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.77, 2.81, 2.92 [02:34:55] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.78, 8.56, 7.88 [02:35:56] RECOVERY - wikipariksha.com - reverse DNS on sslhost is OK: SSL OK - wikipariksha.com reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [02:36:19] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.93, 6.86, 6.38 [02:38:19] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.28, 7.08, 6.50 [02:40:18] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.73, 6.69, 6.44 [02:43:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.41, 3.10 [02:44:16] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.26, 7.29, 6.72 [02:46:15] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.03, 6.97, 6.68 [02:46:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.04, 7.42, 7.90 [02:49:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRKdu [02:49:30] [02miraheze/puppet] 07paladox 0323c0871 - Don't use nice for puppet [02:49:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.88, 3.34, 3.19 [02:50:16] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.17, 6.28, 6.49 [02:51:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRKdh [02:51:35] [02miraheze/puppet] 07paladox 03dbe96ca - mediawiki::jobqueue::runner: Fix nice priority so it'll be 20 rather than -19 [02:52:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.65, 7.64, 7.77 [02:52:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRKFY [02:52:33] [02miraheze/puppet] 07paladox 032b68757 - Don't use nice in puppet-run [02:54:15] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.89, 1.44, 1.21 [02:56:14] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.87, 1.20, 1.15 [02:57:42] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.63, 3.38 [02:59:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.65, 3.42 [03:01:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.38, 3.31, 3.33 [03:01:46] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.37, 7.69, 6.68 [03:02:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRKNC [03:02:43] [02miraheze/puppet] 07paladox 0375a2f66 - jobrunner: Re add nice [03:03:16] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.45, 7.87, 7.09 [03:05:45] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.70, 7.49, 6.89 [03:07:44] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.97, 8.97, 7.49 [03:17:44] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.37, 7.78, 7.70 [03:19:11] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.76, 7.67, 7.94 [03:19:44] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.77, 7.97, 7.76 [03:21:11] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.61, 8.59, 8.25 [03:21:43] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.33, 7.97, 7.80 [03:25:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.61, 7.51, 7.89 [03:25:20] PROBLEM - diamowiki.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:32:18] RECOVERY - diamowiki.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'diamowiki.ga' will expire on Sat 18 Sep 2021 14:34:47 GMT +0000. [03:33:07] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.07, 7.24, 7.47 [03:33:45] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.64, 7.43, 7.30 [03:35:44] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.73, 6.65, 7.03 [03:37:06] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.66, 7.39, 7.51 [03:39:44] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.80, 6.18, 6.74 [03:45:45] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.14, 6.48, 6.64 [03:47:43] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.65, 6.64, 6.69 [03:55:06] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.72, 7.61, 7.26 [03:56:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.15, 7.39, 6.87 [03:57:06] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.74, 7.62, 7.31 [04:00:25] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.36, 7.16, 6.90 [04:02:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.48, 6.76, 6.77 [04:08:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.06, 6.30, 7.63 [04:09:06] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.44, 5.90, 6.63 [04:12:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.96, 6.43, 7.69 [04:14:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JR6qH [04:14:46] [02miraheze/puppet] 07paladox 03b4b9729 - jobrunner: Re add nice [04:20:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.44, 5.74, 6.75 [04:22:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.44, 7.47, 7.41 [04:24:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.54, 7.33, 7.36 [04:30:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.49, 6.13, 6.75 [04:32:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.22, 6.54, 6.68 [04:34:51] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.25, 8.61, 7.45 [04:36:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.86, 6.91, 6.91 [04:38:17] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.55, 6.03, 6.58 [04:38:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.59, 7.55, 7.34 [04:46:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.05, 7.07, 7.05 [04:48:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.90, 7.22, 7.10 [04:50:51] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.09, 7.92, 7.40 [04:51:07] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.84, 7.73, 6.94 [04:52:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.49, 7.38, 7.26 [04:53:05] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.16, 7.27, 6.89 [04:59:03] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.76, 6.78, 6.75 [05:00:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.31, 5.63, 6.50 [05:04:51] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.18, 7.28, 6.93 [05:04:54] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.73, 8.62, 7.41 [05:06:54] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.82, 7.50, 7.15 [05:06:54] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.53, 6.42, 6.67 [05:12:49] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.98, 6.36, 6.72 [05:22:40] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.90, 6.91, 6.85 [05:24:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.82, 6.14, 6.57 [05:36:30] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.94, 7.23, 6.79 [05:42:23] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.66, 6.60, 6.69 [05:54:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.22, 6.91, 6.76 [05:56:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.72, 6.12, 6.48 [06:05:29] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.56, 7.60, 6.77 [06:06:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.24, 8.61, 7.28 [06:07:25] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.93, 7.41, 6.81 [06:08:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.72, 7.97, 7.20 [06:08:53] RECOVERY - dbbackup1 APT on dbbackup1 is OK: APT OK: 25 packages available for upgrade (0 critical updates). [06:09:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.65, 8.15, 7.17 [06:09:31] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu] [06:10:18] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.51, 8.25, 7.38 [06:11:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.50, 7.76, 7.14 [06:13:21] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.05, 8.32, 7.40 [06:15:17] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.16, 7.90, 7.36 [06:19:15] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.31, 8.01, 7.51 [06:21:12] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.50, 7.95, 7.56 [06:24:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.09, 7.50, 7.55 [06:27:02] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.69, 5.83, 6.73 [06:28:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.78, 5.55, 6.74 [06:35:32] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:52:19] !log [reception@jobrunner4] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=polandballes /home/reception123/espolandball_pages_full.xml --username-prefix wikia:polandball --report 1 (END - exit=65280) [06:52:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:52:34] !log [reception@jobrunner4] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=polandballeswiki /home/reception123/espolandball_pages_full.xml --username-prefix wikia:polandball --report 1 (END - exit=256) [06:52:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:53:04] ^ fixed [07:01:56] [02puppet] 07Reception123 closed pull request 03#1865: Add gitinfo argument to deploy-mediawiki - 13https://git.io/JRo0W [07:01:58] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JR65I [07:01:59] [02miraheze/puppet] 07Universal-Omega 032b89d0f - Add gitinfo argument to deploy-mediawiki (#1865) [07:06:30] !log [reception@jobrunner3] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=zhpolandballwiki /home/reception/zhpolandball_pages_full.xml --username-prefix wikia:bolanqiu --report 1 (END - exit=2) [07:06:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:12:23] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 29.14, 21.74, 13.80 [07:12:30] !log [reception@jobrunner4] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=polandballwiki /home/reception/polandball_pages_full.xml --username-prefix wikia:polandball --report 1 [07:12:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:14:22] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 10.93, 17.17, 13.09 [07:35:24] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.85, 7.47, 6.22 [07:35:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.36, 7.16, 5.95 [07:37:22] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.25, 6.93, 6.17 [07:37:50] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.13, 6.28, 5.76 [07:39:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.77, 6.09, 5.95 [07:45:15] !log [reception@jobrunner3] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=402611wiki /home/reception/ptpolandball_pages_full.xml --username-prefix wikia:polandball [07:45:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:05:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.44, 7.20, 6.26 [08:08:53] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRitP [08:08:54] [02miraheze/mw-config] 07RhinosF1 03960f6c7 - security@ hasn't created tasks in a while [08:09:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.93, 6.38, 6.15 [08:09:56] miraheze/mw-config - RhinosF1 the build passed. [08:27:35] PROBLEM - ns1 APT on ns1 is CRITICAL: connect to address 192.184.82.120 port 5666: Connection refusedconnect to host 192.184.82.120 port 5666: Connection refused [08:28:23] PROBLEM - ns1 Current Load on ns1 is CRITICAL: connect to address 192.184.82.120 port 5666: Connection refusedconnect to host 192.184.82.120 port 5666: Connection refused [08:28:46] PROBLEM - ns1 NTP time on ns1 is CRITICAL: connect to address 192.184.82.120 port 5666: Connection refusedconnect to host 192.184.82.120 port 5666: Connection refused [08:28:52] PROBLEM - ns1 Disk Space on ns1 is CRITICAL: connect to address 192.184.82.120 port 5666: Connection refusedconnect to host 192.184.82.120 port 5666: Connection refused [08:28:59] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: connect to address 192.184.82.120 port 5666: Connection refusedconnect to host 192.184.82.120 port 5666: Connection refused [08:30:37] * RhinosF1 here [08:30:56] Reception123: can we reboot ns1 [08:31:03] It appears to have got more broken [08:31:25] There's a task for adding more memory waiting on your +1 [08:31:53] https://phabricator.miraheze.org/T7772 [08:31:54] [url] ⚓ T7772 [Existing] Server Resource Request for ns1 | phabricator.miraheze.org [08:33:09] !log [@test3] starting deploy of {'config': True} to skip [08:33:10] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS [08:33:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:33:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:33:41] And \o/ [08:33:54] RhinosF1: worked! But still it should log it differently I think so it isn't just `@test3` if possible. But not a requirement. [08:34:13] Universal_Omega: that comes from logsalmsg [08:34:36] I know if it can't be corrected not a big deal then. [08:37:31] [02puppet] 07RhinosF1 opened pull request 03#1866: logsalmsg: keep $USER is $SUDO_USER blank - 13https://git.io/JRiZB [08:38:04] Universal_Omega: ^ might work to show www-data@test3 but I want JohnLewis to review [08:44:45] [02puppet] 07Universal-Omega reviewed pull request 03#1866 commit - 13https://git.io/JRicT [08:45:33] [02puppet] 07Universal-Omega reviewed pull request 03#1866 commit - 13https://git.io/JRic8 [08:45:43] RhinosF1: ^ [08:47:13] Universal_Omega: yeah actually that may be a point [08:47:31] [02puppet] 07RhinosF1 closed pull request 03#1866: logsalmsg: keep $USER is $SUDO_USER blank - 13https://git.io/JRiZB [08:48:16] In theory there shouldn't be anything special about www-data bar it's uid [08:49:48] RhinosF1: wouldn't adding `|| [ "$USER" = 'www-data' ]` to the existing statement work? [08:50:06] Whoops not with quotation around $USER though. [08:50:14] Universal_Omega: but what would $SUDO_USER evaluate too [08:50:20] It's running automated [08:50:29] It's puppet that triggered that [08:51:31] RhinosF1: oh yeah, true. [08:54:12] RhinosF1: What about `if [ !$ACTOR ] then ACTOR='www-data' fi` that would probably solve it but there may be other circumstances then www-data guess. [08:55:15] I mean it would work but I'd like JohnLewis feedback on if there's a more sensible way [08:55:29] Yeah sounds good. [08:55:43] Automated logging might not happen forever if it's successful to be honest or might be tweaked [08:55:59] It's more at the moment while we find ways to break it [08:56:16] Sounds good. Anyways night. [08:58:17] RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 1.31, 1.17, 0.61 [08:58:47] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset -0.0001218914986 secs [08:58:50] RECOVERY - ns1 Disk Space on ns1 is OK: DISK OK - free space: / 11245 MB (87% inode=91%); [08:58:56] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:59:08] Ns1 has woke back up [08:59:09] RECOVERY - ns1 APT on ns1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [08:59:25] Reception123: please approve the resource req for it [09:51:58] Will look into that [10:05:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.66, 7.29, 5.64 [10:07:25] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.75, 6.51, 5.54 [10:51:46] PROBLEM - jobchron1 APT on jobchron1 is CRITICAL: connect to address 51.195.236.215 port 5666: Connection refusedconnect to host 51.195.236.215 port 5666: Connection refused [10:51:47] PROBLEM - jobchron1 NTP time on jobchron1 is CRITICAL: connect to address 51.195.236.215 port 5666: Connection refusedconnect to host 51.195.236.215 port 5666: Connection refused [10:51:47] PROBLEM - jobchron1 PowerDNS Recursor on jobchron1 is CRITICAL: connect to address 51.195.236.215 port 5666: Connection refusedconnect to host 51.195.236.215 port 5666: Connection refused [10:51:53] PROBLEM - jobchron1 Disk Space on jobchron1 is CRITICAL: connect to address 51.195.236.215 port 5666: Connection refusedconnect to host 51.195.236.215 port 5666: Connection refused [10:52:34] PROBLEM - jobchron1 Puppet on jobchron1 is CRITICAL: connect to address 51.195.236.215 port 5666: Connection refusedconnect to host 51.195.236.215 port 5666: Connection refused [10:52:40] PROBLEM - jobchron1 Current Load on jobchron1 is CRITICAL: connect to address 51.195.236.215 port 5666: Connection refusedconnect to host 51.195.236.215 port 5666: Connection refused [11:05:03] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.91, 20.79, 17.83 [11:06:21] PROBLEM - jobchron1 Current Load on jobchron1 is WARNING: WARNING - load average: 1.95, 1.35, 0.74 [11:06:34] PROBLEM - jobchron1 Puppet on jobchron1 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file [11:07:02] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.15, 19.33, 17.72 [11:07:43] RECOVERY - jobchron1 PowerDNS Recursor on jobchron1 is OK: DNS OK: 0.199 seconds response time. miraheze.org returns 2001:41d0:801:2000::58af,51.38.69.175 [11:07:43] RECOVERY - jobchron1 NTP time on jobchron1 is OK: NTP OK: Offset -0.01057216525 secs [11:07:49] RECOVERY - jobchron1 Disk Space on jobchron1 is OK: DISK OK - free space: / 5930 MB (77% inode=87%); [11:08:18] RECOVERY - jobchron1 Current Load on jobchron1 is OK: OK - load average: 0.87, 1.17, 0.75 [11:18:22] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JRiAy [11:18:24] [02miraheze/ssl] 07Reception123 03bbec42e - add duepedia.uk.to cert [11:21:25] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.71, 6.60, 5.70 [11:23:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.41, 6.31, 5.68 [11:27:11] [02puppet] 07RhinosF1 opened pull request 03#1867: mwdeploy: record time taken - 13https://git.io/JRihf [11:28:40] Reception123: enhancement ^ if someone wants to test it [11:29:12] RhinosF1: ah is that the thing I wanted? [11:29:35] Reception123: no that tells you how long deploys take [11:29:41] I'll do long running later [11:29:52] oh, let me look but that's interesting [11:30:55] [02puppet] 07Reception123 closed pull request 03#1867: mwdeploy: record time taken - 13https://git.io/JRihf [11:30:56] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRihy [11:30:58] [02miraheze/puppet] 07RhinosF1 030aa2418 - mwdeploy: record time taken (#1867) [11:31:25] Easiest to test would be pull and just run on test3 [11:31:56] yeah, what should I run as a command? [11:32:09] deploy-mediawiki --sync=skip [11:32:16] And then again with --no-log [11:32:43] It'll very likely be 0s [11:32:48] Reception123: ^ [11:33:56] ok, let's try it [11:35:56] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.49, 7.60, 6.46 [11:36:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.80, 7.59, 6.23 [11:37:25] !log [reception@test3] starting deploy of {} to skip [11:37:26] !log [reception@test3] finished deploy of {} to skip - SUCCESS in 0s [11:37:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:37:30] Yey! [11:37:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:37:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.11, 7.35, 6.50 [11:38:29] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.37, 6.48, 5.99 [11:39:47] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.34, 6.68, 6.36 [11:40:24] !log renamed mywikinamewiki to ukrainianlatinwiki [11:40:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:56:09] [02miraheze/puppet] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JRPU2 [11:56:11] [02miraheze/puppet] 07JohnFLewis 03fcbc9c3 - add jobchron1 as a redis host [11:56:12] [02miraheze/puppet] 07JohnFLewis 0308f9bdf - Merge branch 'master' of github.com:/miraheze/puppet [12:09:42] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/JRPIR [12:09:44] [02miraheze/puppet] 07JohnFLewis 03a9b4e4b - make jobchron independent ofjobrunner [12:12:08] PROBLEM - jobrunner3 Puppet on jobrunner3 is WARNING: WARNING: Puppet is currently disabled, message: John, last run 7 minutes ago with 0 failures [12:15:28] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRPts [12:15:29] [02miraheze/puppet] 07JohnFLewis 0376f4277 - jobrunner config file needed as well [12:22:18] PROBLEM - mwtask1 Puppet on mwtask1 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file [12:22:39] PROBLEM - mwtask1 APT on mwtask1 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (22 critical updates). [12:24:04] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JRPmH [12:24:05] [02miraheze/puppet] 07JohnFLewis 036e587c9 - jobchron1 is now jobqueue master [12:24:55] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRPYU [12:24:57] [02miraheze/mw-config] 07JohnFLewis 0389c496a - jobchron1 is now jobqueue master [12:26:01] miraheze/mw-config - JohnFLewis the build passed. [12:31:22] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.19, 6.84, 6.30 [12:33:18] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.92, 6.50, 6.24 [12:34:07] !log [@test3] starting deploy of {'config': True} to skip [12:34:08] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [12:34:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:34:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:13] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.72, 7.44, 6.74 [12:43:02] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.78, 6.31, 6.47 [12:45:00] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JRPZS [12:45:01] [02miraheze/puppet] 07JohnFLewis 03a0b56c8 - remove jobqueue stuff from jobrunner3 [12:48:53] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.64, 6.95, 6.57 [12:50:48] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.98, 19.60, 17.60 [12:52:05] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.89, 7.02, 6.26 [12:52:46] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.08, 7.70, 7.02 [12:53:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.04, 7.07, 6.15 [12:55:03] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JRPC3 [12:55:05] [02miraheze/puppet] 07JohnFLewis 03bac7720 - deploy mwtask1 [12:56:00] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.10, 6.64, 6.29 [12:56:42] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.82, 6.58, 6.70 [12:56:50] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.40, 18.69, 17.88 [12:57:32] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.69, 7.14, 6.42 [12:59:27] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.05, 6.75, 6.36 [13:04:31] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.52, 7.14, 6.74 [13:05:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.93, 7.01, 6.60 [13:06:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.94, 6.78, 6.65 [13:07:45] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.04, 6.37, 6.42 [13:08:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.96, 6.07, 6.40 [13:22:04] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.14, 7.18, 6.46 [13:22:19] PROBLEM - mwtask1 JobRunner Service on mwtask1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [13:22:29] PROBLEM - mwtask1 Check Gluster Clients on mwtask1 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [13:22:38] PROBLEM - mwtask1 MediaWiki Rendering on mwtask1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:22:49] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.36, 7.37, 6.76 [13:23:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.09, 7.42, 6.89 [13:23:23] PROBLEM - mwtask1 HTTPS on mwtask1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:30] JohnLewis: that you? [13:24:19] RECOVERY - mwtask1 JobRunner Service on mwtask1 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [13:24:39] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.37, 7.16, 6.11 [13:24:46] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.73, 7.05, 6.71 [13:24:55] mwtask1 is just being left to puppet now [13:24:58] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.73, 7.65, 7.04 [13:25:20] Ok [13:25:22] RECOVERY - mwtask1 HTTPS on mwtask1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 565 bytes in 0.030 second response time [13:25:30] There it goes [13:26:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.09, 6.71, 6.07 [13:26:43] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.19, 6.58, 6.60 [13:26:53] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.84, 9.20, 7.65 [13:30:45] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.55, 6.86, 6.74 [13:32:44] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.28, 7.79, 7.09 [13:33:37] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.99, 24.11, 20.11 [13:34:26] RECOVERY - mwtask1 Check Gluster Clients on mwtask1 is OK: PROCS OK: 2 processes with args '/usr/sbin/glusterfs' [13:34:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.18, 7.72, 7.16 [13:35:29] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.43, 7.25, 6.51 [13:37:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.70, 23.29, 20.75 [13:38:26] PROBLEM - mwtask1 Check Gluster Clients on mwtask1 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [13:39:17] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.18, 7.50, 6.75 [13:41:11] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.70, 6.94, 6.64 [13:41:22] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.51, 19.86, 19.87 [13:42:39] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.57, 6.31, 6.72 [13:44:58] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.41, 6.73, 6.62 [14:01:57] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.21, 20.28, 19.74 [14:03:56] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.04, 19.83, 19.64 [14:16:01] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.71, 6.76, 6.49 [14:19:59] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.77, 6.42, 6.43 [14:20:26] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.16, 4.62, 3.28 [14:20:46] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.20, 25.16, 22.01 [14:22:23] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.71, 4.62, 3.44 [14:22:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.61, 23.42, 21.77 [14:28:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.01, 18.05, 19.95 [14:34:24] RECOVERY - mwtask1 Check Gluster Clients on mwtask1 is OK: PROCS OK: 2 processes with args '/usr/sbin/glusterfs' [14:38:24] PROBLEM - mwtask1 Check Gluster Clients on mwtask1 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [14:46:20] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.75, 6.59, 8.00 [14:50:20] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.32, 7.65, 8.02 [14:56:20] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.78, 7.18, 7.78 [14:57:34] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.90, 7.08, 7.87 [15:02:21] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.25, 7.45, 7.66 [15:03:33] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.43, 7.05, 7.53 [15:07:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.38, 7.64, 7.71 [15:11:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.88, 6.66, 6.13 [15:11:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.09, 7.55, 7.63 [15:12:19] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.75, 7.35, 7.96 [15:14:21] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.10, 7.85, 8.04 [15:16:20] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.85, 7.63, 7.92 [15:18:19] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.51, 7.66, 7.88 [15:21:08] PROBLEM - jobchron1 JobChron Service on jobchron1 is CRITICAL: PROCS CRITICAL: 2 processes with args 'redisJobChronService' [15:22:59] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.79, 6.68, 6.63 [15:23:10] RECOVERY - jobchron1 JobChron Service on jobchron1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [15:26:50] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.29, 6.92, 6.73 [15:28:47] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.91, 6.46, 6.58 [15:31:19] PROBLEM - mail2 APT on mail2 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:34:39] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.50, 7.51, 6.92 [15:36:39] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.16, 6.88, 6.76 [15:38:37] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.35, 5.56, 6.29 [15:42:19] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.15, 7.18, 7.90 [15:47:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.33, 7.39, 7.97 [15:48:19] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.63, 7.74, 7.90 [15:49:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.41, 8.93, 8.50 [15:54:19] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.43, 7.30, 7.77 [16:00:21] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.68, 8.35, 7.95 [16:03:05] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.24, 6.96, 6.44 [16:04:28] RECOVERY - mwtask1 Check Gluster Clients on mwtask1 is OK: PROCS OK: 2 processes with args '/usr/sbin/glusterfs' [16:05:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.28, 6.78, 6.44 [16:07:01] JohnLewis: should mediawiki admins ot get jobchron access [16:07:53] Reception123: would you be able to do https://phabricator.miraheze.org/T7772 please? :) [16:07:54] [url] ⚓ T7772 [Existing] Server Resource Request for ns1 | phabricator.miraheze.org [16:08:10] Oh wait [16:08:18] They approved [16:08:22] That just requires doing it on the cloud [16:08:25] PROBLEM - mwtask1 Check Gluster Clients on mwtask1 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [16:08:36] Yeh but neither me or John have an account on ramnode [16:09:22] Oh [16:09:58] That should be setup [16:12:43] Yeh [16:12:50] RhinosF1: mediawiki-admins don't have any redis access or need for such access to my knowledge [16:13:18] JohnLewis: ok [16:13:33] I've given JohnLewis access to RN so he can handle. paladox I should get you an account too now that you're here :) [16:13:38] If a need is identified, it would need to go through the access request system [16:15:04] JohnLewis: did you also see the improving @test in logsalmsg conversation [16:16:09] Briefly but I've been busy so not had the chance to look [16:17:23] Not a big deal right now anyways RhinosF1, bigger issues then that I think. It's really not a big deal if it's never done anyways. [16:17:52] Universal_Omega: no I just wanted to know if he'd seen it [16:18:06] I just wanted to do that because it doesn't look right, right now but don't mind if it's not done. [16:18:06] It probably won't be logged forever anyway [16:18:10] I understand. [16:18:11] It's only so I can see it works [16:18:19] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.40, 7.31, 7.90 [16:18:33] Humans should be but automated less so [16:20:22] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.05, 8.31, 8.22 [16:21:26] RhinosF1: also last time when I used `--folders=w/extensions/ManageWiki` with deploy-mediawiki it didn't seem to update ManageWiki from my test3 changes at all. Not sure if I did something wrong though. That was a couple days ago but I just remembered. I had to just do it normally with `--world` and not `--folders` at all. [16:21:36] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.05, 6.86, 6.54 [16:22:03] Universal_Omega: weird [16:22:06] It should do [16:22:13] It worked for me & OAuth [16:22:21] Any output? [16:22:43] it said it succeeded but the changes appeared to not be taking effect. [16:23:47] Strange [16:23:53] It's litterally just cp [16:24:22] diff -qr would have told you if it copied right [16:25:26] Yeah I didn't think about that but it didn't seem to work, I even made a change that would've fataled it (and it did when using `--world`) just to see. [16:30:24] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.90, 6.96, 6.33 [16:32:24] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.13, 6.65, 6.28 [16:33:25] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.77, 8.39, 7.41 [16:34:27] RECOVERY - mwtask1 Check Gluster Clients on mwtask1 is OK: PROCS OK: 2 processes with args '/usr/sbin/glusterfs' [16:35:23] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.17, 7.73, 7.30 [16:36:25] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.02, 7.55, 6.76 [16:38:24] PROBLEM - mwtask1 Check Gluster Clients on mwtask1 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [16:38:24] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.81, 7.17, 6.71 [16:39:20] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.21, 7.89, 7.46 [16:41:17] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.59, 7.85, 7.50 [16:44:26] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.89, 6.43, 6.60 [16:47:10] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.99, 7.50, 7.40 [16:49:07] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.35, 6.59, 7.08 [16:53:06] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.44, 6.93, 7.04 [16:55:03] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.67, 6.65, 6.92 [16:57:01] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.14, 6.08, 6.67 [17:04:25] RECOVERY - mwtask1 Check Gluster Clients on mwtask1 is OK: PROCS OK: 2 processes with args '/usr/sbin/glusterfs' [17:04:58] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.31, 6.83, 6.81 [17:05:56] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.09, 20.59, 19.80 [17:06:25] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.75, 7.09, 6.57 [17:06:55] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.96, 6.09, 6.54 [17:08:24] PROBLEM - mwtask1 Check Gluster Clients on mwtask1 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [17:09:57] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.09, 17.72, 18.86 [17:10:25] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.93, 6.64, 6.54 [17:24:37] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.63, 7.78, 6.87 [17:26:24] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.82, 7.15, 6.76 [17:26:35] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.91, 7.05, 6.72 [17:28:25] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.15, 6.78, 6.67 [17:28:34] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.54, 6.20, 6.45 [17:33:50] !log setup ns1 in the cloud [17:33:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:34:21] !log Removed the old ns1 from puppet in order to switch it over to the new server (doesn't affect any traffic, and puppet was failing to run anyways) [17:34:24] RECOVERY - mwtask1 Check Gluster Clients on mwtask1 is OK: PROCS OK: 2 processes with args '/usr/sbin/glusterfs' [17:34:25] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.72, 7.04, 6.74 [17:34:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:36:24] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.34, 6.38, 6.54 [17:38:24] PROBLEM - mwtask1 Check Gluster Clients on mwtask1 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [17:40:00] JohnLewis: can ^ be fixed or downtime's [17:41:01] you need to add its ip to the firewall list [17:41:26] https://github.com/miraheze/puppet/blob/master/modules/gluster/data/firewall.yaml [17:41:27] [url] puppet/firewall.yaml at master · miraheze/puppet · GitHub | github.com [17:41:28] Can’t that be automated? Why is it manual when everything else is? [17:42:21] it can be automated i guess [17:48:49] I think we're ready to fallover the old ns1 to the new ns1 in the DNS now JohnLewis [17:49:57] Okay [17:50:53] PROBLEM - ping6 on ns1 is CRITICAL: CRITICAL - Destination Unreachable (fe80::f816:3eff:feab:8b9f) [17:51:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2001:41d0:801:2000::58af/cpweb, 2607:5300:205:200::1c30/cpweb [17:52:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JRXzA [17:52:47] [02miraheze/puppet] 07paladox 03a6f37bb - Switch over old ns1 ip to the new ns1 ip [17:52:48] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [17:52:50] [02puppet] 07paladox opened pull request 03#1868: Switch over old ns1 ip to the new ns1 ip - 13https://git.io/JRXzx [17:53:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JRXgv [17:53:22] [02miraheze/puppet] 07paladox 030d4c7fa - Update recursor.conf [17:53:23] [02puppet] 07paladox synchronize pull request 03#1868: Switch over old ns1 ip to the new ns1 ip - 13https://git.io/JRXzx [17:53:50] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JRXgI [17:53:52] [02miraheze/dns] 07paladox 03d453b4a - Update ns1 ip [17:53:53] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbQXl [17:53:55] [02dns] 07paladox opened pull request 03#212: Update ns1 ip - 13https://git.io/JRXgL [17:54:15] JohnLewis: does the ip have to be updated in miraheze domain provider? (for the ns) [17:57:07] paladox: glue records will ye [17:57:16] thanks! [17:57:28] who has access to the provider? [17:57:36] [02miraheze/jobrunner-service] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRXgQ [17:57:37] [02miraheze/jobrunner-service] 07JohnFLewis 033c703de - Remove h-data hash when emptying job queue list [17:57:45] Southparkfan definately does [17:58:29] Does Reception123 ? [17:58:53] Reception123: ^ [18:00:23] JohnLewis: job queue board shows some stuff duplicated [18:00:48] RhinosF1: because jobrunner3 is still feeding data [18:01:52] I unfortunately don't have access to that as of now. [18:02:04] JohnLewis: can we kill it as it isn't actually the master now [18:02:46] It will be when jobrunner3 is decommed. Technically jobrunner3 generated jobs still go to jobrunner3's redis [18:03:01] Ah ok [18:03:33] RECOVERY - mwtask1 Check Gluster Clients on mwtask1 is OK: PROCS OK: 2 processes with args '/usr/sbin/glusterfs' [18:04:26] JohnLewis: i guess we can still switch but keep the old ns1 up until Southparkfan is around? [18:05:08] Yeah, though ns1 old if it isn't updating is useless as it will only display old and outdated info [18:05:10] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.93, 6.77, 6.11 [18:05:18] If ns1 goes down you'll just push stuff to 2 [18:05:59] It's fairly useless old anyway [18:06:03] Oh that is right... i can create a cron. [18:07:07] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.13, 6.37, 6.03 [18:07:21] PROBLEM - mwtask1 Check Gluster Clients on mwtask1 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [18:08:09] 24,54 * * * * root cd /etc/gdnsd && git pull && /usr/sbin/gdnsd checkconf && service gdnsd restart [18:08:33] That'll work [18:08:49] thanks! [18:10:43] hmm /usr/sbin/gdnsd checkconf gets killed [18:12:00] going to switch it over now [18:12:11] [02dns] 07paladox closed pull request 03#212: Update ns1 ip - 13https://git.io/JRXgL [18:12:12] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRXVS [18:12:14] [02miraheze/dns] 07paladox 031400e86 - Update ns1 ip (#212) [18:12:22] [02puppet] 07paladox closed pull request 03#1868: Switch over old ns1 ip to the new ns1 ip - 13https://git.io/JRXzx [18:12:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRXVH [18:12:25] [02miraheze/puppet] 07paladox 03acac37e - Switch over old ns1 ip to the new ns1 ip (#1868) [18:12:27] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [18:12:28] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [18:12:40] paladox: shutting old ns1 down shouldn't cause outage as long as ns2 can cope with load [18:12:57] what about the glue record? [18:13:01] I have Ferran's Miraheze number he used somewhere if it's urgent though [18:13:06] It should fall back to 2 [18:13:12] I'd hope both are listed [18:13:59] I guess it's kinda urgent as the old ns1 is OOM'ing [18:14:15] and the dns won't update because of the OOM and because puppet wouldn't run [18:14:28] See -sec [18:14:48] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.83, 22.97, 19.84 [18:16:48] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.28, 20.16, 19.20 [18:22:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 3.42, 2.78 [18:22:52] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.06, 20.57, 19.78 [18:24:38] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.10, 3.15, 2.75 [18:24:52] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.95, 19.75, 19.58 [18:27:41] [02miraheze/puppet] 07JohnFLewis pushed 032 commits to 03master [+0/-2/±10] 13https://git.io/JRXoM [18:27:43] [02miraheze/puppet] 07JohnFLewis 032669348 - deom jobrunner3 [18:27:44] [02miraheze/puppet] 07JohnFLewis 03f3201b1 - decom jobrunner4 [18:29:56] jobrunner* is no longer known to puppet, but they're currently in use for imports, so they can't be properly decom'd until Reception123 is done [18:30:47] Ack [18:31:03] Can we make sure task has alerts fixed [18:31:07] Rendering was moaning too [18:31:27] Also puppet [18:31:35] tasks has been passed over to MWE now as part of their Infra. It's been added to the firewall above [18:31:45] Okay [18:33:38] One of them is very large so I'll transfer it somewhere else once the others are done [18:34:26] Reception123: when both servers are no longer in use, just re-assign the task to me unless you want to reinstall them as mw12/mw13 [18:34:47] RECOVERY - mwtask1 APT on mwtask1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:34:59] Ok, will do. [18:35:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:35:36] RECOVERY - jobchron1 APT on jobchron1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:35:56] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.04, 21.86, 20.34 [18:36:00] PROBLEM - jobrunner4 php-fpm on jobrunner4 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.2' [18:36:14] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.65, 6.90, 6.50 [18:36:58] RECOVERY - mail2 APT on mail2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:37:55] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.07, 20.00, 19.87 [18:38:13] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.05, 6.48, 6.38 [18:40:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRX6P [18:40:33] [02miraheze/puppet] 07paladox 03f498180 - Missed an ip in recursor.conf [18:42:24] PROBLEM - diamowiki.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:44:33] RECOVERY - jobchron1 Puppet on jobchron1 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [18:46:36] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 90.71 ms [18:47:45] Reception123: around? [18:49:04] Yes [18:49:19] RECOVERY - diamowiki.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'diamowiki.ga' will expire on Sat 18 Sep 2021 14:34:47 GMT +0000. [18:51:08] Reception123: as a non-costing solution to the capacity reduction for GB cp (given there's been numerous issues recently), as cp13 is SSD backed, cloud3 has sufficient spare resources that we could provision a new cp on cloud3 similarly to cp13. Unsure if cloud3 has spare IPs so it would cost about £2 to do it though - just wanted to run it by you first [18:52:10] what about ram ? I thought we're running out of ram on cloud3 [18:54:51] https://phabricator.miraheze.org/P386 suggests we have 8G RAM free [18:54:52] [url] ✎ P386 Resources Table | phabricator.miraheze.org [18:54:56] a cp would only take 2 [18:55:50] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=cloud3.miraheze.org&var-port=9100&viewPanel=78 shows we have 3gb left [18:55:51] [url] Grafana | grafana.miraheze.org [18:56:48] CPU is below 50% and load is minimal though, so it is under resourced [18:56:54] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.88, 8.77, 7.41 [18:57:16] yeh [18:57:17] cp13 currently is only using 1G anyway [18:57:33] If there's no issues with RAM then I'd be fine with it [18:58:11] cp3 uses less than 1G and cp12 uses 1G [18:58:13] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.70, 6.91, 6.47 [18:58:25] so 2G would be room to not OOM, not a base line usage [19:01:23] sounds good to me [19:04:00] https://phabricator.miraheze.org/T7783 [19:04:02] [url] ⚓ T7783 [New] Server Resource Request for UK Cache Proxy | phabricator.miraheze.org [19:04:07] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.24, 7.92, 7.01 [19:04:18] PROBLEM - mwtask1 Puppet on mwtask1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 33 seconds ago with 3 failures. Failed resources (up to 3 shown): Exec[git_clone_MediaWiki config],Exec[ExtensionMessageFiles],Mount[/mnt/mediawiki-static] [19:06:37] Reception123: I don't have access to the OVH with cloud3 [19:09:58] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.40, 7.72, 7.34 [19:10:04] i'm in JohnLewis [19:10:09] i'll create you an account and me [19:10:12] i used the root pass [19:10:24] paladox: how did you use the root password to get into OVH? [19:10:35] oh ovh i thought you meant cloud3 [19:11:03] I'll create you an account on uk.ovh [19:12:05] hmm, you have an account [19:12:11] so i guess you meant the US account? [19:13:51] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.66, 6.00, 6.75 [19:14:15] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.32, 7.58, 7.82 [19:14:40] Yeah, "the OVH with cloud3" [19:20:35] Oh is that the CA one? [19:22:11] Yeah [19:22:13] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.58, 5.78, 6.79 [19:26:10] CA? You mean CentralAuth? [19:26:24] Canada [19:26:34] Oh. [19:32:59] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.73, 8.25, 7.26 [19:34:17] JohnLewis: paladox both of you added to OVH CA [19:34:27] Thanks! [19:34:40] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.37, 6.66, 6.27 [19:35:03] Reception123: do you know when your import will be finished? [19:35:54] paladox: I'll look at them but I'll transfer the larger one [19:36:03] thanks! [19:36:13] are you going to handle the decom then? I thought John was [19:36:33] Reception123: can i once everything is done on the jobrunners reinstall them as mw12/13? Only need to reduce the load on the other MWs asap. [19:37:10] Ok, I think it's best if I transfer them all anyway now [19:37:17] I'll let you know when I'm done [19:37:17] thanks! [19:37:22] thanks!! [19:37:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.12, 21.23, 19.64 [19:38:43] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.52, 7.24, 7.22 [19:39:15] A copy of your installation's LocalSettings.php [19:39:15] must exist and be readable in the source directory. [19:39:15] Use --conf to specify it. [19:39:21] ^ paladox hmm mwtask1 doesn't seem operational yet [19:39:23] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.38, 20.03, 19.40 [19:39:29] oh [19:39:32] let me have a look [19:39:58] config is empty yeah [19:40:29] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.54, 6.63, 6.41 [19:40:56] PROBLEM - mwtask1 MediaWiki Rendering on mwtask1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 261 bytes in 0.018 second response time [19:41:03] yeah, puppet failing [19:41:53] https://www.irccloud.com/pastebin/6ecuoFME/ [19:42:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.96, 8.37, 7.63 [19:44:18] Reception123: i'm re-cloning right now [19:44:24] though gluster isn't mounting [19:44:27] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.62, 7.82, 6.95 [19:44:50] PROBLEM - mwtask1 MediaWiki Rendering on mwtask1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 0.015 second response time [19:44:51] thanks. Do you know why gluster isn't? [19:44:51] https://www.irccloud.com/pastebin/dzpPlVH2/ [19:44:52] oh [19:45:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JRX77 [19:45:34] [02miraheze/puppet] 07paladox 03509b396 - Revert "gluster: Remove gluster.[pem|key|ca] file (#1741)" [19:45:58] Reception123: ^ [19:46:42] ah [19:51:02] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 17.76, 22.04, 17.29 [19:52:47] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.63, 1.63, 1.28 [19:53:00] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.81, 20.20, 17.19 [19:53:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.83, 7.01, 7.97 [19:54:40] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.18, 7.90, 7.50 [19:54:40] RECOVERY - mwtask1 Check Gluster Clients on mwtask1 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [19:54:48] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.86, 1.78, 1.38 [19:54:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRXdr [19:55:01] [02miraheze/puppet] 07paladox 03270161c - Ensure mw config is cloned before adding OAuth2.key [19:56:38] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.56, 8.45, 7.77 [19:57:41] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.34, 7.34, 7.84 [19:58:34] RECOVERY - mwtask1 Puppet on mwtask1 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [19:58:46] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.36, 2.04, 1.56 [19:59:39] Reception123: should work now [20:00:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.67, 7.37, 7.92 [20:00:46] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.82, 1.84, 1.54 [20:01:01] RECOVERY - mwtask1 MediaWiki Rendering on mwtask1 is OK: HTTP OK: HTTP/1.1 200 OK - 20024 bytes in 3.424 second response time [20:01:36] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.71, 6.74, 7.55 [20:02:48] !log transferred all of today's imports to mwtask1 [20:02:50] paladox: all done :) [20:02:51] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.94, 1.45, 1.43 [20:03:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:14] Reception123: great, so just to confirm i can now reinstall jobrunner3/4 as mw12/13? [20:03:38] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.67, 7.79, 7.83 [20:03:40] yes as far as my files are concerned I've saved everything I need on my personal PC and am transferring them towards task1 now [20:03:54] awesome! [20:04:17] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.98, 7.49, 7.77 [20:04:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.87, 3.29 [20:05:17] hmm [20:05:24] which server hosts acme/letsencrypt [20:05:27] JohnLewis? [20:05:30] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=polandballfanonwiki /home/reception/polandballfanon_pages_full.xml --username-prefix wikia:polandballfanon --report 1 [20:05:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:24] paladox: maybe do have a look at homedirs and if you see anything in anyone's that looks important maybe transfer it [20:06:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.70, 3.29 [20:07:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.95, 7.55, 7.79 [20:07:43] Paladox: should be mwtask1, but depends on whether it is puppetised or not [20:08:11] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.96, 7.82, 7.89 [20:08:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRXb5 [20:08:53] [02miraheze/puppet] 07paladox 033ac9734 - Enable letsencrypt on mwtask1 [20:09:24] JohnLewis: ack [20:09:31] Reception123: ack [20:09:33] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.97, 7.97, 7.90 [20:10:46] guess i need to rsync letsencrypt from jobrunner3 to mwtask1 [20:11:04] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=blankballwiki /home/reception/linediscovery_pages_full.xml --username-prefix wikia:blankball --report 1 [20:11:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.81, 7.56, 7.77 [20:12:16] JohnLewis: ty on extra cp [20:12:35] Are we pooling it into the gb dns selector or as new [20:15:33] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.96, 7.49, 7.63 [20:16:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.24, 8.30, 7.93 [20:16:13] It would need to be pooled in as GB. Unsure the benefits of introducing it as it’s own DC besides needing to randomly decide how to distribute rather than letting DNS round robin it [20:16:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.08, 3.05, 3.23 [20:17:59] Yeah kinda seemed obvious but [20:22:33] PROBLEM - mwtask1 MirahezeRenewSsl on mwtask1 is CRITICAL: connect to address 198.244.181.23 and port 5000: Connection refused [20:24:09] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.63, 7.69, 8.00 [20:24:34] RECOVERY - mwtask1 MirahezeRenewSsl on mwtask1 is OK: TCP OK - 0.000 second response time on 198.244.181.23 port 5000 [20:30:12] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.06, 7.96, 7.90 [20:31:54] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.17, 2.60, 1.84 [20:34:42] PROBLEM - jobrunner4 APT on jobrunner4 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:34:44] PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:34:54] PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:34:57] PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:36:02] PROBLEM - jobrunner3 APT on jobrunner3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:36:08] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:36:40] PROBLEM - mwtask1 APT on mwtask1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:36:44] PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:37:51] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.42, 1.96, 1.79 [20:38:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.52, 7.56, 8.00 [20:39:42] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.17, 7.28, 7.99 [20:39:50] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.08, 1.66, 1.70 [20:41:42] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 7.61, 7.68, 8.07 [20:41:58] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.78, 6.49, 7.60 [20:43:43] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.29, 6.87, 7.73 [20:44:00] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.67, 7.43, 7.80 [20:45:59] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.67, 7.15, 7.66 [20:48:11] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.25, 7.20, 7.95 [20:49:57] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.65, 7.22, 7.52 [20:50:11] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.16, 7.87, 8.09 [20:50:20] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.04, 7.32, 7.45 [20:51:44] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: T7441 -Void, last run 17 minutes ago with 0 failures [20:51:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.47, 3.35, 3.04 [20:52:12] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.77, 7.32, 7.86 [20:53:43] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.85, 7.67, 7.51 [20:54:10] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.42, 8.64, 8.28 [20:55:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.96, 3.43, 3.16 [20:55:55] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.68, 7.37, 7.60 [20:57:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.22, 2.92, 3.00 [20:57:42] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.69, 7.72, 7.65 [20:58:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.67, 7.31, 7.91 [20:58:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.45, 7.15, 7.53 [21:03:45] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.80, 8.25, 7.72 [21:03:51] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.89, 7.13, 7.29 [21:04:10] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.25, 8.55, 8.18 [21:04:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.00, 7.83, 7.55 [21:05:51] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.64, 7.19, 7.32 [21:06:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.15, 7.31, 7.78 [21:06:22] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.08, 7.01, 7.28 [21:09:44] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.67, 7.24, 7.55 [21:09:50] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.23, 5.71, 6.69 [21:17:43] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.03, 7.40, 7.37 [21:18:22] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.41, 7.48, 7.25 [21:19:45] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.21, 6.36, 6.98 [21:20:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.35, 7.29, 7.20 [21:20:59] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.57, 6.87, 6.87 [21:21:43] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.90, 5.96, 6.76 [21:22:15] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.61, 6.22, 6.71 [21:22:58] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.77, 6.49, 6.74 [21:23:50] RhinosF1: around? [21:24:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.12, 5.94, 6.66 [21:25:23] Universal_Omega: around? [21:32:41] nvm [21:34:48] ok, backed up some of yours home drive [21:34:56] backups are under salt-user [21:38:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JR1Lo [21:38:36] [02miraheze/puppet] 07paladox 0355268ed - Install mw12 [21:38:37] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [21:38:39] [02puppet] 07paladox opened pull request 03#1869: Install mw12 - 13https://git.io/JR1LK [21:39:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+1/-0/±0] 13https://git.io/JR1LD [21:39:20] [02miraheze/puppet] 07paladox 03181afa3 - Create mw12.yaml [21:39:21] [02puppet] 07paladox synchronize pull request 03#1869: Install mw12 - 13https://git.io/JR1LK [21:44:03] !log recreate jobrunner3 as mw12 [21:44:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:46:53] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:46:57] paladox: sorry, was away for a bit but I am now. Thanks for doing that though! [21:47:33] PROBLEM - ping6 on jobrunner3 is CRITICAL: PING CRITICAL - Packet loss = 100% [21:48:02] PROBLEM - jobrunner3 HTTPS on jobrunner3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:08] PROBLEM - jobrunner3 MediaWiki Rendering on jobrunner3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:10] PROBLEM - jobrunner3 Current Load on jobrunner3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:48:41] PROBLEM - jobrunner3 SSH on jobrunner3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:47] PROBLEM - ping4 on jobrunner3 is CRITICAL: PING CRITICAL - Packet loss = 100% [21:48:57] PROBLEM - jobrunner3 Check Gluster Clients on jobrunner3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:49:03] PROBLEM - jobrunner3 php-fpm on jobrunner3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:49:08] PROBLEM - jobrunner3 MirahezeRenewSsl on jobrunner3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:49:10] PROBLEM - jobrunner3 PowerDNS Recursor on jobrunner3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:49:30] PROBLEM - jobrunner3 JobRunner Service on jobrunner3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:49:32] PROBLEM - jobrunner3 JobChron Service on jobrunner3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:50:06] PROBLEM - jobrunner3 Disk Space on jobrunner3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:50:12] PROBLEM - jobrunner3 Redis Process on jobrunner3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:50:16] PROBLEM - jobrunner3 NTP time on jobrunner3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:50] PROBLEM - Host jobrunner3 is DOWN: PING CRITICAL - Packet loss = 100% [21:54:27] [02mw-config] 07Universal-Omega closed pull request 03#4020: reserve jobchron.* and mwtask.* subdomains - 13https://git.io/JBuJq [21:54:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JR1q5 [21:54:30] [02miraheze/mw-config] 07The-Voidwalker 03e29a129 - reserve jobchron.* and mwtask.* subdomains (#4020) [21:54:32] [02mw-config] 07Universal-Omega deleted branch 03The-Voidwalker-patch-1 - 13https://git.io/vbvb3 [21:54:33] [02miraheze/mw-config] 07Universal-Omega deleted branch 03The-Voidwalker-patch-1 [21:54:53] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JR1mv [21:54:55] [02miraheze/dns] 07paladox 036e5472e - jobrunner3 -> mw12 [21:55:26] miraheze/mw-config - Universal-Omega the build passed. [21:57:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JR1m2 [21:57:21] [02miraheze/mw-config] 07Universal-Omega 033056579 - jobrunner* -> mwtask1 [21:57:22] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-5 - 13https://git.io/vbvb3 [21:57:24] [02mw-config] 07Universal-Omega opened pull request 03#4047: jobrunner* -> mwtask1 - 13https://git.io/JR1ma [21:58:08] paladox: ^ not 100% sure if only mwtask1 is needed there so just left it as PR for now. [21:58:17] miraheze/mw-config - Universal-Omega the build passed. [21:58:35] Universal_Omega: ack [22:02:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-7 [+0/-0/±1] 13https://git.io/JR1Yz [22:03:01] [02miraheze/mw-config] 07Universal-Omega 03ccfed77 - Add mw12 and mw13 to wgLinterSubmitterWhitelist [22:03:02] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-7 - 13https://git.io/vbvb3 [22:03:04] [02mw-config] 07Universal-Omega opened pull request 03#4048: Add mw12 and mw13 to wgLinterSubmitterWhitelist - 13https://git.io/JR1Yg [22:03:35] [02mw-config] 07Universal-Omega commented on pull request 03#4048: Add mw12 and mw13 to wgLinterSubmitterWhitelist - 13https://git.io/JR1Yi [22:04:03] miraheze/mw-config - Universal-Omega the build passed. [22:04:41] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.43, 6.99, 6.43 [22:06:51] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.36, 7.47, 6.78 [22:09:08] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.16, 6.77, 6.53 [22:13:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.98, 7.79, 7.08 [22:15:17] RECOVERY - Host jobrunner3 is UP: PING OK - Packet loss = 0%, RTA = 0.33 ms [22:15:17] RECOVERY - ping4 on jobrunner3 is OK: PING OK - Packet loss = 0%, RTA = 0.31 ms [22:15:35] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.98, 6.66, 6.66 [22:16:12] RECOVERY - jobrunner3 SSH on jobrunner3 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [22:17:56] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.39, 6.41, 6.73 [22:23:19] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.62, 1.78, 1.40 [22:24:10] I’ll brb in a bit [22:25:18] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.62, 1.72, 1.42 [22:27:18] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.30, 1.55, 1.39 [22:27:35] [02mw-config] 07Universal-Omega deleted a comment on pull request 03#4048: Add mw12 and mw13 to wgLinterSubmitterWhitelist - 13https://git.io/JR1Yi [22:31:58] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:33:05] !log [@test3] starting deploy of {'config': True} to skip [22:33:06] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:33:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:33:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:35:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.55, 6.42, 6.03 [22:37:34] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.24, 6.08, 5.96 [22:40:20] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.33, 6.99, 6.24 [22:41:47] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.81, 7.56, 6.41 [22:41:47] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: T7441 -Void, last run 8 minutes ago with 0 failures [22:43:42] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.25, 6.66, 6.21 [22:44:15] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.31, 6.58, 6.27 [22:46:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.51, 6.91, 6.53 [22:48:23] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.87, 1.52, 1.48 [22:52:25] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.84, 1.85, 1.59 [22:52:55] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.75, 6.23, 6.39 [22:54:24] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.14, 1.59, 1.53 [22:59:45] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:04:45] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.70, 8.01, 6.62 [23:06:44] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.35, 7.14, 6.46 [23:08:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.25, 6.55, 6.34 [23:12:13] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.85, 1.61, 1.48 [23:16:13] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.89, 1.33, 1.40 [23:26:11] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.72, 1.69, 1.51 [23:28:10] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.35, 1.56, 1.48 [23:53:09] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.32, 1.84, 1.50 [23:55:08] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.70, 1.41, 1.38