[00:02:45] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.71, 2.17, 1.60 [00:02:46] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:45] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.35, 1.93, 1.59 [00:05:03] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.20, 19.50, 17.08 [00:05:06] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 14.00, 9.15, 6.71 [00:05:20] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.27, 8.89, 6.70 [00:05:45] !log [@test3] starting deploy of {'config': True} to skip [00:05:46] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [00:05:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:06:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:06:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.27, 9.11, 6.61 [00:06:45] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.16, 1.61, 1.51 [00:07:03] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.30, 21.80, 18.25 [00:07:47] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CRITICAL: Full, 5987 files, 44.29MB, 2021-08-08 00:06:00 (2.3 weeks ago) [00:08:50] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.34, 2.92 [00:09:02] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.61, 20.89, 18.32 [00:09:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.38, 7.60, 6.73 [00:10:05] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.36, 7.32, 6.49 [00:10:47] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.47, 3.01 [00:11:03] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.83, 7.20, 6.74 [00:11:12] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.01, 6.76, 6.52 [00:12:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.74, 6.18, 6.18 [00:12:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.35, 3.49, 3.08 [00:13:05] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.26, 6.44, 6.51 [00:13:05] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.44, 23.25, 19.82 [00:14:46] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.26, 3.00, 2.95 [00:15:02] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.72, 22.84, 20.06 [00:19:03] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.77, 23.10, 20.73 [00:21:01] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.60, 7.97, 7.02 [00:21:01] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.70, 23.76, 21.35 [00:23:00] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.96, 6.83, 6.72 [00:23:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.38, 3.46, 3.20 [00:24:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.33, 5.98, 6.42 [00:25:01] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.50, 19.29, 20.08 [00:25:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.65, 3.29 [00:27:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 3.24, 3.19 [00:30:37] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [00:34:02] !log [@test3] starting deploy of {'config': True} to skip [00:34:03] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [00:34:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:34:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:53:04] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.85, 3.71, 3.40 [00:55:00] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.13, 3.10, 3.21 [01:03:56] !log [@test3] starting deploy of {'config': True} to skip [01:03:57] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:04:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:04:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:28:24] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.73, 3.50, 3.21 [01:30:22] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.65, 3.10, 3.09 [01:33:51] !log [@test3] starting deploy of {'config': True} to skip [01:33:52] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:34:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:34:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:53:53] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 3.63, 3.28 [01:55:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.57, 3.31 [01:59:42] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.96, 3.36, 3.29 [02:03:43] !log [@test3] starting deploy of {'config': True} to skip [02:03:44] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [02:03:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:03:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:12:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.32, 3.14 [02:14:25] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.38, 3.21, 3.12 [02:22:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.36, 3.14 [02:24:22] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.48, 3.48, 3.19 [02:26:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.73, 3.11, 3.09 [02:26:36] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.53, 7.56, 5.14 [02:27:15] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.97, 6.91, 5.13 [02:28:33] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.55, 6.29, 4.96 [02:29:14] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.82, 5.74, 4.91 [02:33:46] !log [@test3] starting deploy of {'config': True} to skip [02:33:47] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [02:33:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:33:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:36:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.18, 3.02 [02:38:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.59, 2.84, 2.91 [02:50:22] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.62, 3.77, 3.27 [02:52:23] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.60, 3.27 [02:56:24] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.35, 3.35, 3.24 [03:03:29] !log [@test3] starting deploy of {'config': True} to skip [03:03:30] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [03:03:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:03:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:33:43] !log [@test3] starting deploy of {'config': True} to skip [03:33:44] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [03:33:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:33:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:36:23] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.95, 3.38 [03:38:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.68, 3.44, 3.26 [03:44:22] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.20, 2.92, 3.15 [04:03:29] !log [@test3] starting deploy of {'config': True} to skip [04:03:30] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [04:03:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:03:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:33:54] !log [@test3] starting deploy of {'config': True} to skip [04:33:55] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [04:33:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:34:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:53:30] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.16, 1.60, 1.11 [04:55:29] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.96, 1.36, 1.08 [05:03:48] !log [@test3] starting deploy of {'config': True} to skip [05:03:49] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [05:03:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:03:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:55] !log [@test3] starting deploy of {'config': True} to skip [05:33:56] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [05:33:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:34:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:03:48] !log [@test3] starting deploy of {'config': True} to skip [06:03:49] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [06:03:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:03:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:21:54] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 6.22, 3.91, 2.02 [06:33:34] !log [@test3] starting deploy of {'config': True} to skip [06:33:35] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [06:33:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:33:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:33:45] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 0.62, 1.44, 1.92 [06:37:42] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.65, 1.04, 1.64 [06:49:36] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 7.30, 4.19, 2.51 [06:55:35] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 0.35, 1.75, 1.99 [06:59:31] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.89, 1.07, 1.63 [07:03:00] !log [@test3] starting deploy of {'config': True} to skip [07:03:01] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [07:03:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:03:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:33:01] !log [@test3] starting deploy of {'config': True} to skip [07:33:02] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [07:33:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:33:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:03:02] !log [@test3] starting deploy of {'config': True} to skip [08:03:03] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [08:03:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:03:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:33:48] !log [@test3] starting deploy of {'config': True} to skip [08:33:49] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [08:33:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:33:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:03:29] !log [@test3] starting deploy of {'config': True} to skip [09:03:30] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [09:03:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:03:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:10:30] PROBLEM - sportsanalytics.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sportsanalytics.wiki' expires in 15 day(s) (Thu 09 Sep 2021 09:01:42 GMT +0000). [09:12:02] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JEZ3o [09:12:03] [02miraheze/ssl] 07MirahezeSSLBot 03e2887e7 - Bot: Update SSL cert for sportsanalytics.wiki [09:32:56] !log [@test3] starting deploy of {'config': True} to skip [09:32:57] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [09:33:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:33:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:35:42] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.68, 6.19, 4.32 [09:36:03] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.49, 6.24, 4.30 [09:36:11] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.16, 6.52, 4.61 [09:37:40] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.94, 5.31, 4.22 [09:37:53] RECOVERY - sportsanalytics.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'sportsanalytics.wiki' will expire on Mon 22 Nov 2021 08:11:56 GMT +0000. [09:38:04] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.15, 5.30, 4.19 [09:38:09] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.63, 5.41, 4.43 [09:51:38] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.79, 6.79, 5.15 [09:53:38] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.47, 6.09, 5.08 [10:03:20] !log [@test3] starting deploy of {'config': True} to skip [10:03:21] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [10:03:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:03:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:33:03] !log [@test3] starting deploy of {'config': True} to skip [10:33:04] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [10:33:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:33:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:03:16] !log [@test3] starting deploy of {'config': True} to skip [11:03:17] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [11:03:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:03:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:33:29] !log [@test3] starting deploy of {'config': True} to skip [11:33:30] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [11:33:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:33:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:03:42] !log [@test3] starting deploy of {'config': True} to skip [12:03:43] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [12:03:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:03:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:56] !log [@test3] starting deploy of {'config': True} to skip [12:33:57] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [12:33:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:34:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:03:26] !log [@test3] starting deploy of {'config': True} to skip [13:03:27] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [13:03:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:03:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:33:50] !log [@test3] starting deploy of {'config': True} to skip [13:33:51] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [13:33:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:34:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:03:43] !log [@test3] starting deploy of {'config': True} to skip [14:03:44] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [14:03:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:03:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:06:05] https://phabricator.miraheze.org/T7859 [14:06:06] [url] ⚓ T7859 test3 puppet is constantly refreshing config | phabricator.miraheze.org [14:34:23] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.59, 3.39, 2.87 [14:36:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.41, 2.96 [14:40:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.93, 3.22, 2.99 [14:59:27] Reception123: access? [15:20:27] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [15:22:04] PROBLEM - test4 APT on test4 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [15:22:13] PROBLEM - db11 APT on db11 is CRITICAL: APT CRITICAL: 64 packages available for upgrade (2 critical updates). [15:22:20] PROBLEM - db12 APT on db12 is CRITICAL: APT CRITICAL: 73 packages available for upgrade (2 critical updates). [15:22:41] PROBLEM - mon2 APT on mon2 is CRITICAL: APT CRITICAL: 15 packages available for upgrade (3 critical updates). [15:22:51] PROBLEM - gluster3 APT on gluster3 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [15:23:03] PROBLEM - cp12 APT on cp12 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [15:23:09] PROBLEM - mem2 APT on mem2 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [15:23:52] PROBLEM - puppet3 APT on puppet3 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (2 critical updates). [15:25:05] PROBLEM - db13 APT on db13 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (2 critical updates). [15:26:20] PROBLEM - gluster4 APT on gluster4 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [15:26:54] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [15:27:38] PROBLEM - cloud5 APT on cloud5 is CRITICAL: APT CRITICAL: 70 packages available for upgrade (2 critical updates). [15:29:50] PROBLEM - mail2 APT on mail2 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [15:30:16] PROBLEM - cp13 APT on cp13 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [15:31:55] PROBLEM - dbbackup1 APT on dbbackup1 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (2 critical updates). [15:32:01] PROBLEM - graylog2 APT on graylog2 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (2 critical updates). [15:33:25] PROBLEM - phab2 APT on phab2 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [15:35:07] PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [15:35:34] PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [15:35:35] PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [15:35:52] PROBLEM - mw13 APT on mw13 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [15:35:58] PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [15:35:58] PROBLEM - mw12 APT on mw12 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [15:36:08] PROBLEM - mwtask1 APT on mwtask1 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [15:36:32] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [15:36:43] PROBLEM - mem1 APT on mem1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [15:38:06] PROBLEM - ldap2 APT on ldap2 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [15:39:29] PROBLEM - cp14 APT on cp14 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [15:41:11] PROBLEM - services4 APT on services4 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [15:44:20] PROBLEM - services3 APT on services3 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [15:44:39] PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [15:45:13] PROBLEM - cloud4 APT on cloud4 is CRITICAL: APT CRITICAL: 70 packages available for upgrade (2 critical updates). [15:47:04] PROBLEM - jobchron1 APT on jobchron1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [15:49:54] PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 106 packages available for upgrade (2 critical updates). [15:52:58] JohnLewis: I have zero idea what test3 is doing [15:53:09] Reception123 was supposed to have fixed puppet yesterday [15:53:31] Puppet isn't likely to be the problem [15:54:35] [02puppet] 07RhinosF1 edited pull request 03#1885: mwdeploy: add rsync stuff - 13https://git.io/J0Rrq [15:55:36] JohnLewis: it's puppet that triggers it [15:55:49] For some reason it's stuck refreshing config on every run [15:55:59] But puppet runs git [15:56:04] true [15:56:27] Puppet doesn't run randomly and consistently unless the underlying command causes a difference [15:57:00] So git for config can't be claiming there's no change [15:59:11] !log re-cloned config on test3 as the repo was ahead by numerous commits and had stashed, uncommitted changes [15:59:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:00:32] JohnLewis: can you renick ls bot too [16:09:23] * Reception123 is around now [16:12:52] Reception123: Welcome, did you see my comment on that task? :(( [16:12:59] Ugochimobi: hi. I'll have a look [16:13:19] Okay, I hope so, [16:17:00] I guess since there's no more jobrunner there's no more https://phabricator.miraheze.org/T7801 ? [16:17:01] [url] ⚓ T7801 Segmentation fault in jobrunner | phabricator.miraheze.org [16:24:19] Reception123: there is jobrunner [16:24:27] The software not the server [16:24:34] Look at the date anyway [16:25:15] ah, I thought it could've perhaps been fixed by the switch but considering the date seems like it's not the case [16:25:18] (It's after jobrunner* was decom'd) [16:25:24] Reception123: no [16:25:35] yeah, realised now [16:30:32] Reception123: https://phabricator.wikimedia.org/T289585 [16:30:32] [url] ⚓ T289585 Release MW 1.37.0 | phabricator.wikimedia.org [16:30:51] RhinosF1: ack, it's a start then [16:30:56] 4 weeks to cut [16:31:28] 17:04:18 Aha, my calendar just reminded me that it's a month to the REL1_37 / 1.38.0-wmf.1 branch point. [16:34:42] I hope we figure things out by then (November) [16:35:47] We will [16:36:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.56, 3.13 [16:38:04] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.25, 3.37, 3.11 [18:16:58] !log upgrade grafana to 8.1.2 [18:17:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.61, 3.02 [18:25:07] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.74, 3.04, 2.89 [18:34:59] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.79, 6.81, 5.50 [18:34:59] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.40, 7.65, 6.01 [18:35:41] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.06, 6.88, 5.35 [18:36:56] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.83, 5.92, 5.34 [18:36:57] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.70, 6.52, 5.79 [18:37:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.14, 6.02, 5.21 [18:44:20] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JEcBa [18:44:22] [02miraheze/puppet] 07JohnFLewis 039336e01 - Enable Grafana v2 alerts system [19:35:50] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.44, 7.47, 5.84 [19:36:45] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.09, 6.56, 5.26 [19:36:49] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.98, 8.17, 6.14 [19:36:59] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.53, 7.52, 5.99 [19:37:18] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.83, 3.84, 3.23 [19:38:43] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.38, 6.43, 5.37 [19:39:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.25, 3.74, 3.28 [19:45:04] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 3.75, 3.40 [19:47:00] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.34, 3.33, 3.30 [19:48:35] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.47, 7.91, 7.19 [19:50:31] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.03, 8.64, 7.57 [19:51:36] PROBLEM - ping4 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 151.07 ms [19:51:38] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.66, 7.98, 7.50 [19:52:30] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.79, 7.92, 7.43 [19:52:47] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.83, 7.87, 7.74 [19:53:39] RECOVERY - ping4 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 102.37 ms [19:54:27] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.92, 8.37, 7.66 [19:56:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.94, 7.73, 7.50 [19:57:38] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.33, 6.00, 6.80 [20:02:40] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.41, 5.17, 6.45 [20:04:20] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.40, 7.25, 7.19 [20:06:17] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.52, 6.63, 6.98 [20:08:15] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.02, 5.68, 6.59 [20:27:25] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.39, 1.79, 1.35 [20:29:23] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.99, 1.78, 1.39 [20:31:22] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.21, 1.55, 1.36 [20:35:06] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.60, 7.10, 6.01 [20:37:05] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.89, 6.20, 5.81 [20:52:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.99, 3.50, 3.13 [20:56:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.65, 2.70, 2.89 [21:04:41] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.09, 6.53, 5.65 [21:06:37] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.00, 5.49, 5.37 [21:22:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.33, 3.45, 3.04 [21:24:24] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.32, 3.34, 3.05 [22:05:41] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.45, 7.38, 5.51 [22:06:14] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.83, 7.13, 5.72 [22:07:38] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.75, 6.52, 5.42 [22:08:11] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.16, 6.39, 5.62 [22:34:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.84, 6.99, 5.64 [22:36:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.54, 6.08, 5.48 [22:54:37] PROBLEM - ping4 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 90.82 ms [22:56:39] RECOVERY - ping4 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 90.39 ms [23:05:41] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.47, 6.50, 5.46 [23:07:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.56, 6.06, 5.42 [23:17:02] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.64, 7.11, 6.00 [23:17:13] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.42, 6.85, 5.77 [23:17:15] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.05, 7.47, 6.08 [23:18:59] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.73, 6.83, 6.03 [23:19:11] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.28, 6.40, 5.74 [23:19:12] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.20, 7.20, 6.14 [23:20:57] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.99, 6.20, 5.90 [23:21:09] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.23, 6.20, 5.91 [23:36:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.75, 7.01, 6.14 [23:38:40] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.83, 5.84, 5.82