[00:02:45] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown) [00:05:53] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.90, 7.99, 5.75 [00:06:14] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.04, 7.31, 5.67 [00:06:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.12, 7.25, 5.66 [00:07:03] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.99, 23.18, 18.34 [00:07:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.85, 7.40, 5.82 [00:08:12] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.07, 6.73, 5.69 [00:08:30] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.81, 6.50, 5.59 [00:09:00] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.53, 20.57, 17.97 [00:09:47] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.72, 6.04, 5.50 [00:13:02] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.80, 23.02, 19.54 [00:21:02] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.98, 23.69, 21.70 [00:22:22] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.70, 3.19 [00:26:24] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.56, 3.38, 3.21 [00:29:00] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.11, 19.23, 20.32 [00:30:38] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [00:35:04] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.34, 21.49, 20.85 [00:37:02] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.83, 18.79, 19.95 [01:05:31] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.47, 18.06, 16.85 [01:07:27] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.38, 16.66, 16.46 [01:17:48] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 6.00, 2.60, 1.70 [01:19:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.66, 3.27 [01:21:48] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.65, 2.00, 1.65 [01:21:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.77, 3.37 [01:23:48] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 3.02, 2.31, 1.80 [01:25:48] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 0.99, 1.79, 1.67 [01:29:46] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.67, 1.36, 1.53 [01:31:35] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.78, 3.31, 3.36 [01:39:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.74, 3.52 [01:41:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.94, 3.71, 3.53 [01:41:47] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.72, 1.97, 1.68 [01:43:19] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.72, 4.07, 3.68 [01:45:45] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.58, 1.65, 1.61 [01:47:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.86, 3.69 [01:49:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.91, 3.72 [01:51:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.87, 3.73 [01:55:07] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.83, 3.71 [02:02:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.69, 3.69 [02:13:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.35, 7.28, 5.48 [02:13:16] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.19, 7.99, 5.92 [02:13:27] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.63, 7.17, 5.37 [02:13:53] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.28, 8.37, 6.11 [02:13:56] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.91, 7.76, 5.72 [02:14:18] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.24, 7.48, 5.91 [02:15:05] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.86, 6.63, 5.47 [02:15:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.48, 7.20, 5.87 [02:15:25] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.68, 6.26, 5.26 [02:15:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.82, 7.19, 5.97 [02:15:55] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.03, 6.32, 5.44 [02:16:14] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.21, 6.41, 5.71 [02:16:44] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.05, 3.02, 3.40 [02:17:13] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.84, 5.98, 5.58 [02:17:47] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.60, 6.37, 5.82 [02:40:34] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 8.42, 2.91, 1.74 [02:44:34] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.05, 1.77, 1.53 [02:46:36] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.86, 1.48, 1.45 [03:08:39] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.43, 3.31 [03:10:35] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.65, 3.22, 3.26 [03:11:36] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.94, 1.98, 1.62 [03:13:35] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.09, 1.65, 1.54 [03:19:59] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.60, 6.35, 5.61 [03:21:55] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.82, 5.67, 5.44 [03:35:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.66, 7.11, 5.65 [03:35:50] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.10, 6.36, 5.24 [03:36:29] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.09, 7.42, 5.90 [03:36:47] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.75, 9.37, 6.83 [03:37:48] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.94, 5.47, 5.05 [03:38:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.76, 3.44 [03:38:30] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.35, 6.25, 5.67 [03:38:45] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.18, 7.65, 6.49 [03:39:05] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.09, 6.45, 5.84 [03:39:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.69, 2.96, 3.18 [03:40:43] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.19, 6.09, 6.06 [03:51:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 3.66, 3.32 [03:53:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.49, 3.29 [03:57:34] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.79, 3.24, 3.24 [04:05:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.28, 3.19 [04:11:16] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.35, 3.32, 3.22 [04:11:17] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.65, 7.26, 5.78 [04:12:16] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.77, 7.34, 6.09 [04:13:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.06, 6.82, 5.81 [04:14:15] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.74, 6.27, 5.84 [04:15:13] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.76, 6.13, 5.68 [04:17:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.61, 3.36 [04:21:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.69, 3.42 [04:23:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.08, 3.36, 3.33 [04:35:41] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.82, 6.52, 5.27 [04:37:38] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.74, 5.38, 4.99 [05:33:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.57, 3.18, 2.83 [05:35:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.34, 3.06, 2.82 [05:45:04] RECOVERY - jobchron1 APT on jobchron1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:03:08] RECOVERY - db13 APT on db13 is OK: APT OK: 50 packages available for upgrade (0 critical updates). [06:03:40] RECOVERY - dbbackup1 APT on dbbackup1 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [06:03:46] RECOVERY - cp14 APT on cp14 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:03:52] RECOVERY - mw12 APT on mw12 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:06:23] RECOVERY - mw9 APT on mw9 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:08:09] RECOVERY - db11 APT on db11 is OK: APT OK: 62 packages available for upgrade (0 critical updates). [06:10:20] PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 136.29 ms [06:10:41] RECOVERY - mon2 APT on mon2 is OK: APT OK: 11 packages available for upgrade (0 critical updates). [06:10:54] PROBLEM - ping4 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 138.85 ms [06:10:54] PROBLEM - ping4 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 138.77 ms [06:11:02] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 154.11 ms [06:12:24] RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 97.60 ms [06:12:54] RECOVERY - ping4 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 101.58 ms [06:12:56] RECOVERY - ping4 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 100.21 ms [06:13:02] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 102.80 ms [06:13:23] RECOVERY - mw10 APT on mw10 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:14:43] RECOVERY - mem1 APT on mem1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:15:54] RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 104 packages available for upgrade (0 critical updates). [06:16:38] RECOVERY - cp3 APT on cp3 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:17:14] RECOVERY - cloud4 APT on cloud4 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [06:19:09] RECOVERY - mem2 APT on mem2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:20:50] RECOVERY - gluster3 APT on gluster3 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:21:34] RECOVERY - test3 APT on test3 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:21:58] RECOVERY - graylog2 APT on graylog2 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [06:25:40] RECOVERY - cloud5 APT on cloud5 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [06:28:00] RECOVERY - mail2 APT on mail2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:28:24] RECOVERY - mw13 APT on mw13 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:29:21] RECOVERY - phab2 APT on phab2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:33:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.18, 3.43, 3.13 [06:34:14] RECOVERY - cp13 APT on cp13 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:35:23] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.90, 2.85, 2.95 [06:35:39] RECOVERY - services4 APT on services4 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:40:04] RECOVERY - test4 APT on test4 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:40:12] RECOVERY - mwtask1 APT on mwtask1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:41:39] RECOVERY - ns1 APT on ns1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:43:28] RECOVERY - mw11 APT on mw11 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:44:00] RECOVERY - gluster4 APT on gluster4 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:46:15] RECOVERY - services3 APT on services3 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:46:27] RECOVERY - mw8 APT on mw8 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:47:51] RECOVERY - puppet3 APT on puppet3 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [06:48:06] RECOVERY - ldap2 APT on ldap2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:54:22] RECOVERY - ns2 APT on ns2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:56:58] RECOVERY - cp12 APT on cp12 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:58:20] RECOVERY - db12 APT on db12 is OK: APT OK: 71 packages available for upgrade (0 critical updates). [07:05:40] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.58, 6.26, 4.79 [07:06:13] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.87, 6.59, 5.25 [07:07:38] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.81, 5.63, 4.74 [07:08:11] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.37, 5.96, 5.16 [07:35:11] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.52, 7.31, 5.34 [07:36:12] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.27, 6.85, 5.57 [07:37:09] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.87, 6.43, 5.24 [07:38:10] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.07, 5.81, 5.35 [07:47:41] PROBLEM - parapedia.pl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'parapedia.pl' expires in 15 day(s) (Fri 10 Sep 2021 07:44:29 GMT +0000). [07:49:15] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JEWgw [07:49:17] [02miraheze/ssl] 07MirahezeSSLBot 03ee26afa - Bot: Update SSL cert for parapedia.pl [07:49:58] PROBLEM - heavyironmodding.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'heavyironmodding.org' expires in 15 day(s) (Fri 10 Sep 2021 07:41:26 GMT +0000). [07:53:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JEW21 [07:53:37] [02miraheze/ssl] 07MirahezeSSLBot 036eb8293 - Bot: Update SSL cert for heavyironmodding.org [08:00:30] PROBLEM - ping4 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 153.03 ms [08:02:31] RECOVERY - ping4 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 90.14 ms [08:04:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.32, 6.18, 4.98 [08:06:55] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.24, 5.58, 4.89 [08:07:50] RECOVERY - parapedia.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'parapedia.pl' will expire on Tue 23 Nov 2021 06:49:10 GMT +0000. [08:10:48] RECOVERY - heavyironmodding.org - LetsEncrypt on sslhost is OK: OK - Certificate 'heavyironmodding.org' will expire on Tue 23 Nov 2021 06:53:30 GMT +0000. [08:14:38] PROBLEM - ping4 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 143.19 ms [08:16:38] RECOVERY - ping4 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 89.70 ms [08:35:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.81, 7.78, 6.40 [08:37:38] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.42, 6.75, 6.18 [08:50:43] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JEWDO [08:50:45] [02miraheze/ssl] 07Reception123 030adceb8 - add nl.xliving.tk cert [11:05:51] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.48, 6.73, 5.34 [11:07:52] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.96, 5.66, 5.11 [11:25:12] !log DROPPED https://phabricator.miraheze.org/P442 (wiki databases that didn't have corresponding cw_wiki entries) [11:25:13] [url] ✎ P442 Dropped wiki databases 25/08/2021 (no cw_wikis entry) | phabricator.miraheze.org [11:25:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:15:56] paladox: hi. How should I do the bullseye update for mw* servers? Would I 'upgrade' and follow https://www.debian.org/releases/bullseye/mips64el/release-notes/ch-upgrading.en.html#backup or would a reimage be better? [12:15:56] [url] Chapter 4. Upgrades from Debian 10 (buster) | www.debian.org [12:22:42] [02puppet] 07RhinosF1 opened pull request 03#1888: test3: update php for bullseye testing - 13https://git.io/JElgf [12:23:10] Reception123: ^ [12:23:31] I imagine you'll have to manually purge old 7.3 packages [12:23:37] thanks [12:24:34] [02puppet] 07RhinosF1 opened pull request 03#1889: users: allow MediaWiki-admin php7.4; drop 7.2 - 13https://git.io/JElg8 [12:24:43] Reception123: and ^ [12:25:04] ack, that I'll merge now since there's no more 7.2 anyway [12:25:19] [02puppet] 07Reception123 closed pull request 03#1889: users: allow MediaWiki-admin php7.4; drop 7.2 - 13https://git.io/JElg8 [12:25:20] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JElgV [12:25:22] [02miraheze/puppet] 07RhinosF1 03e94efc5 - users: allow MediaWiki-admin php7.4; drop 7.2 (#1889) [12:26:01] Reception123: perfect [12:26:37] The php version can probably go right before the update [12:27:02] Someone just needs to note what gets installed so we can purge 7.3 packages [12:28:17] I would run apt-get upgrade before dist-upgrade [12:28:20] yeah [12:28:21] Seperately [12:28:28] And also note the update ssh first [12:28:36] Or you'll be crying [12:29:13] Obvious thing is expect downtime [12:29:43] * RhinosF1 also sees you might need to run twice if anything is held back [12:30:00] oh yeah there's that too [12:30:07] and apt autoremove after [12:30:25] yup that was in the docs too [12:30:30] Don't forget https://www.debian.org/releases/bullseye/mips64el/release-notes/ch-information.en.html#security-archive [12:30:31] [url] Chapter 5. Issues to be aware of for bullseye | www.debian.org [12:31:50] yup, a lot to do [12:32:38] Looking at it, best would be openssh server, then do the php change, then apt-get upgrade --without-new-pkgs, then apt-get upgrade, then apt-get dist-upgrade [12:32:45] That will probably be least downtime [12:34:01] yeah I'll have a list of things and paste it here to make sure I've got everything [12:41:20] https://www.irccloud.com/pastebin/54MRtfCc/ [12:41:27] something like this, does this look right? [12:43:25] Reception123: run puppet after the php change [12:43:43] That will probably install php updates so note what it does in case 7.3 don't get removed [12:43:52] yeah, of course [12:45:02] Full upgrade needs to be before autoremove [12:45:39] And of course the collation script [12:45:50] (Do not run from mwtask, it must be test3) [12:45:59] switched and yeah the collation script would be last then [12:49:01] Don't forget run puppet after the php change [12:49:15] But otherwise lgtm [12:50:23] ok, sounds good. Will do later the [12:50:25] *then [12:57:10] Collation needs --force Reception123 [12:57:47] ah, noted [12:59:59] Reception123: reimage [13:00:22] paladox: I have been suggesting that for weeks [13:00:24] RhinosF1: you probably want to allow 7.4 first then drop 7.2 [13:00:37] Since some mw would be 7.4 and some 7.2 [13:00:46] paladox: it was done in one or [13:00:49] PR [13:00:52] Oh [13:00:58] Servers are 7.3 at moment paladox [13:01:30] 7.2 is legacy [13:01:34] I misread :P [13:04:59] [02miraheze/CreateWiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JElPJ [13:05:00] [02miraheze/CreateWiki] 07RhinosF1 031d5fb43 - Update continuousIntegration.yml [13:05:02] [02CreateWiki] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/vpJTL [13:05:03] [02CreateWiki] 07RhinosF1 opened pull request 03#237: Update continuousIntegration.yml - 13https://git.io/JElPT [13:05:22] [02miraheze/landing] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JElPt [13:05:24] [02miraheze/landing] 07RhinosF1 0351147bb - Update continuousIntegration.yml [13:05:43] [02CreateWiki] 07RhinosF1 closed pull request 03#237: Update continuousIntegration.yml - 13https://git.io/JElPT [13:05:44] [02miraheze/CreateWiki] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JElPB [13:05:46] [02miraheze/CreateWiki] 07RhinosF1 0365b6b81 - Update continuousIntegration.yml (#237) [13:05:47] [02miraheze/CreateWiki] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [13:05:49] [02CreateWiki] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/vpJTL [13:05:51] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.26, 6.90, 5.74 [13:06:11] miraheze/CreateWiki - RhinosF1 the build passed. [13:06:11] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JElPz [13:06:13] [02miraheze/mw-config] 07RhinosF1 03c0a7467 - Update continuousIntegration.yml [13:06:14] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-3 - 13https://git.io/vbvb3 [13:06:16] [02mw-config] 07RhinosF1 opened pull request 03#4062: Update continuousIntegration.yml - 13https://git.io/JElP2 [13:06:22] miraheze/landing - RhinosF1 the build passed. [13:06:30] [02miraheze/ManageWiki] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JElPo [13:06:31] [02miraheze/ManageWiki] 07RhinosF1 03afbe43f - Update continuousIntegration.yml [13:06:40] miraheze/CreateWiki - RhinosF1 the build passed. [13:06:55] [02mw-config] 07RhinosF1 closed pull request 03#4062: Update continuousIntegration.yml - 13https://git.io/JElP2 [13:06:56] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JElPD [13:06:58] [02miraheze/mw-config] 07RhinosF1 03a34441e - Update continuousIntegration.yml (#4062) [13:06:59] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-3 [13:07:01] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-3 - 13https://git.io/vbvb3 [13:07:18] [02miraheze/MirahezeMagic] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JElPQ [13:07:18] miraheze/mw-config - RhinosF1 the build passed. [13:07:20] [02miraheze/MirahezeMagic] 07RhinosF1 03a79b1e6 - Update continuousIntegration.yml [13:07:21] [02MirahezeMagic] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/fQRGX [13:07:23] [02MirahezeMagic] 07RhinosF1 opened pull request 03#283: Update continuousIntegration.yml - 13https://git.io/JElP7 [13:07:30] miraheze/ManageWiki - RhinosF1 the build passed. [13:07:40] [02miraheze/MatomoAnalytics] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JElPA [13:07:42] [02miraheze/MatomoAnalytics] 07RhinosF1 038fbf4b7 - Update continuousIntegration.yml [13:07:43] [02MatomoAnalytics] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/fN4LT [13:07:45] [02MatomoAnalytics] 07RhinosF1 opened pull request 03#53: Update continuousIntegration.yml - 13https://git.io/JElPh [13:07:50] miraheze/mw-config - RhinosF1 the build passed. [13:07:53] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.78, 5.77, 5.47 [13:08:17] [02miraheze/WikiDiscover] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JElXm [13:08:19] [02miraheze/WikiDiscover] 07RhinosF1 03a4f1561 - Update continuousIntegration.yml [13:08:20] [02WikiDiscover] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/vhUAp [13:08:22] [02WikiDiscover] 07RhinosF1 opened pull request 03#56: Update continuousIntegration.yml - 13https://git.io/JElX3 [13:08:34] miraheze/MirahezeMagic - RhinosF1 the build passed. [13:08:35] [02MirahezeMagic] 07RhinosF1 closed pull request 03#283: Update continuousIntegration.yml - 13https://git.io/JElP7 [13:08:36] [02miraheze/MirahezeMagic] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JElXG [13:08:38] [02miraheze/MirahezeMagic] 07RhinosF1 03a44b22c - Update continuousIntegration.yml (#283) [13:08:39] [02MirahezeMagic] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/fQRGX [13:08:41] [02miraheze/MirahezeMagic] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [13:08:43] [02MatomoAnalytics] 07RhinosF1 closed pull request 03#53: Update continuousIntegration.yml - 13https://git.io/JElPh [13:08:45] [02miraheze/MatomoAnalytics] 07RhinosF1 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JElXC [13:08:46] [02miraheze/MatomoAnalytics] 07RhinosF1 03afd8e91 - Merge pull request #53 from miraheze/RhinosF1-patch-1 [13:08:46] miraheze/MatomoAnalytics - RhinosF1 the build passed. [13:08:48] [02MatomoAnalytics] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/fN4LT [13:08:49] [02miraheze/MatomoAnalytics] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [13:08:51] [02WikiDiscover] 07RhinosF1 closed pull request 03#56: Update continuousIntegration.yml - 13https://git.io/JElX3 [13:08:53] [02miraheze/WikiDiscover] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JElX8 [13:08:54] [02miraheze/WikiDiscover] 07RhinosF1 03bc961ce - Update continuousIntegration.yml (#56) [13:08:56] [02miraheze/WikiDiscover] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [13:08:57] [02WikiDiscover] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/vhUAp [13:09:24] miraheze/WikiDiscover - RhinosF1 the build passed. [13:09:25] [02miraheze/DataDump] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JElXz [13:09:26] [02miraheze/DataDump] 07RhinosF1 03b5d1ad0 - Update continuousIntegration.yml [13:09:28] [02DataDump] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/fhhKV [13:09:38] miraheze/MirahezeMagic - RhinosF1 the build passed. [13:09:41] [02miraheze/IncidentReporting] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JElXo [13:09:43] [02miraheze/IncidentReporting] 07RhinosF1 03bb3d2d7 - Update continuousIntegration.yml [13:09:45] miraheze/MatomoAnalytics - RhinosF1 the build passed. [13:09:51] miraheze/WikiDiscover - RhinosF1 the build passed. [13:09:54] [02DataDump] 07RhinosF1 opened pull request 03#33: Update continuousIntegration.yml - 13https://git.io/JElXX [13:10:05] [02DataDump] 07RhinosF1 closed pull request 03#33: Update continuousIntegration.yml - 13https://git.io/JElXX [13:10:06] [02miraheze/DataDump] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JElX9 [13:10:08] [02miraheze/DataDump] 07RhinosF1 032a22c9c - Update continuousIntegration.yml (#33) [13:10:09] [02miraheze/DataDump] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [13:10:11] [02DataDump] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/fhhKV [13:10:34] [02miraheze/RottenLinks] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JElXj [13:10:36] [02miraheze/RottenLinks] 07RhinosF1 03421553b - Update continuousIntegration.yml [13:10:37] [02RottenLinks] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/fxBrr [13:10:38] miraheze/IncidentReporting - RhinosF1 the build passed. [13:10:40] [02RottenLinks] 07RhinosF1 opened pull request 03#40: Update continuousIntegration.yml - 13https://git.io/JEl1v [13:10:56] miraheze/DataDump - RhinosF1 the build passed. [13:11:00] [02miraheze/RemovePII] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JEl1U [13:11:01] [02miraheze/RemovePII] 07RhinosF1 03c95d524 - Update continuousIntegration.yml [13:11:03] [02RemovePII] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/Jsfpj [13:11:05] [02RemovePII] 07RhinosF1 opened pull request 03#18: Update continuousIntegration.yml - 13https://git.io/JEl1I [13:11:06] miraheze/DataDump - RhinosF1 the build passed. [13:11:29] [02miraheze/GlobalNewFiles] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JEl1G [13:11:30] [02miraheze/GlobalNewFiles] 07RhinosF1 03d1324a4 - Update continuousIntegration.yml [13:11:32] [02GlobalNewFiles] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/JtShO [13:11:33] [02GlobalNewFiles] 07RhinosF1 opened pull request 03#18: Update continuousIntegration.yml - 13https://git.io/JEl1n [13:11:41] [02RottenLinks] 07RhinosF1 closed pull request 03#40: Update continuousIntegration.yml - 13https://git.io/JEl1v [13:11:42] [02miraheze/RottenLinks] 07RhinosF1 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JEl1W [13:11:44] [02miraheze/RottenLinks] 07RhinosF1 0384896fe - Merge pull request #40 from miraheze/RhinosF1-patch-1 [13:11:45] [02RottenLinks] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/fxBrr [13:11:47] [02miraheze/RottenLinks] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [13:11:49] [02RemovePII] 07RhinosF1 closed pull request 03#18: Update continuousIntegration.yml - 13https://git.io/JEl1I [13:11:51] [02miraheze/RemovePII] 07RhinosF1 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JEl1R [13:11:52] [02miraheze/RemovePII] 07RhinosF1 03e5231af - Merge pull request #18 from miraheze/RhinosF1-patch-1 [13:11:53] miraheze/RottenLinks - RhinosF1 the build passed. [13:11:54] [02RemovePII] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/Jsfpj [13:11:55] [02miraheze/RemovePII] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [13:11:59] [02GlobalNewFiles] 07RhinosF1 closed pull request 03#18: Update continuousIntegration.yml - 13https://git.io/JEl1n [13:12:00] [02miraheze/GlobalNewFiles] 07RhinosF1 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JEl1E [13:12:02] [02miraheze/GlobalNewFiles] 07RhinosF1 031473d96 - Merge pull request #18 from miraheze/RhinosF1-patch-1 [13:12:03] [02GlobalNewFiles] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/JtShO [13:12:05] [02miraheze/GlobalNewFiles] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [13:12:05] miraheze/RemovePII - RhinosF1 the build passed. [13:12:21] [02miraheze/PDFEmbed] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JEl1a [13:12:22] [02miraheze/PDFEmbed] 07RhinosF1 03463fe7b - Update continuousIntegration.yml [13:12:24] [02PDFEmbed] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/JsrE2 [13:12:25] [02PDFEmbed] 07RhinosF1 opened pull request 03#6: Update continuousIntegration.yml - 13https://git.io/JEl1w [13:12:38] miraheze/RottenLinks - RhinosF1 the build passed. [13:12:46] [02miraheze/YouTube] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JEl1M [13:12:47] [02miraheze/YouTube] 07RhinosF1 030820beb - Update continuousIntegration.yml [13:12:49] [02YouTube] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/fNXMh [13:12:50] [02YouTube] 07RhinosF1 opened pull request 03#12: Update continuousIntegration.yml - 13https://git.io/JEl1y [13:12:53] miraheze/GlobalNewFiles - RhinosF1 the build passed. [13:12:55] [02PDFEmbed] 07RhinosF1 closed pull request 03#6: Update continuousIntegration.yml - 13https://git.io/JEl1w [13:12:57] [02miraheze/PDFEmbed] 07RhinosF1 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JEl19 [13:12:57] miraheze/GlobalNewFiles - RhinosF1 the build passed. [13:12:58] [02miraheze/PDFEmbed] 07RhinosF1 0397ec549 - Merge pull request #6 from miraheze/RhinosF1-patch-1 [13:13:00] [02miraheze/PDFEmbed] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [13:13:01] [02PDFEmbed] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/JsrE2 [13:13:15] [02miraheze/SpriteSheet] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JEl1N [13:13:17] [02miraheze/SpriteSheet] 07RhinosF1 03c9c986e - Update continuousIntegration.yml [13:13:19] [02SpriteSheet] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/JTPvX [13:13:20] [02SpriteSheet] 07RhinosF1 opened pull request 03#6: Update continuousIntegration.yml - 13https://git.io/JEl1A [13:13:22] miraheze/PDFEmbed - RhinosF1 the build passed. [13:13:42] [02miraheze/CustomHeader] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JElMk [13:13:43] [02miraheze/CustomHeader] 07RhinosF1 0328359a3 - Update continuousIntegration.yml [13:13:45] [02CustomHeader] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/JsrEY [13:13:46] [02CustomHeader] 07RhinosF1 opened pull request 03#6: Update continuousIntegration.yml - 13https://git.io/JElMt [13:13:48] miraheze/YouTube - RhinosF1 the build passed. [13:13:50] [02YouTube] 07RhinosF1 closed pull request 03#12: Update continuousIntegration.yml - 13https://git.io/JEl1y [13:13:52] [02miraheze/YouTube] 07RhinosF1 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JElMY [13:13:53] [02miraheze/YouTube] 07RhinosF1 03cad9390 - Merge pull request #12 from miraheze/RhinosF1-patch-1 [13:13:55] [02YouTube] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/fNXMh [13:13:56] miraheze/PDFEmbed - RhinosF1 the build passed. [13:13:56] [02miraheze/YouTube] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [13:13:58] [02SpriteSheet] 07RhinosF1 closed pull request 03#6: Update continuousIntegration.yml - 13https://git.io/JEl1A [13:13:59] [02miraheze/SpriteSheet] 07RhinosF1 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JElMG [13:14:01] [02miraheze/SpriteSheet] 07RhinosF1 031463ebd - Merge pull request #6 from miraheze/RhinosF1-patch-1 [13:14:02] [02SpriteSheet] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/JTPvX [13:14:03] miraheze/RemovePII - RhinosF1 the build passed. [13:14:04] [02miraheze/SpriteSheet] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [13:14:11] [02CustomHeader] 07RhinosF1 closed pull request 03#6: Update continuousIntegration.yml - 13https://git.io/JElMt [13:14:12] [02miraheze/CustomHeader] 07RhinosF1 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JElMc [13:14:14] [02miraheze/CustomHeader] 07RhinosF1 03873ae69 - Merge pull request #6 from miraheze/RhinosF1-patch-1 [13:14:15] [02CustomHeader] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/JsrEY [13:14:17] [02miraheze/CustomHeader] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [13:14:22] miraheze/SpriteSheet - RhinosF1 the build passed. [13:14:45] miraheze/CustomHeader - RhinosF1 the build passed. [13:14:50] miraheze/YouTube - RhinosF1 the build passed. [13:14:51] Reception123: CI works everywhere [13:15:00] miraheze/SpriteSheet - RhinosF1 the build passed. [13:15:07] .op [13:15:11] miraheze/CustomHeader - RhinosF1 the build passed. [13:15:23] MirahezeBot: load channelmgnt [13:15:24] RhinosF1: Could not load plugin channelmgnt: (wheel 0.37.0 (/srv/sopelbots/prodvenv/lib/python3.7/site-packages), Requirement.parse('wheel<0.37.0,>=0.36.2')) [13:20:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.30, 2.76 [13:21:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.22, 6.46, 5.58 [13:21:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.84, 6.61, 5.75 [13:21:59] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.28, 3.67, 2.96 [13:23:36] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.43, 6.25, 5.63 [13:23:55] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.96, 6.02, 5.64 [13:24:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.23, 3.60, 3.03 [13:25:58] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.61, 3.32, 3.00 [13:34:00] !log [@test3] starting deploy of {'config': True} to skip [13:34:01] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [13:34:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:34:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:57:43] paladox: thanks for the info [14:15:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.32, 2.90 [14:17:01] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.31, 3.06, 2.86 [14:24:24] Hi, I am getting a spammer on socdemwiki. https://socdemwiki.miraheze.org/wiki/Special:Log/newusers [14:24:26] [url] User creation log - SocDemWiki | socdemwiki.miraheze.org [14:35:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.18, 7.03, 5.14 [14:36:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.93, 6.75, 5.40 [14:36:33] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.36, 6.18, 4.89 [14:37:51] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.88, 6.06, 5.01 [14:38:26] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.04, 5.81, 5.23 [14:38:36] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.11, 5.57, 4.84 [14:42:14] GR (he/him) [SocDemWiki]: Not all of them are spam though, Like Agent Isai, Ugochimobi, TheBurningPrincess [15:02:43] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.21, 3.43, 2.97 [15:04:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.50, 3.05 [15:08:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.34, 2.72, 2.85 [15:24:20] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.50, 3.12 [15:30:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.56, 3.28 [15:32:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.71, 3.37, 3.25 [15:36:10] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.14, 6.69, 5.32 [15:36:11] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.51, 6.55, 5.39 [15:38:03] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.44, 3.71, 3.40 [15:38:09] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.44, 5.76, 5.24 [15:38:10] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.69, 5.88, 5.18 [15:40:00] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.46, 3.15, 3.22 [15:47:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.52, 3.37 [15:49:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.61, 3.78, 3.47 [15:53:52] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.41, 3.34, 3.40 [15:55:33] not all, ofc [15:55:42] but i mean the ones that had to be deleted too [16:05:33] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.13, 7.08, 6.09 [16:05:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.46, 3.33 [16:06:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.27, 7.95, 6.26 [16:09:28] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.74, 6.61, 6.08 [16:09:51] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.99, 3.02, 3.21 [16:10:28] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.59, 6.31, 5.98 [16:20:56] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JE801 [16:20:58] [02miraheze/puppet] 07JohnFLewis 039a12843 - icinga: disable notifications for johnflewis [16:48:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.96, 3.52, 3.17 [16:51:09] paladox: just to make sure, is https://meta.miraheze.org/wiki/Tech:Puppet#Adding_a_new_puppet_agent_(server)_to_the_Puppetserver still accurate for re-puppetising servers? [16:51:10] [url] Tech:Puppet - Miraheze Meta | meta.miraheze.org [16:51:15] along with https://meta.miraheze.org/wiki/Tech:Server_lifecycle#Reimage of course [16:51:16] [url] Tech:Server lifecycle - Miraheze Meta | meta.miraheze.org [16:52:10] Reception123: Yes [16:53:09] JohnLewis: ok, thanks! was just double checking [16:54:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.09, 3.34, 3.22 [16:58:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.86, 3.47, 3.29 [17:00:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.53, 3.32 [17:02:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.49, 3.16, 3.21 [17:08:08] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=osaindexwiki /home/reception/osaimages --search-recursively (END - exit=0) [17:08:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:11:45] Reception123: yes [17:11:54] thanks :) [17:16:58] paladox: how would the reimage actually work on Promox? Would I click the reset button and see from there? though where does the ISO file come from? [17:33:09] You need to download the dvd image on the cloud server. You save the iso to cd /var/lib/vz/template/iso [17:33:25] You then change the image via the proxmox ui and then change the boot order [17:33:59] https://cdimage.debian.org/debian-cd/current/amd64/iso-dvd/ [17:34:00] [url] Index of /debian-cd/current/amd64/iso-dvd | cdimage.debian.org [17:34:02] Iso is in ^ [17:34:06] Reception123: ^ [17:34:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.75, 6.69, 5.69 [17:35:11] paladox: ah ok, but where would I change the image in the promox UI? I only see Shutdown and Migrate [17:35:24] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.61, 6.81, 5.54 [17:35:48] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.71, 6.15, 5.30 [17:35:49] There should be an option somewhere [17:35:51] To do it [17:35:57] I just forgot what it’s called [17:36:59] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.05, 6.40, 5.71 [17:37:23] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.57, 6.59, 5.61 [17:37:45] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.75, 5.58, 5.19 [17:40:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.63, 3.12 [17:43:57] ^ JohnLewis do you have any idea what the reimage option is called? [17:44:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.22, 3.40, 3.13 [17:44:03] I can't seem to find where it is [17:44:46] Can you take a pic of the screen? [17:44:55] (Options for the vm) [17:45:00] Reception123: ^ [17:45:17] https://usercontent.irccloud-cdn.com/file/h1MrdWYz/image.png [17:45:52] Reception123: hardware -> CD/DVD -> ISO Image, then reboot should take you to installer [17:46:15] Aha [17:46:20] Thanks JohnLewis ! [17:46:57] ah I see, so I go to CD/DVD and then after that Edit -> Use CD/DVD disc image file (iso) and then I select the file? [17:47:20] Yeah [17:47:58] ok, thanks! [17:56:56] Reception123: https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/IUT256OSFXT6I5QEMJT6JMUE7RIJ6JAC/ [17:56:56] [url] MediaWiki 1.37-alpha will be branched as a beta on 14 September 2021 - Wikitech-l - lists.wikimedia.org | lists.wikimedia.org [18:01:46] ack [18:02:25] @Reception123 You free to read a DM for a moment? [18:02:31] sure [18:03:35] Reception123: did you start the reimage proccess? [18:03:50] RhinosF1: not yet, will check homedirs now and then begin [18:04:01] Ok [18:04:57] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.06, 6.11, 5.28 [18:06:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.31, 3.87, 3.36 [18:06:57] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.42, 5.41, 5.13 [18:08:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.80, 3.40 [18:08:40] Reception123: remember to merge the puppet pr before first run [18:09:35] I can merge it now actually since I'll definitely start before :28 [18:10:30] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 3.88, 3.47 [18:11:35] [02puppet] 07Reception123 closed pull request 03#1888: test3: update php for bullseye testing - 13https://git.io/JElgf [18:11:36] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JE8H9 [18:11:38] [02miraheze/puppet] 07RhinosF1 0322f5a9f - test3: update php for bullseye testing (#1888) [18:12:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.05, 3.18, 3.26 [18:13:48] !log downtime test3 while Reception123 reimage s [18:13:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:15:33] I set it so there's 1 hour to start then 2 hours from DOWN [18:15:40] thanks [18:16:30] I've copied the homedirs to my local PC so I'll just add them back once it's reimaged, easier that way so I don't need to ask everyone if they still need stuff [18:17:04] !log removed test3 from puppet3 [18:17:08] RhinosF1: https://grafana.miraheze.org/alerting/grafana/1tO-Rm47z/view?returnTo=%2Falerting%2Flist [18:17:09] [url] Grafana | grafana.miraheze.org [18:17:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:17:10] Let's go [18:17:38] JohnLewis: it's alerting now! [18:18:09] Also where does said alert go? [18:18:17] Normal icinga + Irc [18:18:26] Email [18:19:06] JohnLewis: I see the email but it uses labels.instance rather than the instance name [18:19:43] No instance fired the alert [18:20:11] JohnLewis: an email came [18:20:16] JohnLewis: hmm I did that (CD/DVD and then after that Edit -> Use CD/DVD disc image file (iso)) and then clicked reset but it didn't change anything [18:20:21] Yes, but no instance caused it [18:20:29] Reception123: reset? [18:20:39] https://usercontent.irccloud-cdn.com/file/vzIoTBzf/image.png [18:21:27] Reception123: John said reboot should take you to the installer [18:21:44] I tried reboot as well and then connected to the old test3 and everything is as if I did nothing [18:21:51] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.17, 6.30, 5.62 [18:22:06] Reception123: Options -> Boot order [18:22:27] It is set to boot the hard drive, then the CD, it should boot the CD first [18:22:33] JohnLewis: ah, then I'd add the ide2 before scsi0 [18:22:45] Yeah [18:24:47] ah I just realised, how would I get the root password to actually access the new server? [18:25:05] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.90, 6.38, 5.70 [18:25:09] You set the root password [18:25:43] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.21, 7.24, 6.18 [18:25:43] but how would I access the server to do that? [18:25:57] Proxmox console [18:26:26] JohnLewis: that's what I thought too, but when accessing it I get "Failed to run vncproxy" and "Failed to connect to server" so I imagined there was another way [18:26:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.39, 3.49, 3.25 [18:26:45] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.24, 6.18, 5.35 [18:27:04] You need to be on the same cloud server in proxmox as the server [18:27:04] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.35, 6.21, 5.73 [18:27:21] So if it’s on cloud5, you need to be logged into proxmox on cloud5 [18:27:24] ah, I see. I guess I was on cloud4 and not 5 [18:27:27] thanks! [18:28:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.47, 3.07, 3.12 [18:28:43] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.41, 5.93, 5.36 [18:29:34] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.68, 6.77, 6.29 [18:35:20] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.88, 8.34, 7.02 [18:35:28] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.98, 7.82, 6.69 [18:36:35] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.00, 6.90, 6.11 [18:37:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.22, 7.21, 6.45 [18:37:16] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.61, 7.49, 6.85 [18:37:27] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.11, 6.81, 6.44 [18:38:32] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.54, 6.58, 6.09 [18:39:04] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.05, 6.37, 6.23 [18:39:25] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.75, 6.60, 6.42 [18:39:41] Reception123: are you alive [18:39:48] yup [18:39:52] still installing [18:41:06] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.36, 6.30, 6.52 [18:43:45] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.87, 20.04, 17.96 [18:45:42] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.86, 19.75, 18.12 [18:47:27] huh not sure why after all that I got directed back to the install menu [18:47:35] might be the boot order, I'll move that back [18:51:26] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.45, 6.94, 6.43 [18:55:25] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.06, 7.19, 6.62 [18:56:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.64, 6.96, 6.25 [18:56:57] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.07, 8.52, 6.77 [18:57:24] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.41, 7.26, 6.71 [18:57:32] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.77, 7.57, 6.55 [18:58:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.02, 7.43, 6.52 [18:59:22] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.49, 6.64, 6.55 [18:59:30] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.84, 7.62, 6.69 [19:00:09] JohnLewis: not sure why when I finish the install it just takes me back to the initial install screen... [19:00:28] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.53, 7.32, 6.58 [19:00:37] I moved the boot order back to hard drive first [19:00:52] should I maybe try a reboot again? [19:00:56] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.22, 7.87, 6.95 [19:01:11] Reception123: did you reboot after changing boot order? [19:01:51] ah, that might be it, I may have not [19:02:29] Reception123: do I need to say much? [19:02:37] ah I see, I thought I did but there was actually an error [19:02:44] "Error: VM quit/powerdown failed" [19:02:56] so when I thought I rebooted I actually didn't because it didn't work for some odd reason [19:03:02] it just failed now too.. [19:03:29] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.04, 7.47, 6.82 [19:03:48] I know why [19:04:02] paladox: oh, really? It doesn't seem to want to reboot at all [19:04:08] When your in the installer you will see the setting button in the vm screen to the left [19:04:10] Click on it [19:04:18] Then click the power button and click shutdown [19:04:49] oh so I should shutdown instead of reboot then [19:04:54] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.61, 7.73, 7.05 [19:05:02] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.15, 7.80, 6.57 [19:05:21] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.98, 7.93, 7.06 [19:05:26] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.84, 8.49, 7.23 [19:05:28] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.68, 6.77, 6.65 [19:06:26] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.17, 6.54, 6.52 [19:06:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.01, 7.04, 6.88 [19:07:19] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.85, 7.47, 6.99 [19:07:23] paladox: huh, we've got a new error "Error: can't lock file '/var/lock/qemu-server/lock-131.conf' - got timeout [19:07:32] Well you have to do a workaround to get the vm to reboot when you are in the installer [19:07:50] Don’t click the shutdown button from the drop down outside the cm screen [19:07:58] You have to click it within the vm screen [19:08:53] https://usercontent.irccloud-cdn.com/file/cAadLkVt/image.png [19:08:53] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.24, 6.72, 6.78 [19:08:58] don't you mean the one here to the left? [19:08:58] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.01, 7.17, 6.62 [19:09:16] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.42, 7.76, 7.20 [19:09:43] Yup [19:10:01] that's what I did, the power button on the left of the screen and it gave me that timeout error [19:10:58] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.31, 6.47, 6.43 [19:11:54] Reception123: what does lsof /var/lock/qemu-server/lock-.conf say from the cloud server [19:11:54] Take a pic when you click on the power button within the setting button [19:12:25] https://usercontent.irccloud-cdn.com/file/cK9Ajci0/image.png [19:13:15] Click stop [19:13:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.71, 6.47, 6.72 [19:13:20] RhinosF1: I checked and it was empty so not helpful [19:13:24] paladox: ok [19:13:37] ah that did it, so it had to be stop and not shutdown [19:13:44] I wasn't sure if doing that would do the reboot properly [19:14:02] well let's hope the install works now as the order is changed, otherwise I won't have any idea why it wouldn't [19:14:40] Make sure to start it up again :) [19:15:04] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.92, 6.27, 6.74 [19:15:42] hmm now weirdly it's giving me a graphical screen. Is that normal? [19:16:32] and it doesn't seem to like the password either, great [19:16:42] What did you select [19:16:49] the regular install option [19:16:54] Did you unselect graphics? [19:17:02] Did you also select ssh? [19:17:12] I didn't see an option to unselect graphics [19:17:29] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.38, 6.41, 6.20 [19:17:30] Though clearly I missed something then so I guess I'll have to redo. Will a new install override the older one? [19:17:50] I'm guessing it will but just to be sure [19:18:26] Yeh [19:18:32] Reinstalling will erase everything [19:18:35] It’ll be like new [19:18:48] ok, I guess I'll redo it again and watch out for any mention of graphics [19:18:53] SSH I did tick to install [19:19:31] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.75, 6.44, 6.24 [19:21:10] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.37, 7.01, 6.79 [19:22:54] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.03, 7.50, 6.62 [19:23:50] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.76, 8.34, 7.37 [19:24:13] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.83, 7.51, 6.66 [19:24:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.92, 7.41, 6.72 [19:24:55] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.77, 6.89, 6.51 [19:25:08] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.33, 6.18, 6.57 [19:25:28] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.72, 8.43, 7.22 [19:25:48] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.44, 7.75, 7.25 [19:26:42] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.56, 3.88, 3.29 [19:26:55] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.25, 8.29, 7.13 [19:26:55] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.27, 6.59, 6.44 [19:28:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.37, 3.73, 3.32 [19:29:37] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.65, 7.52, 7.22 [19:30:37] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.16, 3.25, 3.19 [19:30:56] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.72, 6.82, 6.84 [19:31:28] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.02, 7.32, 7.20 [19:31:34] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.03, 6.63, 6.94 [19:31:35] paladox: well I redid the install and clicked no when it asked for graphics but now I'm back at a graphic log in screen [19:32:57] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.11, 6.67, 6.79 [19:33:30] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.32, 7.77, 7.30 [19:33:44] and either way there's a Connection failed indicator [19:34:01] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.70, 6.23, 6.44 [19:34:34] Connection failed indicator? [19:34:49] for 'graphics' you installed the Desktop environment [19:34:57] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.62, 8.12, 6.99 [19:35:04] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.51, 7.19, 6.65 [19:35:23] JohnLewis: not sure how I would've installed the desktop as when it asked for graphics I clicked no [19:35:49] It never gives a Yes/No for Desktop environment [19:36:00] ah, then how can you avoid it? [19:36:01] Desktop is automatically selected, you have to disable it [19:36:06] oh [19:36:20] for some reason I don't remember that the last time I installed a server like this which I think was in 2018 or 2019 [19:36:49] JohnLewis: I searched it and it says 'sudo systemctl set-default multi-user.target' would do it, does that sound right to you? [19:36:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.63, 7.42, 6.87 [19:37:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.41, 6.77, 6.55 [19:37:20] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.66, 7.37, 7.34 [19:37:51] Fresh install, because it would install any number of additional packages [19:38:30] oh, that's what you mean. I don't remember seeing a desktop environment option but I'll pay extra attention for that then [19:40:56] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.99, 6.39, 6.60 [19:41:28] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.23, 5.95, 6.65 [19:43:08] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.53, 6.13, 6.75 [19:44:09] ah there you go, when you mentioned it wasn't Yes/No I imagined it must've been in the same place where you select SSH server [19:47:29] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.32, 7.79, 7.15 [19:48:54] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.55, 22.09, 19.07 [19:49:29] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.16, 7.51, 7.13 [19:50:51] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.56, 20.85, 18.97 [19:51:29] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.28, 7.88, 7.31 [19:51:43] hmm I'm sure I unchecked the desktop environment this time but here we are again on the graphic screen [19:52:47] JohnLewis, paladox: can you maybe screenshare to help Reception123 [19:52:49] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.08, 18.62, 18.38 [19:53:10] Reception123: you left the GNOME desktop enabled [19:53:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.70, 3.29 [19:53:21] Disable everything except standard and SSH server [19:53:27] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.95, 7.07, 7.08 [19:53:47] argh, will do [19:54:00] good thing we have a test server otherwise this wouldn't be fun downtime [19:55:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.79, 3.54, 3.29 [19:55:29] I mean we should be able to afford 1 MW down [19:57:16] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.21, 3.34, 3.24 [19:57:27] yeah in theory though not ideal [19:57:28] Reception123: do remember ideally we only have ~4 weeks to 1.37 prep [19:57:37] yeah [19:57:44] Because we said last time we'd try rc's [19:57:54] So we do need to be confident [19:58:23] Even if it's not fully rolled out [19:58:31] That we won't have issues [19:58:46] JohnLewis: what about standard system utilities? are those fine to keep? [19:59:03] and yeah r.e. 1.37 [19:59:27] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.18, 6.18, 6.68 [20:06:32] Reception123: yes [20:06:49] thanks [20:08:28] JohnLewis: now the next roadblock is network. When trying to wget something it says it's unable to resolve the host address [20:09:13] Have you set up networking? [20:10:05] yeah that's the thing, during install it did appear to be a problem and didn't want to work (I've done the install so many times and ever single one it gave errors and said it couldn't be set up) [20:10:22] though I wanted to make sure at least I have everything else right and then go back to network [20:10:35] you can't do networking during install [20:10:42] You need to do it post install [20:10:58] I see, though what would I have to set up now? [20:11:15] Tech:Proxmox, its all documented there [20:11:47] oh ok, I didn't know there were extra docs there [20:11:49] will take a look [20:14:38] JohnLewis: but the docs do suggest there is some network setup that has to be done during the install phase [20:14:54] since it says " go through the installation, when you get onto the network step enter the following" [20:16:37] Reception123: also don't forget the format change for Debian security [20:16:46] As them docs are wrong for the deb list [20:16:53] yeah [20:16:59] the docs say buster of coures [20:17:01] *course [20:17:13] Yes and that [20:17:36] You can do part of the networking in the install, but you haven't. It doesn't change the docs as they are written solely for post-install anyway [20:18:32] I see but where would I input things like the failover IP and netmask then? [20:19:32] Hm? [20:22:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.94, 3.45, 2.99 [20:22:22] Reception123: where the docs say [20:22:25] well some of the docs seem to be written for the part during the install [20:22:42] Which is why you skip to the part that says after install [20:23:04] ^ [20:23:40] I see [20:24:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.92, 3.53, 3.07 [20:25:35] the docs say "empty /etc/network/interfaces, insert the following in it" but I'm not sure what 'following' it's talking about [20:26:07] oh it's only lower down under Run puppet, that's a bit confusing [20:26:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 3.16, 3.00 [20:26:09] the block of code that has inet in it [20:30:38] not entirely sure where I'd find the failover IP [20:30:50] That's the IP of the VM [20:31:04] oh ok [20:31:10] johnflewis@cloud5:~$ host test3.miraheze.org [20:31:10] test3.miraheze.org has address 51.195.236.247 [20:31:10] test3.miraheze.org has IPv6 address 2001:41d0:800:1bbd::14 [20:31:43] thanks! [20:34:30] got the config done, now my understanding is I need to run ifdown ens18 && ifup ens18 but when I do that it says interface ens18 not configured [20:36:42] nevermind, that seems to have worked now [20:37:22] though wget still comes up with the same error [20:43:36] Have you done the resolv.conf config? [20:44:45] yeah [20:46:13] Post the config files please [20:49:25] https://www.irccloud.com/pastebin/ZIpaSlyU/ [20:49:27] ^ JohnLewis [20:50:29] .242? That’s not the IP for test3 [20:51:19] whoops fixed that [20:51:49] though now the network service won't start because RTNETLINK answers: File exists [20:51:59] and ifup: failed to bring up ens18 [20:52:00] Did you ifdown? [20:52:11] yeah [20:52:12] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.80, 1.69, 1.41 [20:52:30] that gives ens18 not configured [20:52:45] sudo ip addr flush dev ens18 [20:52:52] Then ifup it [20:53:53] done, finally works now! [20:53:55] thanks for the help [20:54:13] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.22, 1.63, 1.43 [20:58:34] So it all works now? [20:59:04] things seem to be working so far yeah [21:04:24] PROBLEM - wiki.usagihime.ml - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.usagihime.ml could not be found [21:05:31] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.06, 6.06, 5.00 [21:06:09] PROBLEM - wiki.usagihime.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:06:26] I definitely didn't expect this would take so long. As far as I've seen everything seems fine on test3 now though I'm quite tired so I'll be back tomorrow morning to make sure that everything is completely fine (since test3 isn't prod I don't think it would be too terrible if there were any minor issues) [21:07:31] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.13, 5.39, 4.89 [21:11:30] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.69, 7.09, 5.52 [21:13:28] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.59, 6.85, 5.63 [21:15:27] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.58, 7.38, 5.96 [21:16:17] !log [@test3] starting deploy of {'config': True} to skip [21:16:18] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [21:16:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:16:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:17:29] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.51, 7.49, 6.18 [21:17:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.20, 6.92, 5.53 [21:18:59] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.22, 1.96, 1.54 [21:19:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.88, 6.19, 5.43 [21:20:59] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.70, 1.82, 1.54 [21:22:18] PROBLEM - test3 MediaWiki Rendering on test3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 251 bytes in 0.135 second response time [21:22:34] PROBLEM - test3 Puppet on test3 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file [21:22:34] PROBLEM - test3 Check Gluster Clients on test3 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [21:22:53] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.54, 1.68, 1.52 [21:23:11] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.69, 6.94, 5.92 [21:23:39] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.70, 3.21 [21:25:10] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.35, 6.76, 5.98 [21:25:34] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.82, 3.24, 3.09 [21:26:01] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.35, 6.35, 6.23 [21:30:32] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.86, 6.32, 5.67 [21:34:28] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.43, 9.03, 6.86 [21:35:40] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.00, 6.94, 6.11 [21:35:50] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.81, 7.41, 6.17 [21:36:13] RECOVERY - test3 Check Gluster Clients on test3 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [21:37:38] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.27, 6.57, 6.06 [21:37:49] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.69, 6.69, 6.05 [21:38:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.61, 7.99, 6.95 [21:42:21] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.30, 6.62, 6.63 [21:55:17] Reception123: looks to just need composer + sync [22:05:38] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.93, 6.30, 5.77 [22:07:38] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.76, 5.68, 5.61 [22:35:53] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.49, 6.91, 6.03 [22:37:52] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.66, 6.15, 5.86 [22:41:06] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.10, 1.71, 1.41 [22:43:01] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.39, 1.53, 1.38 [23:03:04] PROBLEM - test3 MediaWiki Rendering on test3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 251 bytes in 0.014 second response time [23:05:38] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.67, 6.51, 5.39 [23:07:40] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.16, 5.90, 5.30 [23:22:50] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.72, 1.66, 1.34 [23:24:51] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.65, 1.25, 1.23