[00:03:42] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.99, 23.02, 20.75 [00:07:10] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.92, 6.46, 5.68 [00:09:06] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.58, 5.79, 5.53 [00:11:07] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.20, 7.10, 5.61 [00:19:19] PROBLEM - madgenderscience.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:43:37] PROBLEM - madgenderscience.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - madgenderscience.wiki reverse DNS resolves to a51062ecadbb5a26e.awsglobalaccelerator.com [00:47:52] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.14, 1.56, 1.09 [00:49:53] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.03, 1.31, 1.05 [01:01:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.45, 3.09 [01:06:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.13, 3.22, 3.09 [01:06:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.87, 7.36, 5.31 [01:06:29] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.51, 7.61, 5.76 [01:08:16] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.25, 7.32, 5.58 [01:08:26] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.88, 6.91, 5.73 [01:08:35] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.32, 7.92, 5.69 [01:10:12] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.19, 6.09, 5.33 [01:10:21] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.82, 6.03, 5.53 [01:10:29] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.14, 7.11, 5.66 [01:12:26] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.70, 6.32, 5.54 [01:22:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.44, 3.16 [01:24:25] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.26, 3.06, 3.06 [01:34:39] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.17, 3.55, 3.24 [01:36:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.98, 3.65, 3.31 [01:38:44] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.76, 3.25, 3.20 [01:40:06] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.12, 22.41, 23.75 [01:54:11] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.86, 23.11, 23.02 [02:00:16] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.49, 23.57, 23.52 [02:04:21] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.70, 27.21, 24.90 [02:06:23] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.33, 6.70, 5.28 [02:08:18] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.34, 5.93, 5.16 [02:12:17] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.04, 22.86, 23.99 [02:20:34] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.34, 21.83, 22.54 [02:21:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.10, 4.96, 5.81 [02:23:00] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.95, 5.99, 6.08 [02:24:36] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.55, 23.20, 23.07 [02:25:01] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.18, 5.51, 5.89 [02:26:18] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.70, 6.47, 5.48 [02:27:00] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.43, 5.92, 6.00 [02:28:14] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.66, 6.46, 5.59 [02:30:37] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.69, 24.31, 23.41 [02:36:17] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.09, 7.16, 5.54 [02:36:36] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.20, 7.01, 5.54 [02:37:04] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.98, 3.33, 3.03 [02:38:12] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.16, 6.61, 5.55 [02:38:32] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.08, 6.70, 5.61 [02:39:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.21, 3.26, 3.04 [02:42:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.35, 23.69, 23.89 [02:44:53] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.11, 23.36, 23.70 [02:46:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 24.00, 23.89, 23.87 [02:50:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.91, 3.96, 3.33 [02:52:30] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.86, 3.44, 3.21 [02:54:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.86, 3.20, 3.15 [02:55:15] PROBLEM - cp15 Current Load on cp15 is WARNING: WARNING - load average: 1.42, 1.92, 1.35 [02:56:52] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.27, 5.42, 5.88 [02:57:11] RECOVERY - cp15 Current Load on cp15 is OK: OK - load average: 0.52, 1.45, 1.24 [03:00:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.16, 3.51, 3.26 [03:02:46] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.40, 3.07, 3.13 [03:06:32] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.86, 9.24, 7.20 [03:08:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.58, 7.41, 6.77 [03:10:24] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.57, 6.47, 6.50 [03:11:07] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.71, 18.41, 20.23 [03:12:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.39, 3.20 [03:14:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.99, 3.16, 3.13 [03:14:58] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.04, 4.14, 4.99 [03:22:11] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.16, 6.02, 5.47 [03:30:06] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.98, 6.00, 5.74 [03:32:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.71, 3.41 [03:34:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.73, 5.86, 5.69 [03:35:29] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.53, 20.64, 19.99 [03:36:19] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.60, 7.65, 5.90 [03:36:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.05, 3.54, 3.43 [03:38:16] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.00, 6.73, 5.77 [03:38:38] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.26, 3.35, 3.37 [03:40:11] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.67, 5.99, 5.88 [03:47:24] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.55, 19.94, 19.97 [03:48:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.39, 3.31 [03:50:44] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.62, 3.41 [03:51:25] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.52, 21.54, 20.69 [03:52:44] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.29, 3.39, 3.35 [03:55:25] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.21, 19.40, 20.10 [03:56:04] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.38, 5.55, 5.53 [03:56:44] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.90, 3.75, 3.50 [03:58:46] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.12, 3.16, 3.31 [03:59:31] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.12, 21.64, 20.90 [04:01:57] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.99, 5.65, 5.59 [04:03:57] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.01, 5.71, 5.62 [04:04:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 3.43, 3.37 [04:05:35] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.09, 23.05, 21.47 [04:06:06] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.09, 5.26, 5.48 [04:07:34] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.98, 20.85, 20.88 [04:08:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.42, 3.05, 3.24 [04:09:34] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.77, 18.89, 20.14 [04:10:01] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.33, 5.15, 5.34 [04:15:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.70, 5.27, 5.35 [04:19:50] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.08, 5.28, 5.31 [04:21:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.62, 5.15, 5.24 [04:23:47] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.70, 5.69, 5.43 [04:25:45] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.43, 5.87, 5.55 [04:27:29] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.56, 21.12, 20.31 [04:27:44] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.28, 6.31, 5.76 [04:31:27] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.74, 19.98, 20.06 [04:31:41] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.10, 4.97, 5.37 [04:33:41] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.94, 4.13, 5.03 [04:38:26] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.96, 7.31, 5.61 [04:40:22] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.34, 6.05, 5.36 [05:17:00] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JrF7a [05:17:01] [02miraheze/dns] 07Reception123 03331f365 - add zamlung.tk zone [05:31:29] PROBLEM - cp13 Puppet on cp13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [05:33:09] [02miraheze/ssl] 07Reception123 pushed 038 commits to 03master [+1/-6/±13] 13https://git.io/JrbGx [05:33:10] [02miraheze/ssl] 07Reception123 035ef85fd - remove viileapedia.ga cert (no longer pointing) [05:33:11] [02miraheze/ssl] 07Reception123 0340df6e4 - remove wiki.mnzp.xyz cert (no longer pointing) [05:33:13] [02miraheze/ssl] 07Reception123 03644e08f - remove monarchia.wiki cert (no longer pointing) [05:33:14] [02miraheze/ssl] ... and 5 more commits. [05:36:05] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.20, 7.52, 5.60 [05:36:30] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.87, 8.21, 6.05 [05:37:15] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [05:37:16] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [05:37:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.00, 18.35, 15.24 [05:38:03] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.76, 3.97, 3.34 [05:38:05] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.36, 6.38, 5.42 [05:38:15] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [05:38:24] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.88, 6.97, 5.86 [05:38:48] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [05:38:49] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [05:38:58] PROBLEM - mw13 Puppet on mw13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [05:39:30] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [05:39:36] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.93, 16.68, 14.98 [05:40:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.57, 3.27 [05:40:20] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.05, 6.61, 5.85 [05:41:17] PROBLEM - cp14 Puppet on cp14 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [05:42:17] PROBLEM - cp15 Puppet on cp15 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [05:43:29] PROBLEM - mwtask1 Puppet on mwtask1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.lukaibrisimovic.org] [05:46:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.54, 3.20, 3.22 [05:52:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.57, 3.34, 3.25 [05:52:34] PROBLEM - zamlung.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate '*.miraheze.org' expires in 14 day(s) (Sat 23 Oct 2021 23:59:59 GMT +0000). [05:53:18] PROBLEM - test4 Puppet on test4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.lukaibrisimovic.org] [05:54:12] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.09, 3.34, 3.26 [05:55:54] PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.lukaibrisimovic.org] [06:05:56] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.58, 6.80, 5.39 [06:07:51] RECOVERY - jobchron1 APT on jobchron1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:07:52] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.13, 5.76, 5.19 [06:36:57] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.31, 6.50, 5.09 [06:38:53] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.96, 5.58, 4.92 [07:00:34] RECOVERY - test4 APT on test4 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [07:06:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 12.11, 7.15, 5.38 [07:08:29] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.63, 5.94, 5.14 [07:21:30] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jrxe6 [07:21:31] [02miraheze/ssl] 07Reception123 034a731d8 - fix [07:23:36] paladox: I'm still a bit confused about https://phabricator.miraheze.org/T8097 - would I be able to renew that with my own account or do I need access to the registrar? [07:23:37] [url] ⚓ T8097 miraheze.org certificate needs renewing | phabricator.miraheze.org [07:35:26] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:35:35] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:36:09] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:36:33] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [07:36:52] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:37:53] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:37:54] RECOVERY - mabi.myomi.net - LetsEncrypt on sslhost is OK: OK - Certificate 'mabi.myomi.net' will expire on Fri 07 Jan 2022 04:29:15 GMT +0000. [07:38:07] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:20] RECOVERY - thepolicyhub.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thepolicyhub.org.uk' will expire on Fri 07 Jan 2022 04:31:02 GMT +0000. [07:38:32] RECOVERY - cp14 Puppet on cp14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:45] RECOVERY - wiki.misslessfighters.club - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.misslessfighters.club' will expire on Fri 07 Jan 2022 04:27:02 GMT +0000. [07:39:33] RECOVERY - cp15 Puppet on cp15 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:40:09] RECOVERY - wiki.chouverse.cludisciples.cyou - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.chouverse.cludisciples.cyou' will expire on Fri 07 Jan 2022 04:27:49 GMT +0000. [07:40:37] RECOVERY - wiki.homiedb.party - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.homiedb.party' will expire on Fri 07 Jan 2022 04:28:35 GMT +0000. [07:42:08] RECOVERY - mwtask1 Puppet on mwtask1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:43:36] RECOVERY - wiki.consid.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.consid.vn' will expire on Fri 07 Jan 2022 04:30:19 GMT +0000. [07:44:06] RECOVERY - zamlung.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'zamlung.tk' will expire on Fri 07 Jan 2022 04:31:55 GMT +0000. [07:49:41] RECOVERY - test4 Puppet on test4 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [07:53:33] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:58:47] RECOVERY - cp13 Puppet on cp13 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [08:56:18] PROBLEM - steamdecklinux.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'steamdecklinux.wiki' expires in 15 day(s) (Mon 25 Oct 2021 08:49:47 GMT +0000). [09:05:04] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.01, 6.85, 4.89 [09:06:36] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 13.38, 7.79, 5.22 [09:07:00] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.82, 6.66, 5.10 [09:08:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.96, 6.34, 4.99 [09:36:05] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 13.12, 7.98, 5.47 [09:36:29] Reception123: any idea with https://phabricator.miraheze.org/T8127 [09:36:30] [url] ⚓ T8127 Domain not working | phabricator.miraheze.org [09:37:02] PROBLEM - mw10 APT on mw10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:38:58] RECOVERY - mw10 APT on mw10 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [09:40:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.05, 7.70, 6.07 [09:41:59] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.21, 6.57, 5.85 [09:47:10] @Reception123 I think I've seen something! In regards to that duplicated extendedconfirmed usergroup, (I think) On our LocalSettings.php, it is supposed to be in small letters but it's in capital letters. '+famepediawiki' => [ 'Extendedconfirmed' => [ 'editextendedconfirmedprotected' => true, ], 'templateeditor' => [ 'edittemplateprotected' => true, [09:47:10] ], ], This is how it is currently and the E in Extendedconfirmed is supposed to be e Please help configure, thanks. [10:05:57] RhinosF1: nope [10:06:03] @Joseph oh that's a good observation, let me look [10:06:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 16.13, 8.39, 5.75 [10:06:49] Reception123: Alright. [10:07:16] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.34, 5.88, 4.45 [10:08:07] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.10, 6.88, 5.53 [10:08:54] and please add this the LocalSettings.php > wfLoadSkin( 'MinervaNeue' ); > $wgMFDefaultSkinClass = 'SkinMinerva'; Yep. [10:09:12] Or I should make a task? [10:09:13] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.24, 5.12, 4.34 [10:10:06] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 2.55, 5.35, 5.13 [10:10:59] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JoecA [10:11:00] [02miraheze/mw-config] 07Reception123 0370751bc - minor fix [10:12:06] miraheze/mw-config - Reception123 the build passed. [10:22:06] @Joseph: no we can't because ManageWiki to the first but you can turn it on in ManageWiki if it's not already installed. To the latter, we can look. [10:22:56] But please state the problem not always the solution as the ones on mediawiki.org don't always apply [10:25:52] I can not find wfLoadSkin("MinervaNeue") on ManageWiki?? or is it same as the $wgDefaultMobileSkin? [10:26:47] I wanted to set default MobileFrontend's skin [10:27:07] Bur I think it's the $wgDefaultMobileSkin on ManageWiki [10:27:28] for the latter, it's surely not in ManageWiki [10:32:58] PROBLEM - ping6 on bacula2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 112.96 ms [10:33:59] !log [@test3] starting deploy of {'config': True} to skip [10:34:00] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [10:34:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:34:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:35:13] !log [@mw11] starting deploy of {'config': True} to all [10:35:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:35:47] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 33s [10:36:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:37:04] RECOVERY - ping6 on bacula2 is OK: PING OK - Packet loss = 0%, RTA = 72.57 ms [10:37:58] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.19, 7.38, 5.17 [10:39:59] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.04, 6.55, 5.14 [10:43:29] PROBLEM - bacula2 APT on bacula2 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [10:45:32] PROBLEM - bacula2 APT on bacula2 is CRITICAL: APT CRITICAL: 16 packages available for upgrade (2 critical updates). [11:04:53] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.19, 6.59, 5.09 [11:06:52] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.02, 5.97, 5.05 [11:07:04] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.74, 7.49, 5.85 [11:07:58] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.81, 7.10, 6.11 [11:09:00] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.35, 6.35, 5.64 [11:10:00] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.11, 5.95, 5.80 [11:37:41] Reception123: you can setup an account namecheap or with what ever ssl host you want to go with [11:38:01] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 7.94, 8.47, 6.15 [11:38:03] I did the cert using my account on namecheap last year. [11:38:41] paladox: oh ok so just any account will work? [11:38:44] and then I'd just have to generate a CSR? [11:40:00] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.26, 7.45, 6.04 [11:40:38] Yeh, you’ll buy it though it’ll ask you to verify you have control over the domain. [11:42:01] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.38, 6.64, 5.91 [11:47:16] paladox: can we not use last years CSR? [11:48:55] @Joseph: you're focusing on a solution not a problem. Read https://xyproblem.info/ then file a task [11:48:55] [url] The XY Problem | xyproblem.info [12:04:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.08, 6.14, 4.87 [12:06:49] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.69, 5.17, 4.67 [12:07:10] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.59, 6.98, 5.63 [12:07:51] RhinosF1: it’s best we do a new csr each time. I don’t remember where SPF put the csr we did last year tbh either [12:09:08] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.30, 5.62, 5.29 [12:22:19] paladox: ack [12:22:37] paladox: I assume we need a db restart to pick it up? [12:22:47] The rest should be seamless [12:28:32] Also meta.m.miraheze.org is broken [12:28:35] And test3 [12:29:00] PROBLEM - mw9 APT on mw9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:30:58] RECOVERY - mw9 APT on mw9 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [12:41:07] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JoULZ [12:41:08] [02miraheze/dns] 07paladox 03d47f98a - Add acme challenge for m.miraheze.org [12:43:11] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JoUmc [12:43:13] [02miraheze/ssl] 07paladox 039738720 - Update m.miraheze.org.crt [12:45:16] [02puppet] 07paladox commented on pull request 03#1983: csp: add *.wikipedia.org to connect-src - 13https://git.io/JoU3o [12:45:40] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Fri 07 Jan 2022 11:42:07 GMT +0000. [12:45:52] RhinosF1: fixed [12:46:31] paladox: ty [13:05:53] PROBLEM - mw10 APT on mw10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:06:06] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 15.81, 8.54, 5.93 [13:07:51] RECOVERY - mw10 APT on mw10 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [13:08:05] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.16, 7.63, 5.91 [13:10:05] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.10, 6.69, 5.77 [13:27:13] @RhinosF1 @Reception123 If you can remember that task about the Validator Fatal Error on FAMEPedia for Coordinates, I have tried to clear that one with some CSS, everything is working fine for coordinates as it appears on this article [[mh:famepedia:California_State_Capitol]] But when you click on the globe-with-an-arrow near the coordinate It shows This content is blocked. Contact the site owner to fix the issue. I have a [13:27:14] feeling it's a CSP issue, IDK You can check the page out for yourself. [13:27:14] https://famepedia.miraheze.org/wiki/California_State_Capitol [13:27:15] [url] California State Capitol - FAMEPedia | famepedia.miraheze.org [13:34:57] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.76, 6.63, 4.99 [13:34:59] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.55, 6.75, 5.49 [13:37:07] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.84, 6.10, 5.00 [13:37:16] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.63, 18.71, 15.85 [13:37:19] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.12, 7.35, 5.73 [13:37:25] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.96, 10.59, 7.52 [13:39:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.62, 7.48, 5.98 [13:39:15] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.00, 19.29, 16.45 [13:41:11] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.61, 6.94, 5.82 [13:43:05] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.08, 6.37, 5.75 [13:43:09] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.02, 7.82, 6.44 [13:43:20] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.21, 7.36, 7.07 [13:45:03] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.01, 7.55, 6.50 [13:47:11] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.35, 7.79, 7.61 [13:47:21] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.21, 7.39, 7.11 [13:49:20] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.14, 6.97, 7.00 [13:50:49] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.66, 6.05, 6.23 [13:53:16] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.38, 6.16, 6.66 [13:55:02] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.82, 7.11, 6.40 [13:55:11] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.85, 5.68, 6.66 [13:57:01] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.40, 6.69, 6.34 [14:05:24] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.74, 8.08, 6.97 [14:06:21] PROBLEM - mw10 APT on mw10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:06:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.05, 7.64, 7.06 [14:07:04] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.87, 6.52, 6.14 [14:08:23] RECOVERY - mw10 APT on mw10 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [14:08:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.32, 7.06, 6.91 [14:08:58] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.59, 6.50, 6.17 [14:09:14] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.05, 10.38, 7.85 [14:09:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.88, 7.29, 7.02 [14:10:29] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.23, 6.52, 6.74 [14:13:04] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.35, 7.61, 7.26 [14:13:28] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.93, 6.16, 6.63 [14:15:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.62, 3.27, 3.03 [14:17:00] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.90, 5.78, 6.59 [14:17:30] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.75, 3.25, 3.07 [14:21:42] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.49, 4.17, 3.51 [14:23:01] @Joseph: possibly but task please [14:23:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.91, 3.86, 3.49 [14:28:21] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.21, 6.99, 6.45 [14:30:19] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.96, 6.68, 6.41 [14:33:35] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 3.07, 3.29 [14:36:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 17.46, 11.62, 8.35 [14:37:09] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.32, 6.72, 5.89 [14:37:19] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.47, 19.16, 16.79 [14:37:49] PROBLEM - mw11 APT on mw11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:38:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.61, 9.15, 7.20 [14:39:04] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.71, 6.85, 6.05 [14:39:17] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.19, 18.08, 16.70 [14:39:47] RECOVERY - mw11 APT on mw11 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [14:40:57] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.81, 6.52, 6.02 [14:42:27] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.40, 7.23, 6.84 [14:42:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.99, 7.49, 7.60 [14:44:25] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.90, 6.41, 6.58 [14:44:37] paladox: any idea why https://phabricator.miraheze.org/T8127 isn't working? [14:44:38] [url] ⚓ T8127 Domain not working | phabricator.miraheze.org [14:47:41] Nope :/ [14:47:50] it all looks correct for me in the config [14:48:31] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.70, 5.53, 6.67 [14:53:10] JohnLewis: ^ maybe you have an idea? [14:56:52] What does nginx server config say? It sounds like the domain is being served by the Miraheze.org stanza not the wiki one [14:57:53] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.06, 6.71, 5.74 [14:59:54] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.04, 6.15, 5.64 [14:59:57] hmm that's a bit weird, usually nginx says either OK or fail but now when I tried a configtest it's silent [15:00:00] reception@mwtask1:~$ sudo service nginx configtest [15:00:00] Testing nginx configuration:. [15:00:06] I've fixed it [15:00:12] i had to nginx reload on the cache proxies [15:00:19] then had to do it on the application servers [15:00:33] paladox: oh, why was a reload necessary? what happened? [15:01:28] I have no idea, i just ran reload and it started working [15:02:32] hm that's weird [15:06:34] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.79, 7.54, 6.79 [15:08:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.06, 6.78, 6.60 [15:13:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.47, 3.34 [15:17:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 3.25, 3.29 [15:21:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.80, 3.53, 3.39 [15:23:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.72, 3.19, 3.27 [15:36:30] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.54, 10.57, 6.97 [15:37:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 7.19, 8.31, 6.82 [15:37:39] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 6.58, 9.69, 7.14 [15:37:49] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.41, 7.66, 6.37 [15:39:19] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.85, 7.03, 6.53 [15:41:43] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.06, 7.02, 6.64 [15:41:54] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 19.00, 13.70, 8.96 [15:42:24] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.05, 7.67, 6.76 [15:43:28] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.65, 6.41, 6.49 [15:43:39] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.07, 6.20, 6.39 [15:44:21] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.90, 6.41, 6.41 [15:47:49] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.50, 7.23, 7.49 [15:51:48] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.46, 5.04, 6.51 [16:00:00] PROBLEM - ping6 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 133.70 ms [16:00:23] PROBLEM - ping4 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 154.04 ms [16:04:02] RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 93.53 ms [16:04:27] RECOVERY - ping4 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 95.02 ms [16:10:16] !log temporarily directed webmaster@ to my email for domain validation (SSL verif) [16:10:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:19:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.60, 3.73, 3.16 [16:21:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.06, 3.53, 3.16 [16:25:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.55, 3.25, 3.15 [16:27:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.76, 7.08, 5.42 [16:27:33] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.45, 7.41, 5.88 [16:29:10] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.39, 5.97, 5.22 [16:31:28] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.12, 6.44, 5.83 [16:35:30] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.81, 10.30, 7.08 [16:35:31] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.24, 8.38, 6.18 [16:36:02] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.14, 7.80, 6.48 [16:38:19] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.19, 8.16, 6.69 [16:38:21] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.89, 9.78, 7.49 [16:39:45] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.35, 7.98, 6.91 [16:40:20] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.41, 4.19, 3.56 [16:41:40] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.03, 7.84, 6.78 [16:41:42] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.96, 6.69, 6.57 [16:42:21] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.81, 7.88, 7.26 [16:44:06] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.23, 7.13, 6.79 [16:44:16] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.84, 7.10, 7.06 [16:45:21] [02puppet] 07Universal-Omega reviewed pull request 03#1983 commit - 13https://git.io/Joqgo [16:45:34] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.01, 6.12, 6.33 [16:46:15] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.43, 6.30, 6.79 [16:46:17] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.71, 6.17, 6.69 [16:46:23] [02mw-config] 07Universal-Omega edited pull request 03#4131: Profiler changes - 13https://git.io/JwcvG [16:47:57] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.80, 6.34, 6.54 [16:50:37] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/hololivewiki --wiki=initSiteStats.php --active --update (END - exit=256) [16:50:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:51:17] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=hololivewiki --active --update (END - exit=0) [16:51:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:52:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.66, 3.93, 3.87 [17:00:03] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://git.io/Joqda [17:00:05] [02miraheze/mw-config] 07Reception123 0332c6420 - configure wgRightFunctionsUserGroups for quoteunquotecampaignwiki per T8106 [17:00:06] [02mw-config] 07Reception123 created branch 03Reception123-patch-2 - 13https://git.io/vbvb3 [17:00:15] [02mw-config] 07Reception123 opened pull request 03#4136: configure wgRightFunctionsUserGroups for quoteunquotecampaignwiki per… - 13https://git.io/Joqd7 [17:00:17] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.56, 2.85, 3.36 [17:00:29] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://git.io/JoqF3 [17:00:30] [02miraheze/mw-config] 07github-actions 036237328 - CI: lint code to MediaWiki standards [17:00:32] [02mw-config] 07github-actions[bot] synchronize pull request 03#4136: configure wgRightFunctionsUserGroups for quoteunquotecampaignwiki per… - 13https://git.io/Joqd7 [17:01:15] miraheze/mw-config - Reception123 the build passed. [17:01:36] [02mw-config] 07Reception123 closed pull request 03#4136: configure wgRightFunctionsUserGroups for quoteunquotecampaignwiki per… - 13https://git.io/Joqd7 [17:01:37] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-2 - 13https://git.io/vbvb3 [17:01:38] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JoqbX [17:01:40] [02miraheze/mw-config] 07Reception123 03c0f015f - configure wgRightFunctionsUserGroups for quoteunquotecampaignwiki per… (#4136) [17:01:42] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-2 [17:02:38] miraheze/mw-config - Reception123 the build passed. [17:03:57] !log [@test3] starting deploy of {'config': True} to skip [17:03:58] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [17:04:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:04:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:04:23] !log [@mw11] starting deploy of {'config': True} to all [17:04:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:04:34] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 11s [17:04:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:13:57] RhinosF1: Okay, will do that. [17:14:32] !log INSERT INTO content_models (model_id, model_name) VALUES ( 1, 'wikitext') INSERT INTO content_models (model_id, model_name) VALUES ( 2, 'css');INSERT INTO content_models (model_id, model_name) VALUES ( 3, 'javascript');INSERT INTO content_models (model_id, model_name) VALUES ( 4, 'Scribunto');INSERT INTO content_models (model_id, model_name) VALUES ( 5, 'json'); on tscwiki and harrypotterwiki [17:14:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:20:28] [02puppet] 07Reception123 commented on pull request 03#1970: test3: switch MediaWiki to REL1_37 - 13https://git.io/Jom8k [17:20:57] [02puppet] 07Reception123 commented on pull request 03#1983: csp: add *.wikipedia.org to connect-src - 13https://git.io/Jom8Q [17:36:05] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 14.86, 8.70, 6.66 [17:36:53] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.54, 6.69, 6.13 [17:37:03] PROBLEM - mw10 APT on mw10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:37:10] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.53, 7.06, 5.91 [17:37:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.26, 7.83, 6.71 [17:38:48] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.17, 6.20, 6.02 [17:39:00] RECOVERY - mw10 APT on mw10 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [17:39:05] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.77, 6.49, 5.84 [17:39:18] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.43, 6.71, 6.44 [17:42:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.97, 7.00, 6.76 [17:44:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.43, 6.20, 6.49 [18:07:18] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.11, 8.11, 6.24 [18:07:19] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.38, 10.60, 7.44 [18:08:13] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.12, 7.73, 6.57 [18:09:13] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.00, 6.91, 6.02 [18:10:11] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.09, 6.74, 6.34 [18:11:07] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.44, 6.09, 5.82 [18:11:10] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 3.62, 6.94, 6.64 [18:13:06] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 2.71, 5.62, 6.20 [18:25:44] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.34, 2.97 [18:29:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.56, 3.21, 3.02 [18:36:00] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.57, 6.66, 5.57 [18:37:06] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.70, 8.75, 6.39 [18:37:16] RhinosF1 [[phab:8131]] is open now [18:37:16] https://phabricator.miraheze.org/8131 [18:37:17] [url] Login | phabricator.miraheze.org [18:37:18] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.66, 6.84, 5.56 [18:37:57] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.63, 5.84, 5.40 [18:38:47] T missing [18:38:56] [[phab:T8131]] [18:38:57] https://phabricator.miraheze.org/T8131 [18:38:58] [url] ⚓ T8131 CSP related issue for geohack.toolforge.com on FAMEPedia | phabricator.miraheze.org [18:39:00] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.45, 7.20, 6.12 [18:39:14] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.09, 5.75, 5.31 [18:39:37] 😘 [18:41:04] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.90, 6.18, 5.89 [18:53:10] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JoOo7 [18:53:12] [02miraheze/ssl] 07paladox 031cf4f21 - Update wildcard.miraheze.org-2020-2.crt [18:53:13] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [18:53:15] [02ssl] 07paladox opened pull request 03#434: Update wildcard.miraheze.org-2020-2.crt - 13https://git.io/JoOoF [18:55:55] !log disable puppet globally for rollout of ssl certificate [18:55:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:59:13] PROBLEM - mem1 Puppet on mem1 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 25 minutes ago with 0 failures [18:59:19] PROBLEM - services3 Puppet on services3 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 18 minutes ago with 0 failures [18:59:20] PROBLEM - ns2 Puppet on ns2 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 11 minutes ago with 0 failures [18:59:31] PROBLEM - db11 Puppet on db11 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 10 minutes ago with 0 failures [18:59:34] PROBLEM - cp14 Puppet on cp14 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 21 minutes ago with 0 failures [18:59:38] PROBLEM - dbbackup1 Puppet on dbbackup1 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 31 minutes ago with 0 failures [18:59:39] PROBLEM - ldap2 Puppet on ldap2 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 23 minutes ago with 0 failures [18:59:51] PROBLEM - cp12 Puppet on cp12 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 6 minutes ago with 0 failures [18:59:51] PROBLEM - phab2 Puppet on phab2 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 27 minutes ago with 0 failures [18:59:53] PROBLEM - mon2 Puppet on mon2 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 9 minutes ago with 0 failures [18:59:55] PROBLEM - graylog2 Puppet on graylog2 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 29 minutes ago with 0 failures [19:00:03] PROBLEM - mem2 Puppet on mem2 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 10 minutes ago with 0 failures [19:00:17] PROBLEM - cp13 Puppet on cp13 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 31 minutes ago with 0 failures [19:00:20] PROBLEM - test4 Puppet on test4 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 11 minutes ago with 0 failures [19:00:25] PROBLEM - mail2 Puppet on mail2 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 32 minutes ago with 0 failures [19:00:28] PROBLEM - mw8 Puppet on mw8 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 26 minutes ago with 0 failures [19:00:29] Lets begin... cc Reception123 [19:00:29] PROBLEM - gluster4 Puppet on gluster4 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 5 minutes ago with 0 failures [19:00:34] PROBLEM - cloud3 Puppet on cloud3 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 12 minutes ago with 0 failures [19:00:35] PROBLEM - jobchron1 Puppet on jobchron1 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 16 minutes ago with 0 failures [19:00:36] PROBLEM - mw12 Puppet on mw12 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 23 minutes ago with 0 failures [19:00:37] PROBLEM - services4 Puppet on services4 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 21 minutes ago with 0 failures [19:00:38] PROBLEM - db13 Puppet on db13 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 8 minutes ago with 0 failures [19:00:39] PROBLEM - mw9 Puppet on mw9 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 24 minutes ago with 0 failures [19:00:44] PROBLEM - cloud5 Puppet on cloud5 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 5 minutes ago with 0 failures [19:00:50] PROBLEM - db12 Puppet on db12 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 11 minutes ago with 0 failures [19:00:54] PROBLEM - puppet3 Puppet on puppet3 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 8 minutes ago with 0 failures [19:00:54] PROBLEM - cp15 Puppet on cp15 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 22 minutes ago with 0 failures [19:00:55] PROBLEM - cloud4 Puppet on cloud4 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 18 minutes ago with 0 failures [19:00:55] PROBLEM - gluster3 Puppet on gluster3 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 11 minutes ago with 0 failures [19:00:57] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 5 minutes ago with 0 failures [19:01:04] PROBLEM - mw10 Puppet on mw10 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 25 minutes ago with 0 failures [19:01:09] PROBLEM - mw13 Puppet on mw13 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 25 minutes ago with 0 failures [19:01:43] [02ssl] 07paladox closed pull request 03#434: Update wildcard.miraheze.org-2020-2.crt - 13https://git.io/JoOoF [19:01:44] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JoO9O [19:01:46] [02miraheze/ssl] 07paladox 03b63b9b7 - Update wildcard.miraheze.org-2020-2.crt (#434) [19:01:47] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [19:01:49] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [19:02:04] PROBLEM - mw11 Puppet on mw11 is WARNING: WARNING: Puppet is currently disabled, message: paladox - rolling out ssl certificate renewed, last run 25 minutes ago with 0 failures [19:02:50] RECOVERY - db12 Puppet on db12 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:03:18] !log flush ssl on all the database servers [19:03:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:03:33] RECOVERY - db11 Puppet on db11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:03:39] that was successful [19:04:05] great :) [19:04:32] RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:04:41] paladox: you can use show status like 'ssl_server_not%'; apparently to check it actually applied [19:05:03] oh [19:05:05] thanks! [19:05:10] https://mariadb.com/kb/en/ssltls-status-variables/#ssl_server_not_after [19:05:10] [url] SSL/TLS Status Variables - MariaDB Knowledge Base | mariadb.com [19:05:12] hmm [19:05:17] https://www.irccloud.com/pastebin/TN8NAnh1/ [19:05:32] That's not what we want [19:05:43] Maybe try flush ssl again? [19:06:42] i've tried it twice [19:06:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JoOFK [19:06:46] * paladox looks in the log [19:06:46] [02miraheze/mw-config] 07Universal-Omega 0349e72e8 - Enable `slow-parse` debug logging [19:06:48] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [19:06:49] [02mw-config] 07Universal-Omega opened pull request 03#4137: Enable `slow-parse` debug logging - 13https://git.io/JoOFy [19:07:18] hmm i see nothing of relevence in there [19:07:28] Weird [19:07:56] miraheze/mw-config - Universal-Omega the build passed. [19:08:07] CosmicAlpha: pls don't merge yet [19:08:40] paladox: is it worth deploying to mw/cp and seeing after why [19:08:48] yeh [19:08:58] moving to mw [19:09:02] Let's do it then [19:09:21] RhinosF1: no intention to just yet. [19:10:18] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:10:31] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:10:42] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:11:22] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:11:27] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:11:55] paladox: test3/mwtask1 [19:12:10] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:12:21] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.98, 19.96, 16.77 [19:12:24] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 17.11, 10.11, 6.97 [19:12:25] puppet is ran on mwtask1 [19:12:29] test3 isn't important [19:12:33] it's a test server [19:13:15] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.45, 7.91, 6.45 [19:13:29] Ok [19:13:36] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.72, 19.54, 17.50 [19:13:49] CosmicAlpha: mediawiki cluster should be back enabled now [19:14:20] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.11, 18.72, 16.72 [19:14:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 7.55, 8.01, 6.43 [19:15:03] paladox: just cp then I guess [19:15:10] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.68, 7.00, 6.29 [19:15:25] yeh [19:15:30] RECOVERY - cp14 Puppet on cp14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:15:34] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 20.20, 19.66, 17.78 [19:15:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.51, 3.31 [19:16:16] RECOVERY - cp13 Puppet on cp13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:16:18] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.41, 6.95, 6.22 [19:16:22] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:16:53] RECOVERY - cp15 Puppet on cp15 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:18:15] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.05, 6.36, 6.10 [19:18:17] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:18:19] RECOVERY - test2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:18:20] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.24, 7.64, 6.94 [19:19:06] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.40, 6.51, 6.27 [19:19:17] RECOVERY - gluster3 Puppet on gluster3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:19:32] RECOVERY - cnt.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:19:51] RECOVERY - uncyclopediakr.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:20:12] RECOVERY - dc.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:20:25] RECOVERY - mon2 Puppet on mon2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:20:31] RECOVERY - gluster4 Puppet on gluster4 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [19:20:37] RECOVERY - civclassic.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:20:38] RECOVERY - test1.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:20:39] RECOVERY - cloud3 Puppet on cloud3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:20:47] RECOVERY - intp.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:21:00] RECOVERY - dcmultiverse.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:21:30] RECOVERY - cloud4 Puppet on cloud4 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [19:22:03] RECOVERY - phab2 Puppet on phab2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:22:18] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.80, 3.40, 3.36 [19:22:20] RECOVERY - animatedfeet.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:22:38] RECOVERY - piwik.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:22:40] RECOVERY - cloud5 Puppet on cloud5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:22:42] RECOVERY - allthetropes.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:22:43] RECOVERY - commonswiki.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:22:50] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.02, 6.72, 5.96 [19:23:13] RECOVERY - unmade.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:23:17] RECOVERY - ldap2 Puppet on ldap2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:23:26] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:23:26] RECOVERY - animefeet.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:23:33] RECOVERY - marthaspeaks.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:23:35] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.59, 6.61, 5.77 [19:24:08] RECOVERY - wiki.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:24:20] RECOVERY - sims.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:24:23] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.17, 6.48, 6.63 [19:24:44] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.21, 6.23, 5.88 [19:25:05] RECOVERY - hellointernet.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:25:35] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.75, 6.24, 5.73 [19:26:25] RECOVERY - mail2 Puppet on mail2 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [19:28:39] RECOVERY - services4 Puppet on services4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:28:41] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.00, 6.86, 6.23 [19:29:18] RECOVERY - services3 Puppet on services3 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [19:30:21] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.66, 7.07, 6.76 [19:30:45] RECOVERY - puppet3 Puppet on puppet3 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [19:30:48] RECOVERY - jobchron1 Puppet on jobchron1 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [19:31:30] RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:31:31] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:31:40] RECOVERY - mem2 Puppet on mem2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:31:44] RECOVERY - dbbackup1 Puppet on dbbackup1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:32:00] RECOVERY - test4 Puppet on test4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:32:06] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:32:17] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.99, 7.13, 6.80 [19:32:59] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.56, 7.46, 6.39 [19:33:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.85, 3.74, 3.47 [19:34:23] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.68, 8.47, 7.32 [19:34:52] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.27, 8.01, 6.84 [19:35:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.86, 3.55 [19:35:53] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.22, 8.59, 7.20 [19:36:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.36, 7.14, 6.37 [19:37:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.74, 3.54 [19:38:22] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.71, 7.04, 5.98 [19:38:24] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.18, 7.54, 6.59 [19:38:37] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.63, 20.58, 18.87 [19:38:47] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 14.28, 8.03, 3.96 [19:40:17] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.41, 7.62, 6.32 [19:40:24] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.11, 6.87, 6.46 [19:40:32] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.04, 6.82, 7.25 [19:40:35] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.58, 19.78, 18.78 [19:41:51] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.94, 7.97, 7.38 [19:42:11] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.12, 6.77, 6.17 [19:42:25] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.49, 7.50, 6.72 [19:42:49] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.32, 7.81, 7.47 [19:43:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.35, 7.11, 7.15 [19:44:26] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.75, 5.67, 6.67 [19:44:27] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.05, 6.99, 6.65 [19:45:27] [02mw-config] 07Universal-Omega closed pull request 03#4137: Enable `slow-parse` debug logging - 13https://git.io/JoOFy [19:45:29] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [19:45:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jo31B [19:45:32] [02miraheze/mw-config] 07Universal-Omega 03dc140e2 - Enable `slow-parse` debug logging (#4137) [19:45:33] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [19:45:34] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.30, 2.86, 3.20 [19:46:27] miraheze/mw-config - Universal-Omega the build passed. [19:47:11] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.02, 6.27, 6.80 [19:49:37] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.33, 6.42, 6.79 [19:49:59] RECOVERY - bacula2 Disk Space on bacula2 is OK: DISK OK - free space: / 1282680 MB (99% inode=99%); [19:50:22] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:50:29] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.32, 6.58, 6.64 [19:50:38] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.99, 6.20, 6.79 [19:51:33] RECOVERY - bacula2 APT on bacula2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:54:39] !log delete ns1 in ramnode [19:54:43] cc JohnLewis Reception123 ^ [19:54:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:54:54] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 0.75, 1.87, 3.58 [19:56:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03revert-1716-paladox-patch-4 [+0/-0/±3] 13https://git.io/Jo3NC [19:56:05] [02miraheze/puppet] 07paladox 0374cbe02 - Revert "Remove db12 from bacula (#1716)" [19:56:06] [02puppet] 07paladox created branch 03revert-1716-paladox-patch-4 - 13https://git.io/vbiAS [19:56:08] [02puppet] 07paladox opened pull request 03#2014: Revert "Remove db12 from bacula" - 13https://git.io/Jo3N4 [19:56:24] [02puppet] 07paladox closed pull request 03#2014: Revert "Remove db12 from bacula" - 13https://git.io/Jo3N4 [19:56:26] [02miraheze/puppet] 07paladox deleted branch 03revert-1716-paladox-patch-4 [19:56:27] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±6] 13https://git.io/Jo3ND [19:56:29] [02miraheze/puppet] 07paladox 0374cbe02 - Revert "Remove db12 from bacula (#1716)" [19:56:30] [02miraheze/puppet] 07paladox 038116c83 - Merge pull request #2014 from miraheze/revert-1716-paladox-patch-4 [19:56:32] [02puppet] 07paladox deleted branch 03revert-1716-paladox-patch-4 - 13https://git.io/vbiAS [19:56:54] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 0.65, 1.47, 3.22 [20:00:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jo3hp [20:00:05] [02miraheze/puppet] 07paladox 03301eb98 - Update maximum static disk size on bacula [20:00:06] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [20:00:08] [02puppet] 07paladox opened pull request 03#2015: Update maximum static disk size on bacula - 13https://git.io/Jo3hh [20:01:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Josec [20:01:08] [02miraheze/puppet] 07paladox 03e5c6232 - Update bacula-dir.conf [20:01:10] [02puppet] 07paladox synchronize pull request 03#2015: Update maximum static disk size on bacula - 13https://git.io/Jo3hh [20:01:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Joseh [20:01:42] [02miraheze/puppet] 07paladox 03b4ee192 - Update bacula-dir.conf [20:01:43] [02puppet] 07paladox synchronize pull request 03#2015: Update maximum static disk size on bacula - 13https://git.io/Jo3hh [20:01:49] [02puppet] 07paladox closed pull request 03#2015: Update maximum static disk size on bacula - 13https://git.io/Jo3hh [20:01:51] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [20:01:52] [02miraheze/puppet] 07paladox pushed 034 commits to 03master [+0/-0/±4] 13https://git.io/JosvY [20:01:54] [02miraheze/puppet] 07paladox 03301eb98 - Update maximum static disk size on bacula [20:01:55] [02miraheze/puppet] 07paladox 03e5c6232 - Update bacula-dir.conf [20:01:57] [02miraheze/puppet] 07paladox 03b4ee192 - Update bacula-dir.conf [20:01:58] [02miraheze/puppet] 07paladox 03864f022 - Merge pull request #2015 from miraheze/paladox-patch-1 [20:02:00] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [20:02:58] !log [@test3] starting deploy of {'config': True} to skip [20:02:58] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [20:03:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:32] !log [@mw11] starting deploy of {'config': True} to all [20:04:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:43] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 11s [20:04:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:08:00] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.82, 6.81, 6.29 [20:08:43] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.12, 6.57, 6.10 [20:08:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.55, 3.61, 3.28 [20:09:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.96, 6.91, 6.03 [20:09:11] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.26, 7.04, 6.21 [20:10:00] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.96, 7.17, 6.48 [20:10:38] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.92, 6.18, 6.01 [20:11:08] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.39, 6.46, 5.96 [20:11:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.68, 6.93, 6.27 [20:12:00] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.77, 6.59, 6.35 [20:13:08] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.25, 6.42, 6.14 [20:14:43] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.32, 3.71, 3.46 [20:18:00] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [20:18:06] [02puppet] 07Universal-Omega opened pull request 03#2016: Add Python file extension to scripts - 13https://git.io/JosWN [20:18:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.58, 3.79, 3.52 [20:21:14] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.34, 20.66, 18.82 [20:21:44] Reception123: can #2016 be merged? [20:22:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.69, 3.63, 3.55 [20:23:00] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.20, 6.82, 6.36 [20:23:09] PROBLEM - bacula2 Bacula Databases db12 on bacula2 is CRITICAL: CRITICAL: Full, 302318 files, 53.38GB, 2021-03-10 20:47:00 (30.4 weeks ago) [20:23:12] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.51, 18.96, 18.44 [20:23:25] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.41, 6.87, 6.44 [20:25:02] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.70, 5.83, 6.06 [20:26:58] [02puppet] 07Universal-Omega closed pull request 03#1970: test3: switch MediaWiki to REL1_37 - 13https://git.io/JudJx [20:27:22] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.78, 6.67, 6.44 [20:28:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.95, 2.93, 3.28 [20:30:09] [02puppet] 07RhinosF1 synchronize pull request 03#1876: mwscript: add some logging for 'long' scripts - 13https://git.io/JRyOD [20:30:35] CosmicAlpha: rebased ^ [20:30:49] Thanks! [20:31:54] That's fine to be merged [20:35:23] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.14, 7.28, 6.42 [20:35:30] dmehus: hi [20:35:41] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.20, 7.47, 6.31 [20:37:23] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.43, 6.45, 6.21 [20:37:39] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.62, 6.78, 6.20 [20:51:48] [02puppet] 07paladox closed pull request 03#1876: mwscript: add some logging for 'long' scripts - 13https://git.io/JRyOD [20:51:49] [02miraheze/puppet] 07paladox pushed 036 commits to 03master [+7/-11/±94] 13https://git.io/JoGLJ [20:51:51] [02miraheze/puppet] 07RhinosF1 0308e3be7 - mwscript: add some logging for 'long' scripts [20:51:52] [02miraheze/puppet] 07RhinosF1 0303588f0 - Update mwscript.py [20:51:54] [02miraheze/puppet] 07RhinosF1 03b8f9be8 - mwscript: update 'long' scripts [20:51:55] [02miraheze/puppet] ... and 3 more commits. [20:52:04] [02puppet] 07paladox synchronize pull request 03#1983: csp: add *.wikipedia.org to connect-src - 13https://git.io/Jz8yc [20:53:21] [02puppet] 07paladox synchronize pull request 03#1983: csp: add *.wikipedia.org to connect-src - 13https://git.io/Jz8yc [20:53:35] [02puppet] 07paladox closed pull request 03#1983: csp: add *.wikipedia.org to connect-src - 13https://git.io/Jz8yc [20:53:37] [02miraheze/puppet] 07paladox pushed 035 commits to 03master [+0/-0/±5] 13https://git.io/JoGqP [20:53:38] [02miraheze/puppet] 07R4356th 035224413 - csp: add *.wikipedia.org to connect-src [20:53:40] [02miraheze/puppet] 07R4356th 03d79c937 - Add MW.org and *.WM.org [20:53:41] [02miraheze/puppet] 07paladox 030c918cb - Update csp.yaml [20:53:43] [02miraheze/puppet] ... and 2 more commits. [20:55:29] paladox: squash and merge is good to have on by default ;) [20:55:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.93, 3.82, 3.53 [20:56:30] JohnLewis: what does that do exactly? :) [20:56:57] Merges all commits into one [20:57:18] So above instead of merging 11 commits, it’s two [20:57:24] Oh [20:57:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.41, 3.96, 3.61 [20:59:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.50, 3.49, 3.48 [21:01:50] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.37, 3.06, 3.32 [21:36:41] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.69, 19.72, 16.76 [21:37:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.32, 7.08, 5.62 [21:37:34] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.05, 6.82, 5.82 [21:38:40] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.24, 18.10, 16.51 [21:39:20] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.69, 6.28, 5.50 [21:39:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.73, 3.12 [21:41:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.37, 6.66, 6.01 [21:41:51] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.63, 3.31, 3.04 [21:49:59] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.67, 4.04, 3.38 [21:51:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.20, 8.11, 6.80 [21:51:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.92, 3.95, 3.43 [21:53:38] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.60, 7.29, 6.64 [21:53:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.86, 3.45 [21:55:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.17, 3.63, 3.42 [21:58:43] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JoZ1J [21:58:44] [02miraheze/mw-config] 07RhinosF1 03f3ed37a - Cache: raise parser cache expiry to 7 days [21:59:39] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.55, 6.47, 6.53 [21:59:40] miraheze/mw-config - RhinosF1 the build has errored. [21:59:47] !log [rhinos@test3] starting deploy of {'config': True} to skip [21:59:48] !log [rhinos@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [21:59:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:59:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:59:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.88, 3.04, 3.25 [22:00:43] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JoZDQ [22:00:44] [02miraheze/mw-config] 07RhinosF1 03e7e8793 - Update GlobalCache.php [22:00:50] !log [rhinos@test3] starting deploy of {'config': True} to skip [22:00:51] !log [rhinos@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:00:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:01:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:01:29] would help if i pulled the change [22:01:50] miraheze/mw-config - RhinosF1 the build passed. [22:02:10] !log [@test3] starting deploy of {'config': True} to skip [22:02:11] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:02:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:02:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:05:05] RECOVERY - bacula2 Bacula Databases db11 on bacula2 is OK: OK: Full, 443588 files, 56.11GB, 2021-10-09 22:04:00 (1.1 minutes ago) [22:05:14] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.08, 22.30, 18.62 [22:05:48] !log [@mw11] starting deploy of {'config': True} to all [22:05:54] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.64, 7.50, 5.95 [22:05:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:06:09] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 20s [22:06:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:08:48] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.45, 7.19, 6.45 [22:09:09] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.02, 20.79, 18.91 [22:09:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.79, 5.82, 5.64 [22:10:43] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.72, 6.62, 6.32 [22:11:06] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.07, 18.80, 18.43 [22:14:41] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jonf2 [22:14:43] [02miraheze/mw-config] 07RhinosF1 03939f16f - cache: persist on settings changes [22:15:00] paladox: ^ [22:15:10] thanks! [22:15:40] miraheze/mw-config - RhinosF1 the build passed. [22:15:42] !log [@test3] starting deploy of {'config': True} to skip [22:15:43] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:15:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:15:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:17:16] !log [@mw11] starting deploy of {'config': True} to all [22:17:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:17:27] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 11s [22:17:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:25:28] !log switched swap off on mw8 [22:25:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:26:38] SRE, we seem to have a jobs backlog on `testwiki` that doesn't want to clear. Mind checking Grafana? [22:26:53] on it [22:26:56] ty [22:28:04] dmehus: only 1 job reported [22:30:05] dmehus: you just sent a mass message [22:30:37] itt'll clear [22:31:49] RhinosF1, yeah, but it hasn't taken this long since we changed the way unclaimed jobs are handled or whatever. Usually gets done right away. Also, I was showing like 27 jobs on `testwiki` [22:32:34] dmehus: they clear very quickly when it gets to it [22:32:44] @dmehus Ah, so you are here. I've been expecting you on IRC, actually. [22:34:38] Glad you make it. [22:35:07] RhinosF1, yeah, it's just weird... I haven't seen it take 30+ minutes to deliver a mass message since like 6 months ago or more. Usually clear within minutes [22:35:39] @Doug I'm guessing you're referring to April, right? [22:36:30] PROBLEM - mw10 APT on mw10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:37:59] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.71, 7.18, 5.95 [22:38:29] RECOVERY - mw10 APT on mw10 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [22:39:58] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.35, 6.20, 5.74 [22:41:37] DarkMatterMan4500, around there yeah [22:47:25] [02puppet] 07RhinosF1 opened pull request 03#2017: fix syntax - 13https://git.io/Jonyn [22:48:01] [02puppet] 07RhinosF1 synchronize pull request 03#1877: Add tests for python - 13https://git.io/JRyGp [22:48:27] paladox: please merge 2017 [22:48:52] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JonSA [22:48:54] [02miraheze/puppet] 07RhinosF1 03463be9d - fix syntax [22:48:55] [02miraheze/puppet] 07paladox 036a3d01a - Merge pull request #2017 from RhinosF1/patch-98 [22:48:57] [02puppet] 07paladox closed pull request 03#2017: fix syntax - 13https://git.io/Jonyn [22:49:18] ty [22:53:14] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jondb [22:53:16] [02miraheze/mw-config] 07paladox 032804f0f - Disable $wgLanguageConverterCacheType when using cli [22:53:17] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [22:53:19] [02mw-config] 07paladox opened pull request 03#4138: Disable $wgLanguageConverterCacheType when using cli - 13https://git.io/Jondp [22:53:26] [02mw-config] 07paladox closed pull request 03#4138: Disable $wgLanguageConverterCacheType when using cli - 13https://git.io/Jondp [22:53:28] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JonFm [22:53:29] [02miraheze/mw-config] 07paladox 032ed6ec7 - Disable $wgLanguageConverterCacheType when using cli (#4138) [22:53:31] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [22:53:32] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [22:54:13] miraheze/mw-config - paladox the build passed. [22:54:34] miraheze/mw-config - paladox the build passed. [22:54:57] !log [paladox@mw11] starting deploy of {'config': True, 'gitinfo': True} to all [22:55:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:55:32] !log [paladox@test3] starting deploy of {'config': True, 'gitinfo': True} to all [22:55:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:55:49] !log [paladox@mw11] finished deploy of {'config': True, 'gitinfo': True} to all - SUCCESS in 51s [22:55:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:56:49] !log [rhinos@test3] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/test.php (END - exit=2) [22:56:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:57:09] [02puppet] 07RhinosF1 opened pull request 03#2018: irc: fix W503 - 13https://git.io/JonxY [22:57:59] [02puppet] 07RhinosF1 opened pull request 03#2019: mwscript: escape shell msg - 13https://git.io/Jonpc [22:58:06] paladox: ^ [22:58:19] 2019 pls [22:58:22] [02puppet] 07paladox closed pull request 03#2019: mwscript: escape shell msg - 13https://git.io/Jonpc [22:58:24] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JonpQ [22:58:25] [02miraheze/puppet] 07RhinosF1 031961445 - mwscript: escape shell msg [22:58:27] [02miraheze/puppet] 07paladox 03cf416e8 - Merge pull request #2019 from RhinosF1/patch-99 [22:59:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JonjF [22:59:55] [02miraheze/mw-config] 07Universal-Omega 03662cbad - CI: add `continue-on-error: true` for pushing changes [22:59:56] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [22:59:58] [02mw-config] 07Universal-Omega opened pull request 03#4139: CI: add `continue-on-error: true` for pushing changes - 13https://git.io/Jonjp [23:00:30] [02puppet] 07RhinosF1 synchronize pull request 03#2018: irc: fix W503 - 13https://git.io/JonxY [23:00:48] paladox: also ^ [23:01:03] miraheze/mw-config - Universal-Omega the build passed. [23:01:07] [02puppet] 07paladox closed pull request 03#2018: irc: fix W503 - 13https://git.io/JonxY [23:01:08] [02miraheze/puppet] 07paladox pushed 033 commits to 03master [+0/-0/±4] 13https://git.io/Jocvg [23:01:10] [02miraheze/puppet] 07RhinosF1 03fe14b8e - irc: fix W503 [23:01:11] [02miraheze/puppet] 07RhinosF1 03b937aa7 - Update adminlogbot.py [23:01:13] [02miraheze/puppet] 07paladox 0371643f0 - Merge pull request #2018 from RhinosF1/patch-98 [23:01:24] paladox: https://github.com/miraheze/puppet/pull/2016 would be great too! [23:01:25] [url] Add Python file extension to scripts by Universal-Omega · Pull Request #2016 · miraheze/puppet · GitHub | github.com [23:01:42] [02puppet] 07paladox closed pull request 03#2016: Add Python file extension to scripts - 13https://git.io/JosWN [23:01:44] [02miraheze/puppet] 07paladox pushed 0311 commits to 03master [+14/-14/±6] 13https://git.io/Jocfg [23:01:45] [02miraheze/puppet] 07Universal-Omega 037fe5c24 - Rename deploy-mediawiki to deploy-mediawiki.py [23:01:47] [02miraheze/puppet] 07Universal-Omega 03fdde0a6 - Rename check-read-only to check-read-only.py [23:01:48] [02miraheze/puppet] 07Universal-Omega 03eb915ce - Rename cycle-puppet to cycle-puppet.py [23:01:50] [02miraheze/puppet] ... and 8 more commits. [23:01:52] [02mw-config] 07Universal-Omega closed pull request 03#4139: CI: add `continue-on-error: true` for pushing changes - 13https://git.io/Jonjp [23:01:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JocfM [23:01:55] [02miraheze/mw-config] 07Universal-Omega 032104d5c - CI: add `continue-on-error: true` for pushing changes (#4139) [23:01:57] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [23:01:58] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [23:02:08] [02puppet] 07RhinosF1 synchronize pull request 03#1877: Add tests for python - 13https://git.io/JRyGp [23:02:15] oh i forgot about squesh [23:02:53] miraheze/mw-config - Universal-Omega the build passed. [23:03:17] CosmicAlpha: https://github.com/RhinosF1/puppet/runs/3848600166?check_suite_focus=true [23:03:18] [url] Merge branch 'miraheze:master' into patch-67 · RhinosF1/puppet@ef8c8dc · GitHub | github.com [23:04:10] PROBLEM - mw12 APT on mw12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:04:19] !log [@test3] starting deploy of {'config': True} to skip [23:04:21] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 1s [23:04:50] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 28.90, 23.27, 18.00 [23:04:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:04:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:05:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.20, 2.92 [23:05:15] PROBLEM - mw10 APT on mw10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:05:30] RhinosF1: doing PR now to fix the remaining. [23:05:35] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.90, 8.66, 6.19 [23:05:43] [02puppet] 07RhinosF1 synchronize pull request 03#1877: Add tests for python - 13https://git.io/JRyGp [23:05:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.62, 18.16, 15.51 [23:05:55] CosmicAlpha: i updated ^ with a run of autopep8 [23:06:02] but rest can be for you [23:06:12] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.74, 8.54, 6.15 [23:06:49] !log [@mw11] starting deploy of {'config': True} to all [23:06:55] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 12.92, 11.27, 7.59 [23:06:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:07:05] PROBLEM - cp15 Stunnel Http for mw10 on cp15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:07:23] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.51, 3.04, 2.91 [23:07:26] https://github.com/RhinosF1/puppet/runs/3848613005?check_suite_focus=true [23:07:26] [url] few more fixes · RhinosF1/puppet@53b9556 · GitHub | github.com [23:07:51] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.65, 7.65, 5.51 [23:08:00] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 71s [23:08:02] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.85, 18.02, 15.83 [23:08:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:08:45] RECOVERY - mw12 APT on mw12 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [23:09:03] RECOVERY - cp15 Stunnel Http for mw10 on cp15 is OK: HTTP OK: HTTP/1.1 200 OK - 15744 bytes in 0.329 second response time [23:09:10] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.93, 23.98, 19.78 [23:09:36] RECOVERY - mw10 APT on mw10 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [23:09:39] [02puppet] 07Universal-Omega opened pull request 03#2020: Fix Flake8 violations - 13https://git.io/Joc3Z [23:09:49] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.40, 7.02, 5.53 [23:10:10] [02puppet] 07Universal-Omega synchronize pull request 03#2020: Fix Flake8 violations - 13https://git.io/Joc3Z [23:10:14] * RhinosF1 -> sleep [23:11:34] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.11, 7.24, 6.65 [23:11:48] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.24, 6.23, 5.41 [23:12:09] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.83, 6.97, 6.37 [23:12:58] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 3.92, 7.59, 7.42 [23:13:14] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 11.81, 17.81, 18.24 [23:13:35] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 2.96, 5.75, 6.17 [23:14:01] RhinosF1: https://github.com/miraheze/puppet/pull/2018 actually causes W503 I think. [23:14:02] [url] irc: fix W503 by RhinosF1 · Pull Request #2018 · miraheze/puppet · GitHub | github.com [23:14:05] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.79, 5.93, 6.06 [23:14:52] Or maybe it causes W504? [23:16:56] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.94, 5.59, 6.63 [23:17:12] I think W504 it affected [23:17:18] It didn't help much [23:17:25] I hate that error tbh [23:17:57] Let me know if you find a fix and I'll rebase the tests [23:18:05] Then we can probably merge [23:18:19] As it didn't like checking for *.py [23:18:25] [02puppet] 07Universal-Omega edited pull request 03#2020: Fix Flake8 violation - 13https://git.io/Joc3Z [23:35:59] PROBLEM - mw10 APT on mw10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:36:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 12.08, 8.62, 6.45 [23:37:15] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.92, 7.88, 6.50 [23:37:55] RECOVERY - mw10 APT on mw10 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [23:38:01] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.44, 7.63, 6.35 [23:39:13] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 2.90, 6.09, 6.01 [23:39:56] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.18, 6.05, 5.92 [23:44:06] !log [paladox@test3] starting deploy of {'config': True, 'gitinfo': True} to skip [23:44:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:44:37] !log [paladox@test3] finished deploy of {'config': True, 'gitinfo': True} to skip - SUCCESS in 31s [23:44:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log