[00:02:49] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:15:19] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.73, 3.47, 3.07 [00:17:19] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 1.78, 2.86, 2.89 [00:20:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.11, 3.69, 3.24 [00:22:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.14, 3.53, 3.24 [00:23:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:28:28] PROBLEM - puppet3 Current Load on puppet3 is WARNING: WARNING - load average: 3.99, 5.21, 3.37 [00:28:36] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:28:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.39, 3.26, 3.26 [00:29:01] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 348 failures. Last run 2 minutes ago with 348 failures. Failed resources (up to 3 shown): File[pt.graalmilitary.com_private],File[egoish.org],File[egoish.org_private],File[wiki.slimeapedia.org] [00:29:02] PROBLEM - puppet3 Puppet on puppet3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:29:14] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 119 failures. Last run 2 minutes ago with 119 failures. Failed resources (up to 3 shown): File[commons.polandballwiki.com_private],File[wiki.konjitownmc.cf],File[wiki.konjitownmc.cf_private],File[wiki.digitalcandela.com] [00:29:43] PROBLEM - cp13 Puppet on cp13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:30:15] PROBLEM - mail2 Puppet on mail2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:30:21] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:30:27] RECOVERY - puppet3 Current Load on puppet3 is OK: OK - load average: 0.74, 3.63, 3.02 [00:30:55] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:32:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:34:48] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:37:36] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:50:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.27, 4.30, 3.55 [00:52:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.89, 3.75, 3.44 [00:52:55] RECOVERY - puppet3 Puppet on puppet3 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [00:54:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.02, 4.38, 3.72 [00:55:00] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [00:55:13] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [00:56:21] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:56:47] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.82, 3.78, 3.58 [00:58:15] RECOVERY - mail2 Puppet on mail2 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [00:58:43] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.34, 3.17, 3.37 [00:58:54] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [00:59:43] RECOVERY - cp13 Puppet on cp13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:11:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.91, 3.99, 3.52 [01:11:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.75, 17.24, 23.43 [01:13:13] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.26, 3.32, 3.34 [01:19:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.53, 4.81, 5.98 [01:21:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.81, 16.79, 20.16 [01:21:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.47, 3.33 [01:23:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.19, 5.41, 5.92 [01:25:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.33, 4.59, 5.56 [01:25:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.97, 3.57 [01:27:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.14, 3.68, 3.51 [01:29:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.74, 3.54 [01:31:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.91, 3.41, 3.45 [01:33:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 12.10, 7.56, 6.34 [01:35:01] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.63, 24.02, 21.99 [01:35:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.99, 4.42, 3.81 [01:36:58] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.03, 21.61, 21.37 [01:37:01] PROBLEM - cp14 Current Load on cp14 is WARNING: WARNING - load average: 1.75, 1.08, 0.63 [01:39:01] RECOVERY - cp14 Current Load on cp14 is OK: OK - load average: 0.98, 1.06, 0.68 [01:39:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.41, 5.62, 5.95 [01:39:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.90, 3.73 [01:43:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 3.92, 3.75 [01:44:47] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.77, 17.66, 19.70 [01:45:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.18, 3.54, 3.63 [01:47:03] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.99, 3.61, 4.82 [01:51:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.33, 4.10, 3.84 [01:55:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.07, 3.93, 3.86 [01:57:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.95, 3.86 [01:59:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.83, 3.82 [02:09:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.55, 2.96, 3.40 [02:17:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.40, 3.41 [02:19:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.84, 3.83, 3.56 [02:20:50] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.48, 20.61, 18.62 [02:22:15] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.79, 8.20, 5.34 [02:22:47] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.45, 20.96, 18.99 [02:22:59] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.99, 7.06, 5.37 [02:23:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.76, 3.79, 3.65 [02:24:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.22, 7.14, 5.32 [02:24:55] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.76, 6.36, 5.33 [02:26:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.35, 6.37, 5.27 [02:30:35] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.21, 19.78, 19.36 [02:36:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.80, 5.15, 4.64 [02:37:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.11, 3.19, 3.38 [02:38:00] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.46, 4.93, 4.62 [02:41:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 4.00, 3.68 [02:43:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.14, 3.77, 3.64 [02:49:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.86, 4.12, 3.79 [02:50:04] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.57, 20.66, 19.10 [02:50:36] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.92, 6.79, 5.39 [02:52:01] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.24, 20.52, 19.30 [02:53:58] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 11.82, 17.18, 18.21 [02:54:30] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.38, 5.01, 4.98 [02:57:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.49, 4.00, 4.00 [03:06:08] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.04, 5.29, 4.94 [03:06:10] PROBLEM - equestripedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'equestripedia.org' expires in 7 day(s) (Mon 06 Dec 2021 03:00:01 GMT +0000). [03:08:05] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.97, 4.51, 4.70 [03:15:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 3.49, 3.52 [03:19:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.75, 3.74, 3.62 [03:21:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.43, 4.72, 4.02 [03:29:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.78, 3.77, 3.88 [03:35:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.08, 3.88, 3.81 [03:37:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.04, 3.40, 3.64 [03:41:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.46, 3.58 [03:43:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.71, 3.27, 3.50 [03:45:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.84, 2.79, 3.30 [03:57:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 3.69, 3.43 [03:59:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.73, 3.64, 3.45 [04:01:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.18, 3.38, 3.37 [04:18:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.78, 3.54, 3.38 [04:20:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.26, 4.02, 3.56 [04:26:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.50, 3.58, 3.55 [04:27:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.24, 4.93, 4.31 [04:29:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.86, 5.60, 4.62 [04:31:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.69, 5.36, 4.64 [04:32:14] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.22, 4.58, 3.94 [04:33:03] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.02, 4.63, 4.44 [04:35:51] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.240 second response time [04:35:56] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.007 second response time [04:35:59] PROBLEM - cp13 Stunnel Http for mw11 on cp13 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.003 second response time [04:36:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.76, 3.56, 3.67 [04:36:27] PROBLEM - cp14 Stunnel Http for mw11 on cp14 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.003 second response time [04:36:28] PROBLEM - cp12 Stunnel Http for mw11 on cp12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.235 second response time [04:36:34] PROBLEM - cp15 Varnish Backends on cp15 is CRITICAL: 1 backends are down. mw11 [04:36:47] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw11 [04:36:50] PROBLEM - cp15 Stunnel Http for mw11 on cp15 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.244 second response time [04:36:58] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.239 second response time [04:37:13] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.003 second response time [04:37:16] PROBLEM - cp14 Varnish Backends on cp14 is CRITICAL: 1 backends are down. mw11 [04:37:20] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw11 [04:37:21] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.004 second response time [04:37:45] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw11 [04:37:50] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw11 [04:37:59] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw11 [04:38:01] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.41, 3.92, 3.79 [04:38:10] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw11 [04:41:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.96, 3.84 [04:42:56] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.04, 6.90, 4.77 [04:43:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.87, 20.02, 16.96 [04:43:50] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.65, 6.25, 4.43 [04:44:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.15, 6.38, 4.47 [04:45:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.89, 19.13, 16.99 [04:45:50] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.71, 6.17, 4.61 [04:46:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.28, 6.35, 4.69 [04:46:55] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.40, 6.71, 5.32 [04:51:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.50, 21.20, 18.42 [04:52:55] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 6.92, 8.55, 6.59 [04:53:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.54, 18.74, 17.87 [04:54:55] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.11, 7.89, 6.58 [04:55:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.91, 3.03, 3.36 [04:58:56] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.84, 6.71, 6.37 [05:01:21] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.37, 21.78, 19.60 [05:03:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.74, 20.73, 19.50 [05:05:21] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 31.65, 23.72, 20.66 [05:05:47] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.29, 7.27, 6.74 [05:07:16] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.22, 5.59, 4.83 [05:07:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.93, 22.87, 20.71 [05:09:13] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.02, 5.49, 4.89 [05:09:37] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.52, 6.43, 6.52 [05:11:10] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.64, 6.36, 5.28 [05:15:04] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.91, 5.43, 5.15 [05:15:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.23, 18.75, 19.82 [05:17:03] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.27, 4.58, 4.86 [05:49:19] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 11.18, 7.10, 5.21 [05:50:42] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.04, 21.71, 19.32 [05:52:40] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.39, 22.81, 20.02 [05:53:13] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.39, 5.58, 5.04 [05:54:37] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.71, 21.43, 19.79 [05:56:31] hi [05:57:00] the @dmehus is where? [05:57:07] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.02, 5.52, 5.10 [05:57:22] @msnhinet8, here [05:57:44] ok [05:58:15] can you give me global rights admain? [05:58:26] https://meta.miraheze.org/wiki/Requests_for_global_rights [05:58:26] [url] Requests for global rights - Miraheze Meta | meta.miraheze.org [05:59:04] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.46, 5.39, 5.09 [05:59:44] i have interwiki global rights, can you help me do election? [06:00:00] i want to interwiki global rights, can you help me do election? [06:00:33] thanks. [06:01:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.51, 5.83, 5.29 [06:02:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.62, 19.79, 19.69 [06:03:19] can you help me dmehus? [06:03:47] msnhinet8, sure, but don't you mean local interwiki administrator? [06:04:13] what wiki(s) are you wanting interwiki administrator for? [06:04:18] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.79, 5.74, 3.39 [06:04:22] yes local interwiki administrator? [06:04:40] on which wiki(s)/ [06:04:55] s/\/\? [06:05:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.78, 5.30, 5.25 [06:05:10] https://zhinternetpedia.miraheze.org/w/index.php?title=%E9%A6%96%E9%A0%81&variant=zh-tw [06:05:12] [url] 首頁 - 网路百科 wiki | zhinternetpedia.miraheze.org [06:05:24] thanks. [06:05:56] the furture ,i can build the other languages. [06:06:18] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.97, 6.13, 3.83 [06:06:30] so need internetpedia interwiki [06:06:35] thanks. [06:07:09] so you know my mean? [06:07:59] Okay, that wiki has two other contributors, can you post a note on Penarc1's user talk page at [[:mh:zhinternetpedia:User talk:Penarc1]] and Fungster's user talk page at [[:mh:zhinternetpedia:User talk:Fungster]] asking if they have any objections to you holding local interwiki administrator on the wiki? [06:07:59] https://meta.miraheze.org/wiki/:mh:zhinternetpedia:User_talk:Penarc1 https://meta.miraheze.org/wiki/:mh:zhinternetpedia:User_talk:Fungster [06:08:18] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 10.09, 7.41, 4.57 [06:08:28] if they say no, I can treat that as unanimous consensus for you to hold the `interwiki-admin` bit on that wiki :) [06:09:03] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.70, 4.97, 5.10 [06:09:20] @msnhinet: see above [06:09:26] @msnhinet8: see above [06:09:29] they not active user, so you can use. [06:09:41] thanks. [06:10:44] @msnhinet8, that's true of Fungster. We can exclude him [06:10:58] can you post a note on Penarc1's user talk page asking if he supports? [06:11:07] if he says yes, I'll grant the bit :) [06:11:11] yes [06:11:15] ty :) [06:12:00] you can direct interwiki. [06:12:08] thanks. [06:12:18] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.10, 7.37, 5.21 [06:14:59] dmehus, i can like to wikipedia that, build 1 millions articles? [06:15:00] https://cdn.discordapp.com/attachments/808001911868489748/914398951896137789/unknown.png [06:15:10] 6,414,594 articles in English [06:17:20] at dmehus user online now? [06:17:42] yes I am online [06:18:00] i can like to wikipedia that, build 1 millions articles? [06:18:07] 6,414,594 articles in English [06:18:17] I seriously doubt you'll be able to properly manage a wiki with 6 million articles in English, as it would be a target for spam and vandalism [06:18:18] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.61, 6.40, 5.52 [06:18:27] I would not try and shoot so high :) [06:18:43] consider we're a free wiki host and have to conserve resources for all users :) [06:20:05] i know need conserve resources, but could all users very need can like to wikipedia very articles. [06:21:01] Can only hope miraheze. [06:21:06] In my opinion, if you want a Wikipedia-like wiki, try editing Wikipedia itself and apport your knowledge to it [06:21:37] oh, i know. [06:23:06] so my interwiki request global rights can use? [06:26:47] @msnhinet8, don't keep calling it global rights. It's a local right on zhinternetpediawiki. You need to post a note at [[:mh:zhinternetpedia:User talk:Penarc1]] asking if they have any objections. Wait 5-7 days, and if no objection, I'll grant rights [06:26:47] https://meta.miraheze.org/wiki/:mh:zhinternetpedia:User_talk:Penarc1 [06:31:30] ok, thanks. [06:51:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.04, 2.42 [06:55:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.78, 2.54, 2.36 [07:21:28] PROBLEM - avas.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'avas.wiki' expires in 7 day(s) (Mon 06 Dec 2021 07:20:24 GMT +0000). [07:34:13] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.37, 20.66, 18.19 [07:35:41] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.93, 5.12, 4.49 [07:36:10] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.03, 18.19, 17.60 [07:37:38] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.31, 4.56, 4.36 [07:37:46] msnhinet8, np [07:38:05] Nice [07:50:13] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.27, 5.48, 4.73 [07:52:10] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.94, 4.69, 4.52 [07:55:07] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.50, 6.86, 6.24 [07:58:55] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.07, 6.35, 6.19 [08:01:40] PROBLEM - ping6 on bacula2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 131.47 ms [08:02:44] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.10, 7.42, 6.65 [08:03:39] RECOVERY - ping6 on bacula2 is OK: PING OK - Packet loss = 0%, RTA = 73.72 ms [08:04:38] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.00, 7.15, 6.64 [08:06:32] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.11, 6.37, 6.42 [08:09:57] User :Penarc1 he not acitivy user. [08:10:24] https://zhinternetpedia.miraheze.org/wiki/Special:%E7%94%A8%E6%88%B7%E8%B4%A1%E7%8C%AE/Penarc1 [08:10:27] [url] Penarc1的用户贡献 - 网路百科 wiki | zhinternetpedia.miraheze.org [08:12:18] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.08, 6.48, 6.39 [08:15:15] msnhinet8, can't hurt to ask him. you have to wait 5-7 days anyway [08:15:25] so please just post the note on his talk page :) [08:16:18] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.53, 6.65, 6.50 [08:17:37] ok [08:19:20] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.39, 5.26, 4.42 [08:20:18] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.32, 6.98, 6.66 [08:20:31] i already penarc1 user talk edit:Hi, the internetpedia will new interwiki, thanks. [08:21:17] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.90, 4.88, 4.37 [08:21:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.68, 7.10, 5.76 [08:22:18] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.07, 6.53, 6.53 [08:22:22] dmehus : i have a quesations. [08:23:07] my internetpedia can will languages global 300 languages? like to wikipedia. [08:23:37] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.54, 6.52, 5.68 [08:24:14] https://zh.wikipedia.org/wiki/Wikipedia:%E7%BB%B4%E5%9F%BA%E7%99%BE%E7%A7%91%E8%AF%AD%E8%A8%80%E5%88%97%E8%A1%A8 [08:24:14] [wikipedia] Wikipedia:维基百科语言列表 | "本頁主要刊登维基百科多种语言的排序的列表,最近一次全面更新於2018年4月1日。全部的298个语言版本以条目数目为序,列出了各语言版本的中文名称、当地名称和條目數目,ISO […]" [08:27:47] https://meta.wikimedia.org/w/index.php?title=List_of_Wikipedias/zh&uselang=zh [08:27:48] [url] List of Wikipedias/zh - Meta | meta.wikimedia.org [08:29:26] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 13.10, 8.68, 6.62 [08:29:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.97, 20.44, 18.77 [08:31:21] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.78, 7.73, 6.51 [08:31:36] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.16, 20.30, 18.92 [08:32:18] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.25, 6.83, 6.58 [08:33:16] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.45, 6.37, 6.17 [08:34:18] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.05, 6.24, 6.40 [08:44:18] We'd rather not in my opinion as that would take up an unreasonable amount of space and there's no need for so many wikis [08:44:40] also, let's move to #miraheze-relay as this is -sre, for SRE/technical chatter only [08:47:51] ok [08:47:54] thanks. [08:52:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.26, 6.36, 5.47 [08:54:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.29, 5.52, 5.27 [08:54:18] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.22, 7.06, 6.54 [08:55:01] PROBLEM - cp14 Current Load on cp14 is CRITICAL: CRITICAL - load average: 4.83, 3.28, 1.76 [08:56:13] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.81, 19.41, 18.11 [08:56:18] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.82, 6.43, 6.36 [09:00:07] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.94, 17.90, 17.90 [09:01:01] PROBLEM - cp14 Current Load on cp14 is WARNING: WARNING - load average: 0.92, 1.78, 1.59 [09:03:01] RECOVERY - cp14 Current Load on cp14 is OK: OK - load average: 0.44, 1.33, 1.45 [09:21:56] PROBLEM - graylog2 Disk Space on graylog2 is WARNING: DISK WARNING - free space: / 78462 MB (10% inode=99%); [09:29:56] RECOVERY - graylog2 Disk Space on graylog2 is OK: DISK OK - free space: / 78906 MB (11% inode=99%); [09:35:56] PROBLEM - graylog2 Disk Space on graylog2 is WARNING: DISK WARNING - free space: / 78546 MB (10% inode=99%); [09:39:56] RECOVERY - graylog2 Disk Space on graylog2 is OK: DISK OK - free space: / 78898 MB (11% inode=99%); [09:43:56] PROBLEM - graylog2 Disk Space on graylog2 is WARNING: DISK WARNING - free space: / 78544 MB (10% inode=99%); [10:01:13] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15875 bytes in 0.121 second response time [10:01:22] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15869 bytes in 0.006 second response time [10:01:44] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.312 second response time [10:01:45] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 9 backends are healthy [10:01:48] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 0.287 second response time [10:01:50] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [10:01:59] RECOVERY - cp13 Stunnel Http for mw11 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.013 second response time [10:01:59] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 9 backends are healthy [10:02:10] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 9 backends are healthy [10:02:27] RECOVERY - cp14 Stunnel Http for mw11 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.007 second response time [10:02:28] RECOVERY - cp12 Stunnel Http for mw11 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15875 bytes in 0.327 second response time [10:02:34] RECOVERY - cp15 Varnish Backends on cp15 is OK: All 9 backends are healthy [10:02:47] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 9 backends are healthy [10:02:50] RECOVERY - cp15 Stunnel Http for mw11 on cp15 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.323 second response time [10:02:58] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.337 second response time [10:03:16] RECOVERY - cp14 Varnish Backends on cp14 is OK: All 9 backends are healthy [10:03:20] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 9 backends are healthy [10:03:26] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JMWIa [10:03:27] [02miraheze/dns] 07JohnFLewis 03c30c59c - deploy cp30 and cp31 into rotation [10:03:53] !log pushing new Canadian cps (30/31) into rotation. Will monitor their workload [10:03:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:05:41] ooh, that looks nice, John :) [10:07:03] The only ones will be taken out shortly, this is just to get their cache warmed up slowly and not take the full brunt of it :P [10:07:10] *other ones [10:09:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.35, 4.35, 3.55 [10:11:03] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.04, 4.23, 3.60 [10:12:12] oh [10:13:45] We might consider deploying more in the near future, as we are looking at cost savings for our backup infrastructure. But that would be Februrary-ish time. Though we don't have any information to suggest 4 cps aren't capable of handling our current traffic demands at least :) [10:22:11] @dmehus You're up early. [10:25:22] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JMWYv [10:25:23] [02miraheze/dns] 07JohnFLewis 034f3797d - deploy new GB proxies cp20/21 [10:25:39] !log pushing new GB cps (20/21) into rotation. Will monitor their workload [10:25:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:26:38] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JMWYc [10:26:40] [02miraheze/dns] 07JohnFLewis 0378a0916 - fix incorrect v6 syntax [10:38:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.81, 20.85, 18.54 [10:39:29] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.15, 6.37, 5.23 [10:40:48] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.27, 19.66, 18.41 [10:44:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.50, 22.07, 19.81 [10:48:34] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.38, 19.69, 19.35 [10:50:55] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.92, 6.59, 5.05 [10:52:25] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.37, 23.35, 20.98 [10:52:55] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.38, 6.64, 5.25 [10:56:20] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.55, 23.67, 21.69 [10:56:55] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.19, 6.21, 5.37 [10:58:17] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.43, 24.67, 22.26 [10:59:44] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.51, 7.28, 5.73 [11:01:41] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.07, 6.49, 5.63 [11:08:03] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.82, 22.93, 23.10 [11:17:04] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.38, 4.93, 5.79 [11:19:46] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.18, 20.52, 21.16 [11:21:43] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.40, 19.76, 20.82 [11:23:40] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.55, 21.93, 21.47 [11:25:37] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.54, 21.96, 21.60 [11:29:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.80, 5.82, 5.62 [11:31:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.77, 5.82, 5.64 [11:35:23] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.55, 17.86, 20.09 [11:41:03] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.69, 4.62, 5.07 [11:51:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.27, 5.31, 5.18 [11:52:04] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.13, 20.60, 20.02 [11:54:01] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.25, 19.22, 19.60 [11:59:38] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.86, 5.85, 5.39 [12:02:44] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.12, 20.82, 20.16 [12:03:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.34, 5.78, 5.48 [12:05:29] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.01, 5.64, 5.44 [12:17:10] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.97, 5.97, 5.95 [12:18:22] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.72, 20.94, 20.76 [12:19:07] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.14, 6.11, 6.01 [12:22:16] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.36, 21.91, 21.22 [12:23:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.12, 5.85, 5.99 [12:24:13] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.59, 18.76, 20.15 [12:35:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.13, 5.28, 5.31 [12:37:03] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.88, 4.59, 5.05 [12:45:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.81, 5.35, 5.09 [12:47:03] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.36, 4.79, 4.91 [12:51:31] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.32, 20.47, 18.77 [12:55:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.95, 5.47, 5.22 [12:55:25] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.30, 20.13, 19.17 [13:01:22] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.93, 20.85, 19.80 [13:03:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.51, 6.64, 5.76 [13:05:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.87, 5.80, 5.55 [13:05:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.67, 19.49, 19.49 [13:17:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.59, 5.77, 5.43 [13:17:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.75, 21.30, 20.10 [13:21:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.91, 5.20, 5.31 [13:23:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.35, 17.44, 19.02 [13:27:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.65, 5.51, 5.32 [13:29:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.21, 5.16, 5.23 [13:31:03] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.71, 4.68, 5.04 [13:51:50] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.23, 21.26, 18.93 [13:51:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.91, 5.86, 4.90 [13:53:47] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.65, 20.39, 18.88 [13:53:50] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.55, 5.72, 4.97 [13:55:47] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.23, 5.05, 4.81 [14:09:19] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.45, 5.03, 4.77 [14:11:17] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.35, 4.98, 4.78 [14:13:14] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.64, 4.76, 4.72 [14:22:07] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.75, 21.13, 19.45 [14:22:15] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.96, 7.42, 5.55 [14:23:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.36, 5.50, 5.00 [14:24:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.00, 6.69, 5.50 [14:25:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.81, 5.50, 5.06 [14:27:03] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.96, 5.09, 4.97 [14:29:56] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.83, 19.92, 19.72 [14:37:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 2.99, 2.85 [14:39:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.08, 3.05, 2.90 [14:43:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.65, 6.62, 5.53 [14:43:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 33.19, 25.25, 21.45 [14:46:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.84, 6.93, 5.74 [14:48:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.41, 6.61, 5.76 [14:49:25] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.18, 23.32, 22.01 [14:51:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.23, 5.98, 5.81 [14:51:22] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.12, 23.99, 22.40 [14:51:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.45, 3.19 [14:53:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.16, 6.46, 6.04 [14:55:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.06, 5.53, 5.74 [14:55:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.08, 21.49, 21.99 [14:55:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.24, 3.66, 3.32 [14:57:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.69, 3.36, 3.25 [15:00:52] PROBLEM - wiki.globasa.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.globasa.net' expires in 7 day(s) (Mon 06 Dec 2021 14:59:12 GMT +0000). [15:01:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.17, 6.12, 5.93 [15:03:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.77, 5.55, 5.74 [15:07:22] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.37, 18.40, 20.17 [15:09:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.30, 5.56, 5.66 [15:11:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.19, 5.23, 5.52 [15:11:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.39, 20.32, 20.53 [15:17:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.23, 5.64, 5.54 [15:19:22] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.52, 22.94, 21.54 [15:25:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.60, 22.10, 21.90 [15:27:21] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.99, 23.20, 22.28 [15:30:55] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.03, 8.32, 6.12 [15:32:55] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.28, 6.76, 5.82 [15:33:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.04, 23.45, 23.07 [15:35:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.33, 5.30, 5.92 [15:36:42] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.70, 3.86, 3.40 [15:38:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.87, 3.46 [15:42:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.71, 3.35, 3.35 [15:43:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.71, 5.61, 5.74 [15:45:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.74, 5.27, 5.61 [15:45:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.68, 18.78, 20.27 [15:47:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.55, 5.95, 5.82 [15:49:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.32, 5.79, 5.78 [15:49:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.89, 20.12, 20.58 [15:51:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.35, 19.57, 20.33 [15:53:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.28, 6.26, 5.97 [15:55:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.13, 5.56, 5.76 [15:57:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.22, 5.82, 5.83 [15:59:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.20, 5.71, 5.79 [16:01:28] PROBLEM - graylog2 APT on graylog2 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (1 critical updates). [16:03:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.15, 6.19, 5.95 [16:04:47] PROBLEM - phab2 APT on phab2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [16:05:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.94, 5.58, 5.75 [16:06:32] PROBLEM - mem1 APT on mem1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [16:09:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.15, 7.31, 6.38 [16:09:14] PROBLEM - cp15 APT on cp15 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [16:09:33] PROBLEM - cp14 APT on cp14 is CRITICAL: APT CRITICAL: 15 packages available for upgrade (1 critical updates). [16:09:51] PROBLEM - ldap2 APT on ldap2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [16:12:32] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [16:13:24] PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [16:15:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.40, 5.69, 5.93 [16:15:17] PROBLEM - cloud4 APT on cloud4 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [16:16:47] PROBLEM - jobchron1 APT on jobchron1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [16:20:27] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [16:20:28] PROBLEM - bacula2 APT on bacula2 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:20:35] PROBLEM - gluster3 APT on gluster3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [16:20:35] PROBLEM - wiki.newnormal.fans - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.newnormal.fans' expires in 7 day(s) (Mon 06 Dec 2021 16:16:02 GMT +0000). [16:20:41] PROBLEM - db11 APT on db11 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (1 critical updates). [16:21:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.83, 5.58, 5.69 [16:21:36] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.23, 22.87, 20.17 [16:21:45] PROBLEM - mw13 APT on mw13 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [16:21:51] PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [16:22:12] PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [16:22:28] RECOVERY - ns2 APT on ns2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [16:22:31] RECOVERY - mem1 APT on mem1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [16:22:41] RECOVERY - db11 APT on db11 is OK: APT OK: 9 packages available for upgrade (0 critical updates). [16:22:47] RECOVERY - jobchron1 APT on jobchron1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [16:23:14] RECOVERY - cp15 APT on cp15 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [16:23:19] RECOVERY - mw11 APT on mw11 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [16:23:28] RECOVERY - graylog2 APT on graylog2 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [16:23:32] RECOVERY - cp14 APT on cp14 is OK: APT OK: 14 packages available for upgrade (0 critical updates). [16:23:45] RECOVERY - mw13 APT on mw13 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [16:23:50] RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [16:23:51] RECOVERY - ldap2 APT on ldap2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [16:24:11] RECOVERY - mw9 APT on mw9 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [16:24:27] RECOVERY - bacula2 APT on bacula2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:24:33] RECOVERY - mw8 APT on mw8 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [16:24:35] RECOVERY - gluster3 APT on gluster3 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [16:24:47] RECOVERY - phab2 APT on phab2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [16:25:34] PROBLEM - gluster4 APT on gluster4 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [16:26:36] PROBLEM - cp20 APT on cp20 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:26:38] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [16:27:16] PROBLEM - cloud5 APT on cloud5 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [16:29:04] RECOVERY - cloud4 APT on cloud4 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:29:24] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.41, 21.94, 21.92 [16:29:50] PROBLEM - mw12 APT on mw12 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [16:31:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.69, 5.04, 5.78 [16:31:49] RECOVERY - mw12 APT on mw12 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:32:33] RECOVERY - cp20 APT on cp20 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:32:38] RECOVERY - ns1 APT on ns1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:33:15] RECOVERY - cloud5 APT on cloud5 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:33:34] RECOVERY - gluster4 APT on gluster4 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:34:05] !log spree of security patches for libicu63 [16:34:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:35:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.30, 17.32, 19.90 [16:38:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 3.66, 3.26 [16:39:03] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.39, 4.19, 4.98 [16:42:03] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 3.39, 3.25 [16:45:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.07, 3.48, 3.34 [16:49:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.79, 3.19, 3.27 [16:51:21] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.32, 21.54, 19.42 [16:51:45] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.35, 5.26, 4.88 [16:52:56] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.98, 6.45, 4.96 [16:53:08] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.43, 9.31, 6.12 [16:53:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.99, 21.44, 19.67 [16:53:43] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.29, 4.93, 4.81 [16:56:50] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.48, 3.50, 3.41 [16:56:56] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.50, 6.76, 5.49 [16:57:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.83, 20.29, 19.60 [16:59:50] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.58, 6.97, 5.47 [17:00:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.63, 3.48 [17:00:55] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.27, 7.34, 6.51 [17:00:55] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 8.00, 7.92, 6.30 [17:01:50] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.10, 6.43, 5.47 [17:02:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.74, 3.55 [17:05:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.26, 20.78, 20.11 [17:06:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.75, 3.19, 3.36 [17:06:44] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.72, 6.69, 6.53 [17:06:55] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.46, 6.60, 6.26 [17:07:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.90, 19.24, 19.63 [17:13:05] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.25, 6.03, 5.18 [17:14:55] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.08, 7.60, 6.76 [17:17:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.63, 5.25, 5.10 [17:18:56] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.78, 6.46, 6.52 [17:19:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.80, 3.76, 3.52 [17:19:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.12, 5.15, 5.05 [17:20:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.88, 7.33, 6.72 [17:20:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.96, 4.25, 3.74 [17:21:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.75, 5.30, 5.11 [17:22:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.12, 3.41, 3.49 [17:23:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.27, 5.48, 5.18 [17:23:44] [02mw-config] 07dmehus opened pull request 03#4236: Re-add extendedconfirmed protection level to lhmnwiki - 13https://git.io/JMl5Z [17:24:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.14, 6.58, 6.56 [17:24:48] miraheze/mw-config - dmehus the build passed. [17:24:51] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.57, 3.85, 3.64 [17:25:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.66, 5.43, 5.19 [17:25:11] Reception123, if you get a chance, can you merge #4236 before the next puppet run in 6 minutes? :) [17:25:28] or RhinosF1 (or anyone who is available) [17:26:46] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.76, 3.63 [17:29:03] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.42, 4.84, 5.01 [17:30:56] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.40, 7.52, 6.92 [17:31:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.29, 20.93, 20.46 [17:31:42] [02mw-config] 07Reception123 closed pull request 03#4236: Re-add extendedconfirmed protection level to lhmnwiki - 13https://git.io/JMl5Z [17:31:43] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JMlF4 [17:31:45] [02miraheze/mw-config] 07dmehus 03ddd0c05 - Re-add extendedconfirmed protection level to lhmnwiki (#4236) [17:31:59] Reception123, ty :) [17:32:06] np! [17:32:09] did we make the puppet run? [17:32:13] no big deal if not [17:32:16] I'm afraid not :( [17:32:19] oh ok [17:32:19] since it's at :28 [17:32:29] I'll wait 26 more minutes or so heh [17:32:31] thanks :) [17:32:47] miraheze/mw-config - Reception123 the build passed. [17:32:55] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.07, 6.59, 6.65 [17:33:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.77, 20.06, 20.16 [17:38:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.36, 2.97, 3.31 [17:41:31] !log [@mw11] starting deploy of {'config': True} to all [17:41:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:41:50] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 18s [17:41:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:45:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 3.63, 3.47 [17:47:03] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.64, 3.19, 3.32 [17:47:48] dmehus: looks like it did ^ [17:54:33] !log [@test3] starting deploy of {'config': True} to skip [17:54:34] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [17:54:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:54:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:55:18] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.14, 6.26, 5.43 [17:57:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.95, 5.33, 5.19 [17:57:22] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JMlhE [17:57:23] [02miraheze/dns] 07JohnFLewis 034faad96 - depool old cp servers [17:59:12] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.35, 5.03, 5.10 [18:00:37] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.27, 4.48, 3.72 [18:02:41] RhinosF1, you mean puppet ran? [18:02:55] dmehus: it's puppet that triggered the deploy [18:03:05] 17:41:51 !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 18s [18:03:07] RhinosF1, ah, yeah that makes sense [18:03:23] dmehus: see botlogs too [18:03:24] I was wondering what those deploys were. Reception123 clarified for me [18:03:47] They are the ones triggered by puppet's notify => calls [18:03:47] I had thought it was Omega doing DB table changes in advance of the MW 1.37 upgrade :D [18:03:53] ah [18:04:14] Not sure how omega could trigger that [18:04:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.17, 3.88, 3.65 [18:04:30] well I didn't know about the automated deploy thing you deployed [18:04:37] but now I do [18:04:41] this was before I knew that [18:04:48] It's been there for a while [18:04:55] But at least you know now [18:05:00] well a month or two, I think? [18:05:02] yeah [18:05:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.41, 5.03, 5.06 [18:06:24] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.52, 4.13, 3.77 [18:06:45] dmehus: https://github.com/miraheze/puppet/commit/0651ebdc6c4ab58950504ddb959336fcd004757f#diff-fd800f812a93342636f4d07678f65e54b991c79c3e22277240197db8306fc7c9 [18:06:46] [url] mediawiki: automatically sync config (#1838) · miraheze/puppet@0651ebd · GitHub | github.com [18:06:51] August 1st [18:07:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.26, 5.33, 5.16 [18:09:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.78, 5.22, 5.15 [18:11:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.51, 5.47, 5.23 [18:14:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.96, 3.91, 3.77 [18:15:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.32, 5.03, 5.18 [18:17:03] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.23, 4.45, 4.94 [18:19:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.83, 4.08, 3.84 [18:21:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.84, 3.78 [18:23:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.41, 3.93, 3.81 [18:25:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.75, 3.49, 3.66 [18:29:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.79, 3.89, 3.75 [18:31:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.79, 3.74 [18:33:47] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.10, 18.37, 18.15 [18:35:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.76, 3.72 [18:36:40] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.95, 1.64, 0.89 [18:37:41] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.78, 18.09, 18.08 [18:38:40] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.83, 1.30, 0.86 [18:39:13] RhinosF1, ack, thanks :) [18:39:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.05, 3.72, 3.74 [18:45:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 3.74, 3.70 [18:47:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.67, 3.68 [18:56:13] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.16, 22.43, 20.16 [18:56:32] hello! could someone with the requisite access provide a backtrace for this production error? https://phabricator.miraheze.org/T8337 With that I may be able to diagnose what's going wrong and possibly provide a fix [18:56:33] [url] ⚓ T8337 PageForms no longer submit on spcodex.wiki | phabricator.miraheze.org [18:58:10] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.99, 21.46, 20.04 [19:00:07] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.32, 19.11, 19.35 [19:03:40] musikanimal: sure, hold on [19:03:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.79, 3.96, 3.70 [19:05:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.79, 3.66 [19:06:05] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.71, 5.35, 4.86 [19:07:35] musikanimal: done :) [19:07:46] thanks! [19:09:00] https://github.com/wikimedia/mediawiki-extensions-PageForms/commit/8e13415f613e068e69d5fcb5f298615803396946 [19:09:01] [url] Fix more issues spotted by phan · wikimedia/mediawiki-extensions-PageForms@8e13415 · GitHub | github.com [19:09:03] caused by this [19:09:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.17, 4.03, 3.75 [19:09:58] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.70, 5.00, 4.84 [19:11:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.25, 3.73, 3.68 [19:12:12] Special:Version says Page Forms is at fabcd30 (8 October 2021) which is prior to that commit [19:16:33] that commit was in Sept, and it looks directly related to the problem [19:17:16] yeah sorry, there's another newer commit with a very similar message https://github.com/wikimedia/mediawiki-extensions-PageForms/commit/6c11e009d97810112946bdb6dab678e100fed827 [19:17:17] [url] Fix even more issues spotted by phan · wikimedia/mediawiki-extensions-PageForms@6c11e00 · GitHub | github.com [19:17:32] fun and confusing :) [19:18:25] could we try reverting 8e13415f613e068e69d5fcb5f298615803396946 assuming it reverts cleanly? (given it's just phan fixes and not supposed to change any functionality) [19:19:36] Reception123: ^ as MediaWiki [19:21:05] that seems reasonable but how exactly would we 'revert' it? [19:22:47] I assume you can revert directly on Miraheze servers? just a quick fix [19:23:41] or commit the revert to the MW 1.36 branch. I haven't tested the MW 1.37 branch yet to see if it's fixed there, but I will shortly [19:24:06] REL1_36 branch, I mean [19:24:48] Reception123: you could revert on the deploy server [19:25:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.98, 2.95, 3.40 [19:26:19] it should also be fixed upstream [19:26:53] it doesn't revert cleanly on REL1_36 :( [19:36:34] I created https://gerrit.wikimedia.org/r/c/mediawiki/extensions/PageForms/+/742248 to cherry pick the apparent fix to REL1_36. For now we could try manually cherry picking on the deploy server? With great effort I can try to test this locally (rolling back MW is hard, with the db changes and all) [19:37:39] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.52, 3.41 [19:39:35] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.44, 3.39 [19:41:30] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.83, 3.55 [19:43:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.97, 3.11, 3.32 [19:47:19] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.37, 4.21, 3.72 [19:47:50] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.03, 5.57, 4.83 [19:49:22] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.48, 20.94, 18.24 [19:51:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.07, 20.40, 18.38 [19:51:44] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.50, 5.86, 5.14 [19:53:41] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.67, 7.70, 5.92 [19:55:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.99, 3.91, 3.84 [19:55:22] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.63, 21.27, 19.27 [19:57:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.95, 19.99, 19.07 [19:59:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.10, 5.63, 5.60 [20:00:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.33, 3.76, 3.77 [20:01:29] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.31, 6.17, 5.82 [20:02:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.68, 3.74 [20:03:26] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.33, 5.52, 5.62 [20:08:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.52, 4.26, 3.97 [20:09:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.74, 20.40, 19.53 [20:10:20] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.01, 7.24, 5.68 [20:11:13] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.43, 4.29, 4.99 [20:11:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.84, 18.87, 19.06 [20:14:16] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.42, 6.22, 5.66 [20:21:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.52, 21.69, 19.98 [20:21:42] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.46, 10.01, 7.22 [20:22:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.84, 4.00 [20:22:37] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.45, 7.10, 5.89 [20:23:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.41, 20.32, 19.70 [20:25:13] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.83, 6.94, 5.63 [20:26:06] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.09, 7.09, 6.18 [20:26:29] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.36, 6.52, 5.96 [20:27:13] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.26, 6.24, 5.53 [20:27:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.79, 3.91 [20:28:04] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.26, 6.46, 6.06 [20:29:07] PROBLEM - mw12 APT on mw12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:31:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.27, 8.34, 6.77 [20:31:22] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.57, 20.71, 19.96 [20:33:19] RECOVERY - mw12 APT on mw12 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:34:55] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.11, 7.55, 6.79 [20:35:21] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.13, 22.12, 20.53 [20:36:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.89, 8.58, 7.26 [20:37:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.31, 5.26, 4.62 [20:39:03] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.00, 4.43, 4.40 [20:39:05] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.50, 6.99, 6.50 [20:39:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.65, 23.08, 21.44 [20:40:57] PROBLEM - mw12 APT on mw12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:41:02] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.49, 7.91, 6.87 [20:41:21] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 33.07, 26.20, 22.71 [20:42:57] RECOVERY - mw12 APT on mw12 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:43:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.16, 4.97, 4.61 [20:45:03] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.54, 4.68, 4.57 [20:48:05] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.90, 6.87, 6.28 [20:48:35] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.15, 6.77, 5.88 [20:48:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:49:19] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.88, 19.85, 18.01 [20:50:05] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.00, 6.38, 6.19 [20:50:35] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.11, 6.45, 5.89 [20:51:05] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.79, 5.18, 4.86 [20:51:18] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.62, 19.15, 17.97 [20:51:27] PROBLEM - mw12 APT on mw12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:53:04] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.57, 5.05, 4.86 [20:53:28] RECOVERY - mw12 APT on mw12 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:53:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:54:35] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.85, 6.90, 6.23 [20:56:35] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.54, 6.42, 6.13 [20:58:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.02, 7.40, 7.77 [20:59:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.82, 3.67, 3.99 [21:00:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:01:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.93, 5.30, 5.07 [21:01:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.29, 3.83, 4.00 [21:03:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.87, 4.00 [21:04:14] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.91, 8.53, 8.01 [21:05:04] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.72, 4.97, 4.98 [21:05:36] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:05:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.50, 4.38, 4.16 [21:06:10] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.43, 7.66, 7.76 [21:07:33] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.71, 7.38, 6.64 [21:08:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.94, 7.21, 6.30 [21:11:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.15, 7.58, 7.54 [21:12:05] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.88, 7.57, 6.58 [21:13:54] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.64, 7.91, 7.69 [21:14:47] PROBLEM - mw11 APT on mw11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:15:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.65, 6.57, 5.63 [21:15:50] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.69, 8.17, 7.80 [21:16:47] RECOVERY - mw11 APT on mw11 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:17:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.83, 5.89, 5.50 [21:17:29] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.64, 7.79, 7.31 [21:17:50] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.56, 7.40, 7.55 [21:19:05] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.75, 6.03, 5.58 [21:21:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.12, 5.60, 5.47 [21:21:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.30, 21.00, 23.56 [21:23:27] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.90, 5.89, 6.71 [21:24:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.24, 7.31, 7.68 [21:30:55] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.28, 5.60, 7.77 [21:31:22] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.45, 22.38, 22.97 [21:31:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.33, 3.48, 3.86 [21:33:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.13, 5.48, 5.40 [21:34:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.67, 4.83, 6.66 [21:36:55] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.94, 4.97, 6.76 [21:38:11] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.97, 1.79, 1.19 [21:40:09] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.83, 1.42, 1.12 [21:51:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.51, 3.91, 3.75 [21:53:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.15, 3.57, 3.64 [21:55:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.90, 3.76 [21:57:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.77, 3.73, 3.73 [22:01:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.17, 4.24, 3.91 [22:05:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.68, 3.55, 3.73 [22:07:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.55, 19.89, 23.70 [22:09:22] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 32.45, 23.67, 24.56 [22:17:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.64, 2.87, 3.24 [22:23:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.35, 3.31 [22:27:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.11, 3.17, 3.24 [22:33:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.84, 3.49 [22:35:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.65, 3.45 [22:39:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.12, 3.28, 3.34 [22:52:15] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.78, 7.70, 5.53 [22:54:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.20, 6.95, 5.52 [22:56:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.31, 6.29, 5.44 [23:00:01] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:05] !log [@mw11] starting deploy of {'l10nupdate': True} to all [23:00:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.89, 3.39, 3.25 [23:02:52] PROBLEM - mw11 APT on mw11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:03:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.80, 3.29, 3.24 [23:04:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 17.85, 11.69, 7.31 [23:04:47] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.36, 19.86, 16.11 [23:06:47] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.65, 19.13, 16.31 [23:06:58] RECOVERY - mw11 APT on mw11 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [23:07:27] !log [@mw11] finished deploy of {'l10nupdate': True} to all - FAIL: [35072, 0, 0, 0, 0, 0, 0, 0, 0] in 442s [23:07:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:10:05] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.00, 6.66, 6.51 [23:11:03] PROBLEM - mw12 APT on mw12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:12:24] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [23:12:55] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 7.29, 8.83, 6.94 [23:13:03] RECOVERY - mw12 APT on mw12 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:14:55] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.09, 7.41, 6.64 [23:16:55] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.49, 6.36, 6.34 [23:19:12] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp21.miraheze.org - CNAME OK [23:22:42] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.71, 7.23, 6.07 [23:22:55] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.99, 7.10, 6.62 [23:23:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.93, 22.81, 23.99 [23:24:38] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.36, 6.76, 6.02 [23:25:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.65, 4.74, 5.83 [23:28:29] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.72, 7.68, 6.54 [23:28:59] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 16.99, 10.68, 8.06 [23:29:21] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.45, 23.29, 23.71 [23:29:42] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 1780s [23:30:39] PROBLEM - mw12 APT on mw12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:31:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:31:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.21, 23.42, 23.73 [23:32:39] RECOVERY - mw12 APT on mw12 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:33:21] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.19, 23.00, 23.50 [23:35:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.65, 5.06, 5.35 [23:37:06] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:37:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.73, 7.22, 5.92 [23:40:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.04, 7.80, 7.58 [23:41:15] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.99, 7.96, 6.54 [23:42:06] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:42:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.35, 3.55, 3.32 [23:45:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.94, 5.53, 5.73 [23:47:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.28, 6.73, 6.15 [23:47:03] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.71, 6.47, 6.36 [23:50:15] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.61, 7.14, 7.10 [23:51:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.60, 5.80, 5.91 [23:52:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.31, 7.09, 7.08 [23:53:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.00, 3.20, 3.30 [23:56:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.25, 6.28, 6.76 [23:57:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.89, 21.22, 23.50 [23:58:55] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.93, 6.66, 7.91