[00:00:46] RECOVERY - thestarsareright.org - LetsEncrypt on sslhost is OK: OK - Certificate 'thestarsareright.org' will expire on Tue 05 Jul 2022 22:41:50 GMT +0000. [00:01:11] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5a09dd1260ae...0862142ca6a8 [00:01:12] [02miraheze/ssl] 07MirahezeSSLBot 030862142 - Bot: Update SSL cert for rww2.org [00:02:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.72, 3.95, 3.96 [00:02:20] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0862142ca6a8...51c14554b727 [00:02:21] [02miraheze/ssl] 07MirahezeSSLBot 0351c1455 - Bot: Update SSL cert for wiki.vinesh.eu.org [00:04:12] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 721 failures. Last run 3 minutes ago with 721 failures. Failed resources (up to 3 shown): File[pwiki.arkcls.com],File[pwiki.arkcls.com_private],File[kunwok.org],File[kunwok.org_private] [00:05:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.90, 2.78, 3.32 [00:07:53] PROBLEM - es101 Current Load on es101 is CRITICAL: CRITICAL - load average: 4.16, 3.62, 2.92 [00:09:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/51c14554b727...3d8c2bb6225e [00:09:27] [02miraheze/ssl] 07MirahezeSSLBot 033d8c2bb - Bot: Update SSL cert for www.bluepageswiki.org [00:09:48] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.55, 3.45, 2.94 [00:11:43] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 3.17, 3.40, 2.98 [00:14:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.53, 3.55 [00:15:07] RECOVERY - wiki.worldsofweary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worldsofweary.com' will expire on Tue 05 Jul 2022 22:39:26 GMT +0000. [00:15:34] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.40, 3.55, 3.14 [00:15:51] RECOVERY - civwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'civwiki.org' will expire on Tue 05 Jul 2022 22:28:08 GMT +0000. [00:17:31] PROBLEM - es101 Current Load on es101 is CRITICAL: CRITICAL - load average: 4.21, 3.69, 3.23 [00:18:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.81, 3.70, 3.66 [00:19:42] RECOVERY - www.arru.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'www.arru.xyz' will expire on Tue 05 Jul 2022 22:32:16 GMT +0000. [00:20:14] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.20, 1.00, 1.97 [00:20:34] RECOVERY - en.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.pornwiki.org' will expire on Tue 05 Jul 2022 22:45:58 GMT +0000. [00:21:31] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.74, 3.78, 3.39 [00:24:40] https://phabricator.miraheze.org/T9042 ! [00:24:40] [url] ⚓ T9042 ucroniasaltwiki and hispano67wiki down! | phabricator.miraheze.org [00:25:31] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 2.44, 3.20, 3.25 [00:26:29] What is going on now.... [00:27:27] Looks like db121 is down.... [00:28:04] paladox [00:28:11] @Void [00:28:14] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.01, 1.15, 1.69 [00:28:32] oh your here [00:28:38] db121 has been down for a while [00:29:40] RECOVERY - otcg.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'otcg.ml' will expire on Tue 05 Jul 2022 22:53:34 GMT +0000. [00:29:53] I can see that. I just saw some phabricator tasks about it. I don't have access to fix this issue, so I really hope someone here does, thats around right now.... looks like it's been down for over an hour [00:30:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.65, 4.05, 3.74 [00:30:53] !sre is anyone around? [00:30:54] Key was added [00:31:21] What key? [00:31:47] Ugh.... what did I just do with wm-bot now....? [00:32:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.94, 3.79, 3.70 [00:32:07] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [00:33:11] RECOVERY - rww2.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rww2.org' will expire on Tue 05 Jul 2022 23:01:04 GMT +0000. [00:34:58] !log restart db121 - mariadb oom'd [00:35:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:35:14] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 46 Threads: 58 Questions: 1567 Slow queries: 94 Opens: 422 Open tables: 416 Queries per second avg: 34.065 [00:35:17] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 52009 bytes in 3.070 second response time [00:35:44] RECOVERY - www.bluepageswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.bluepageswiki.org' will expire on Tue 05 Jul 2022 23:09:19 GMT +0000. [00:35:47] Thank you paladox! [00:36:00] thank you so much paladox [00:36:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.20, 4.39, 3.89 [00:37:14] RECOVERY - wiki.meregos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meregos.com' will expire on Tue 05 Jul 2022 22:49:10 GMT +0000. [00:38:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.62, 3.06, 2.64 [00:39:37] thanks, thanks, thanks paladox :D [00:40:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.46, 3.66, 3.72 [00:42:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.55, 3.41, 2.84 [00:43:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.04, 4.42, 3.61 [00:44:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:44:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.54, 3.81, 3.73 [00:44:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.84, 3.62, 3.00 [00:45:20] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.09, 8.35, 6.65 [00:46:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.00, 3.48, 3.62 [00:47:19] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 4.81, 7.07, 6.38 [00:48:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.30, 3.84, 3.21 [00:49:18] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.40, 7.30, 6.52 [00:50:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.09, 8.08, 6.13 [00:51:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.07, 3.80, 3.71 [00:51:18] RECOVERY - wiki.vinesh.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.vinesh.eu.org' will expire on Tue 05 Jul 2022 23:02:14 GMT +0000. [00:52:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.62, 3.10, 3.40 [00:52:39] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.82, 7.39, 6.13 [00:52:55] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.17, 3.16, 3.09 [00:53:17] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.96, 7.63, 6.85 [00:55:16] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.00, 8.10, 7.11 [00:57:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.13, 3.06, 3.36 [00:57:15] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.04, 7.55, 7.05 [00:57:38] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:58:03] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:58:09] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:59:14] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.47, 6.57, 6.75 [00:59:42] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.350 second response time [01:00:04] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 2.152 second response time [01:00:10] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.569 second response time [01:01:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.92, 4.18, 3.75 [01:04:35] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.48, 3.99, 3.46 [01:05:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.73, 3.91, 3.79 [01:06:14] [02mw-config] 07Universal-Omega closed pull request 03#4550: Update Wikibase.php with repoSiteName for gratispaideiawiki - 13https://github.com/miraheze/mw-config/pull/4550 [01:06:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d7333490bac3...9b7991ce881e [01:06:17] [02miraheze/mw-config] 07ugochimobi 039b7991c - Update Wikibase.php with repoSiteName for gratispaideiawiki (#4550) [01:06:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.11, 3.54, 3.35 [01:07:18] miraheze/mw-config - Universal-Omega the build passed. [01:08:25] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.25, 4.01, 3.53 [01:08:42] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:10:06] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:10:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.03, 3.57, 3.42 [01:10:50] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.476 second response time [01:11:45] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:12:16] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.99, 3.40, 3.37 [01:13:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.18, 3.51, 3.56 [01:13:51] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 6.958 second response time [01:14:17] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.941 second response time [01:15:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.69, 3.63 [01:18:35] !log [@test101] starting deploy of {'config': True} to all [01:18:36] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [01:19:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:20:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:21:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.10, 3.10, 3.37 [01:22:51] !log [@mwtask111] starting deploy of {'config': True} to all [01:22:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.93, 3.24, 3.15 [01:23:09] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 18s [01:23:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:23:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:23:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.27, 3.45, 3.14 [01:24:00] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.42, 7.63, 7.03 [01:24:55] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.57, 2.98, 3.06 [01:25:38] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.72, 2.95, 1.80 [01:27:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.68, 3.56, 3.21 [01:29:57] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.22, 6.76, 6.82 [01:30:09] CosmicAlpha: you added an infobot key [01:30:20] !sre del [01:30:21] Successfully removed sre [01:30:24] http://wm-bot.wmcloud.org/dump/%23miraheze-sre.htm [01:30:24] @info [01:31:57] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.50, 6.65, 6.77 [01:33:38] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.79, 1.93, 1.95 [01:34:50] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.69, 3.54, 3.46 [01:35:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.32, 3.44, 3.39 [01:35:54] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.71, 6.95, 6.90 [01:37:50] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.35, 3.15, 3.29 [01:37:53] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.21, 6.47, 6.72 [01:38:44] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.51, 3.20, 3.35 [01:45:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.70, 1.53, 1.68 [01:49:38] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.42, 1.75, 1.77 [01:51:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.07, 1.52, 1.68 [01:56:43] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.34, 7.03, 6.56 [01:57:14] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:57:36] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:59:13] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.209 second response time [01:59:36] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.382 second response time [02:00:42] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.16, 6.28, 6.38 [02:05:38] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.41, 2.12, 1.75 [02:07:38] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.52, 1.73, 1.64 [02:09:38] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.54, 6.96, 6.55 [02:09:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.22, 1.60, 1.60 [02:11:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.79, 3.64, 3.29 [02:11:37] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.33, 7.13, 6.67 [02:13:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.51, 3.33, 3.22 [02:13:36] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.19, 7.25, 6.75 [02:15:35] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.88, 7.15, 6.76 [02:17:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.78, 4.19, 3.58 [02:17:34] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 10.55, 7.63, 6.94 [02:17:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.07, 3.41, 3.14 [02:19:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.60, 3.91, 3.55 [02:19:50] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.84, 3.14, 3.07 [02:25:31] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.30, 7.77, 7.35 [02:27:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.70, 3.54 [02:29:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.76, 3.82, 3.61 [02:29:29] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.06, 8.45, 7.72 [02:32:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.57, 8.96, 7.58 [02:34:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:34:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.57, 8.02, 7.15 [02:36:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.60, 7.82, 7.18 [02:36:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.11, 9.03, 7.87 [02:39:25] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.00, 7.98, 7.88 [02:39:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:39:43] MacFan4000: oh, what's that even do? [02:40:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.57, 9.54, 8.32 [02:40:39] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.86, 8.38, 7.91 [02:42:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.71, 8.83, 8.20 [02:43:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.55, 2.85, 3.25 [02:46:23] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.98, 8.46, 8.17 [02:47:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.58, 8.55, 8.13 [02:53:19] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.97, 5.78, 6.68 [02:53:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.71, 8.00, 8.11 [02:54:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:56:50] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.67, 3.34 [02:56:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.43, 3.79, 3.10 [02:58:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.15, 3.52, 3.33 [02:58:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.98, 3.77, 3.18 [03:00:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.72, 3.97, 3.52 [03:00:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.43, 4.69, 3.56 [03:02:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.64, 3.91, 3.55 [03:08:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.28, 3.29, 3.37 [03:08:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.82, 3.56 [03:10:09] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.03, 3.42, 3.18 [03:10:55] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.23, 3.24, 3.38 [03:11:42] !test is result [03:11:42] Key was added [03:11:45] !test [03:11:45] result [03:11:50] !test del [03:11:50] Successfully removed test [03:11:59] CosmicAlpha: ^^ [03:12:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.75, 3.19, 3.12 [03:13:40] MacFan4000: oh. Interesting... Thanks! [03:14:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:16:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.19, 3.91, 3.55 [03:19:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.53, 3.31 [03:21:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.02, 3.06, 3.17 [03:21:38] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.94, 1.47, 1.08 [03:23:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.09, 1.27, 1.06 [03:24:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.72, 3.48, 3.53 [03:28:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.90, 3.87, 3.46 [03:34:55] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.10, 3.38, 3.37 [03:35:49] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.97, 3.10, 3.38 [03:59:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:08:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:13:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:20:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:22:51] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.22, 7.02, 5.81 [04:24:49] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.78, 6.38, 5.73 [04:45:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:45:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.07, 9.10, 7.71 [04:47:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.63, 8.82, 7.77 [04:50:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:55:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:55:29] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.35, 7.90, 7.77 [05:07:41] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.21, 2.66 [05:09:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.49, 2.90, 2.61 [05:17:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:22:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:34:37] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [05:34:37] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:03:51] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=tlhwiki --username-prefix=tlh --report 1 --no-updates /home/reception/tlhwiki-20090508-pages-meta-history.xml (START) [06:04:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:04:23] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [06:05:08] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:09:04] PROBLEM - wiki.msnld.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.msnld.com' expires in 15 day(s) (Sat 23 Apr 2022 05:45:26 GMT +0000). [06:09:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 3.73, 3.27 [06:10:46] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8209dc9c20e2...d898d57e02fd [06:10:47] [02miraheze/puppet] 07Reception123 03d898d57 - re-enable jobrunner on mw102 [06:11:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3d8c2bb6225e...e5cb0b58d533 [06:11:56] [02miraheze/ssl] 07MirahezeSSLBot 03e5cb0b5 - Bot: Update SSL cert for wiki.msnld.com [06:14:17] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=celebswiki --username-prefix=simplewiki --report 1 --no-updates /home/reception/Wikipedia-20220401133916.xml (START) [06:14:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:15:08] ok : [RESOLVED] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:15:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.92, 3.91, 3.54 [06:16:10] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/c60138b0be9f [06:16:11] [02miraheze/mw-config] 07Reception123 03c60138b - add simplewiki import source for celebswiki per T9013 [06:16:13] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config [06:16:14] [02mw-config] 07Reception123 opened pull request 03#4551: add simplewiki import source for celebswiki per T9013 - 13https://github.com/miraheze/mw-config/pull/4551 [06:17:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.13, 3.28, 3.36 [06:17:21] miraheze/mw-config - Reception123 the build passed. [06:17:36] [02mw-config] 07Reception123 closed pull request 03#4551: add simplewiki import source for celebswiki per T9013 - 13https://github.com/miraheze/mw-config/pull/4551 [06:17:37] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config [06:17:39] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [06:17:40] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9b7991ce881e...2d6d850d6b41 [06:17:42] [02miraheze/mw-config] 07Reception123 032d6d850 - add simplewiki import source for celebswiki per T9013 (#4551) [06:17:57] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=tlhwiki --username-prefix=tlh --report 1 --no-updates /home/reception/tlhwiki-20090508-pages-meta-history.xml (END - exit=0) [06:18:01] !log [@test101] starting deploy of {'config': True} to all [06:18:02] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [06:18:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:18:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:18:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:18:47] miraheze/mw-config - Reception123 the build passed. [06:19:02] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.80, 2.21, 1.52 [06:21:12] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=tlhwiki (START) [06:21:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:24:11] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=hirzensawwiki --username-prefix=w /home/reception/Wikipedia-20220407062202.xml (START) [06:24:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:25:02] Reception123: please run with --no-updates [06:25:13] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=tlhwiki (END - exit=0) [06:25:16] RhinosF1: I usually do but this one is like less than 2MB [06:25:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:25:29] Reception123: yeah but we have a broken queue anyway [06:25:42] !log [@mwtask111] starting deploy of {'config': True} to all [06:25:45] You spawning a stack of jobs at once will do loads [06:25:49] since per T8876 Special:Import is basically broken [06:25:51] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 9s [06:25:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:25:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:26:44] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=hirzensawwiki --username-prefix=w /home/reception/Wikipedia-20220407062202.xml (END - exit=0) [06:26:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:27:46] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=cenepublicinterfacewiki --username-prefix=w /home/reception/Wikipedia-20220407062202.xml --no-updates (START) [06:27:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:30:38] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=cenepublicinterfacewiki --username-prefix=w /home/reception/Wikipedia-20220407062202.xml --no-updates (END - exit=0) [06:30:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:32:46] !log RENAMED catalinshopswiki to katatvwiki [06:32:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:38:14] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=fridaynightfunkingwiki --username-prefix=fridaynightfunking --report 1 --no-updates /home/reception/ClassicHOF.xml (START) [06:38:15] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=fridaynightfunkingwiki --username-prefix=fridaynightfunking --report 1 --no-updates /home/reception/ClassicHOF.xml (END - exit=65280) [06:38:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:38:25] Reception123: run deleted wikis backup script + deletion script [06:38:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:38:34] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=funkipediamodswiki --username-prefix=fridaynightfunking --report 1 --no-updates /home/reception/ClassicHOF.xml (START) [06:38:38] RECOVERY - wiki.msnld.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.msnld.com' will expire on Wed 06 Jul 2022 05:11:48 GMT +0000. [06:38:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:42:14] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.93, 1.84, 1.99 [06:44:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.08, 1.92, 2.01 [06:48:14] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.61, 1.90, 1.98 [06:50:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.24, 2.04, 2.02 [06:52:57] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.53, 6.81, 6.07 [06:53:21] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=funkipediamodswiki --username-prefix=fridaynightfunking --report 1 --no-updates /home/reception/ClassicHOF.xml (END - exit=0) [06:53:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:54:56] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.60, 6.42, 6.00 [06:55:37] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=celebswiki --username-prefix=simplewiki --report 1 --no-updates /home/reception/Wikipedia-20220401133916.xml (END - exit=0) [06:55:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:00:38] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.97, 1.48, 1.80 [07:03:54] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [07:04:34] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.28, 1.40, 1.69 [07:08:03] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:12:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.95, 7.99, 6.95 [07:14:06] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.12, 7.24, 6.80 [07:18:03] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:22:58] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=funkipediamodswiki --username-prefix=fridaynightfunking --report 1 --no-updates /home/reception/ClassicHOF.xml (START) [07:23:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:23:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:23:20] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=celebswiki (START) [07:23:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:24:00] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=funkipediamodswiki --username-prefix=fridaynightfunking --report 1 --no-updates /home/reception/ClassicHOF.xml (END - exit=0) [07:24:05] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.92, 2.21, 1.87 [07:24:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:28:01] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.54, 1.97, 1.87 [07:28:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:34:02] [02mw-config] 07ugochimobi opened pull request 03#4552: set repoSiteId for benpediawiki - 13https://github.com/miraheze/mw-config/pull/4552 [07:35:07] miraheze/mw-config - ugochimobi the build passed. [07:37:50] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.84, 2.54, 2.06 [07:40:53] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [07:42:53] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [07:50:03] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:00:03] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:07:17] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.44, 0.79, 1.85 [08:09:15] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.26, 0.61, 1.65 [08:23:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:28:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:39:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:49:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:26:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:28:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.99, 2.57, 1.98 [09:29:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.93, 2.91, 2.29 [09:29:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.16, 2.43 [09:30:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.14, 2.46, 2.02 [09:31:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.09, 2.81, 2.33 [09:31:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.03, 2.68, 2.34 [09:36:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:48:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.61, 8.19, 7.33 [09:50:54] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.80, 7.15, 7.05 [09:52:46] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.10, 1.65, 1.36 [09:54:46] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.21, 1.49, 1.34 [10:06:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:11:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:12:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.68, 8.17, 7.29 [10:14:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.60, 7.76, 7.23 [10:26:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:43:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.39, 7.95, 7.26 [10:45:05] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.12, 6.95, 6.96 [10:46:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:50:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.36, 8.69, 8.15 [10:52:25] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.32, 8.18, 8.02 [10:53:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:58:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:08:41] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/f7a0ab7d88d2...27da9cff965a [11:08:43] [02miraheze/landing] 07translatewiki 0327da9cf - Localisation updates from https://translatewiki.net. [11:09:44] miraheze/landing - translatewiki the build passed. [11:11:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:15:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.28, 9.12, 8.37 [11:16:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:16:42] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.21, 6.68, 6.18 [11:17:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.31, 8.03, 7.24 [11:18:42] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.52, 6.21, 6.07 [11:19:05] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.43, 7.34, 7.08 [11:19:11] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.42, 10.18, 8.94 [11:21:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.67, 8.07, 7.70 [11:23:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:23:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.28, 8.02, 7.72 [11:24:46] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.15, 1.73, 1.44 [11:24:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.66, 8.37, 7.65 [11:25:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.01, 9.84, 9.22 [11:26:46] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.42, 1.59, 1.42 [11:26:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.85, 8.03, 7.60 [11:27:07] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.11, 9.79, 9.27 [11:27:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.95, 8.55, 8.01 [11:29:12] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.54, 8.19, 7.94 [11:33:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:35:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.72, 9.47, 8.50 [11:36:06] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:36:46] Reception123: ^ is looking bad [11:37:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.94, 8.74, 8.34 [11:39:47] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.53, 8.40, 8.27 [11:39:50] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.28, 8.65, 8.18 [11:41:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:41:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.51, 8.50, 7.69 [11:41:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.31, 9.46, 8.52 [11:43:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.71, 9.63, 9.75 [11:43:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.12, 8.81, 8.55 [11:44:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.82, 4.18, 3.17 [11:44:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.30, 9.29, 8.19 [11:46:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.84, 3.17 [11:46:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.11, 8.75, 8.14 [11:46:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.58, 9.16, 8.29 [11:48:44] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.30, 8.21, 8.02 [11:48:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.14, 10.18, 9.89 [11:50:10] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.21, 3.21, 3.08 [11:51:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:51:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.92, 9.92, 9.39 [11:52:54] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.85, 7.90, 8.03 [11:53:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.20, 9.72, 9.05 [11:55:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.14, 9.60, 9.10 [11:57:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.81, 3.35 [11:59:11] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.85, 9.15, 9.08 [11:59:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.67, 3.36 [12:01:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.57, 8.95, 9.02 [12:01:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.59, 3.91, 3.47 [12:01:49] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.10, 8.95, 7.94 [12:03:49] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.02, 8.41, 7.85 [12:04:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.88, 9.46, 9.11 [12:05:00] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.16, 9.29, 9.12 [12:05:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.10, 3.53, 3.42 [12:06:06] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:06:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.38, 8.93, 8.96 [12:06:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.85, 8.40, 8.82 [12:07:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.97, 4.15, 3.64 [12:07:30] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.08, 1.60, 1.39 [12:08:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.06, 9.35, 9.90 [12:09:06] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:09:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.72, 3.62, 3.51 [12:10:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.26, 9.00, 8.94 [12:10:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.62, 8.65, 8.74 [12:10:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.52, 9.77, 9.99 [12:11:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.32, 3.70, 3.54 [12:12:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.86, 8.71, 8.86 [12:12:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.10, 7.85, 8.43 [12:12:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.81, 9.03, 9.69 [12:13:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.50, 3.49 [12:16:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.95, 9.83, 9.81 [12:17:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.19, 8.99, 8.48 [12:17:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.48, 3.13, 3.36 [12:18:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.38, 9.23, 8.99 [12:19:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:20:54] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.74, 7.84, 8.14 [12:20:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.38, 9.14, 9.01 [12:22:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.24, 9.74, 9.88 [12:24:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.78, 10.02, 9.35 [12:24:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.29, 8.97, 8.63 [12:26:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.81, 9.44, 9.66 [12:28:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.02, 7.76, 8.25 [12:31:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.96, 8.65, 7.99 [12:34:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.37, 9.96, 9.96 [12:35:40] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.96, 8.01, 7.92 [12:36:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.18, 9.22, 9.49 [12:41:04] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.01, 5.03, 3.81 [12:41:23] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:41] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:42:54] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:42:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.75, 8.26, 8.11 [12:43:01] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:43:29] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.205 second response time [12:43:43] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.307 second response time [12:44:51] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.011 second response time [12:44:58] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.426 second response time [12:45:08] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.785 second response time [12:46:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.37, 8.41, 8.21 [12:48:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.67, 10.22, 9.84 [12:50:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.55, 9.42, 9.24 [12:50:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.12, 9.70, 9.70 [12:51:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.94, 3.94, 3.71 [12:52:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.57, 9.67, 9.66 [12:54:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.09, 8.61, 8.21 [12:54:42] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.48, 7.68, 6.72 [12:54:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.24, 9.38, 9.29 [12:54:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.75, 9.66, 9.66 [12:56:33] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.97, 8.05, 8.06 [12:56:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.09, 10.54, 9.99 [12:59:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.93, 3.14, 3.39 [13:01:50] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.52, 9.04, 8.56 [13:02:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.59, 8.66, 8.27 [13:04:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.24, 7.77, 7.99 [13:04:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.57, 10.32, 9.68 [13:06:42] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.46, 6.31, 6.50 [13:06:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.23, 9.64, 9.52 [13:08:41] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.38, 4.01, 3.32 [13:09:35] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.09, 8.87, 8.62 [13:10:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.94, 3.87, 3.34 [13:12:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.65, 3.96, 3.42 [13:13:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.45, 4.29, 3.63 [13:14:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 3.93, 3.48 [13:15:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.52, 9.84, 9.22 [13:16:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.32, 3.77, 3.46 [13:18:16] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.30, 3.66, 3.46 [13:18:56] !log runJobs.php on constantnoblewiki to clear backlog [13:19:14] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.92, 1.36, 1.01 [13:19:15] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 9.84, 10.02, 9.45 [13:19:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:20:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.40, 3.38, 3.39 [13:21:11] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.58, 9.87, 9.46 [13:21:14] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.86, 1.13, 0.97 [13:22:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.12, 9.49, 9.88 [13:23:07] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.72, 10.00, 9.54 [13:23:19] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [13:25:03] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.63, 8.90, 9.21 [13:26:11] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:26:14] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:26:26] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:26:30] !log increase matomo101 disk size by 10gb [13:26:47] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:26:48] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:27:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:28:32] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.552 second response time [13:29:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.59, 3.48, 3.39 [13:30:21] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.405 second response time [13:30:22] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 8.157 second response time [13:30:56] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 4.574 second response time [13:30:56] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.234 second response time [13:30:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.55, 9.17, 9.15 [13:30:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.34, 3.68, 3.24 [13:30:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.40, 9.55, 9.64 [13:31:41] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.46, 3.09, 3.25 [13:32:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.40, 9.55, 9.64 [13:34:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.87, 9.01, 8.88 [13:34:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.23, 8.93, 9.13 [13:34:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.06, 10.01, 9.79 [13:35:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.34, 9.07, 8.31 [13:36:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.75, 9.34, 9.02 [13:36:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.31, 3.11, 3.10 [13:36:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.36, 9.77, 9.74 [13:37:05] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.88, 8.48, 8.18 [13:38:39] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/afdb3b9ca897 [13:38:41] [02miraheze/mw-config] 07paladox 03afdb3b9 - Add site notice for emergency database maintenance [13:38:42] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [13:38:44] [02mw-config] 07paladox opened pull request 03#4553: Add site notice for emergency database maintenance - 13https://github.com/miraheze/mw-config/pull/4553 [13:39:17] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/afdb3b9ca897...92525a5b189a [13:39:18] [02miraheze/mw-config] 07github-actions 0392525a5 - CI: lint code to MediaWiki standards [13:39:20] [02mw-config] 07github-actions[bot] synchronize pull request 03#4553: Add site notice for emergency database maintenance - 13https://github.com/miraheze/mw-config/pull/4553 [13:40:02] miraheze/mw-config - paladox the build passed. [13:40:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.70, 9.56, 9.15 [13:40:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.76, 9.54, 9.20 [13:40:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.11, 9.12, 9.40 [13:42:08] [02mw-config] 07paladox synchronize pull request 03#4553: Add site notice for emergency database maintenance - 13https://github.com/miraheze/mw-config/pull/4553 [13:42:10] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/92525a5b189a...babaaee88a9b [13:42:12] [02miraheze/mw-config] 07paladox 03babaaee - Update Sitenotice.php [13:42:32] [02mw-config] 07paladox closed pull request 03#4553: Add site notice for emergency database maintenance - 13https://github.com/miraheze/mw-config/pull/4553 [13:42:33] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [13:42:35] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2d6d850d6b41...64a15cd227ac [13:42:36] [02miraheze/mw-config] 07paladox 0364a15cd - Add site notice for emergency database maintenance (#4553) [13:42:38] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [13:42:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.32, 8.58, 8.29 [13:42:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.27, 9.27, 9.14 [13:42:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.60, 8.83, 9.25 [13:43:00] !log [paladox@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [13:43:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.31, 8.99, 8.50 [13:43:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:43:33] !log [paladox@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 32s [13:43:37] miraheze/mw-config - paladox the build passed. [13:43:39] miraheze/mw-config - paladox the build passed. [13:44:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:44:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.02, 3.82, 3.37 [13:44:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.82, 9.23, 9.14 [13:44:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.52, 9.45, 9.43 [13:46:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.65, 9.84, 9.40 [13:48:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.61, 3.40 [13:48:19] !log [@test101] starting deploy of {'config': True} to all [13:48:20] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [13:48:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.51, 9.31, 9.26 [13:48:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.64, 9.68, 9.55 [13:50:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.54, 9.56, 8.88 [13:50:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.93, 10.25, 9.65 [13:51:00] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.87, 8.07, 8.37 [13:51:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/b563c6bf0481 [13:51:22] [02miraheze/puppet] 07paladox 03b563c6b - jobrunner: Increase memory to 300M and time limit to 300 [13:51:23] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [13:51:25] [02puppet] 07paladox opened pull request 03#2468: jobrunner: Increase memory to 300M and time limit to 300 - 13https://github.com/miraheze/puppet/pull/2468 [13:51:48] [02puppet] 07RhinosF1 opened pull request 03#2469: jobrunner: allow higher memory for refreshLinks - 13https://github.com/miraheze/puppet/pull/2469 [13:51:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.27, 3.93, 3.53 [13:52:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b563c6bf0481...9ee420bec6a8 [13:52:36] [02miraheze/puppet] 07paladox 039ee420b - Update jobrunner-hi.json.erb [13:52:37] [02puppet] 07paladox synchronize pull request 03#2468: jobrunner: Increase memory to 300M and time limit to 300 - 13https://github.com/miraheze/puppet/pull/2468 [13:52:45] [02puppet] 07paladox closed pull request 03#2468: jobrunner: Increase memory to 300M and time limit to 300 - 13https://github.com/miraheze/puppet/pull/2468 [13:52:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.01, 9.29, 8.89 [13:52:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.73, 9.89, 9.58 [13:52:56] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [13:52:58] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [13:53:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.46, 3.54, 3.17 [13:53:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.45, 3.55, 3.43 [13:55:17] [02puppet] 07RhinosF1 synchronize pull request 03#2469: jobrunner: allow higher memory for refreshLinks - 13https://github.com/miraheze/puppet/pull/2469 [13:55:40] [02puppet] 07paladox closed pull request 03#2469: jobrunner: allow higher memory for refreshLinks - 13https://github.com/miraheze/puppet/pull/2469 [13:55:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d898d57e02fd...e65b242b2bbc [13:55:43] [02miraheze/puppet] 07RhinosF1 03e65b242 - jobrunner: allow higher memory for refreshLinks (#2469) [13:57:00] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.11, 7.43, 8.34 [13:57:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.29, 4.23, 3.73 [13:57:39] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.03, 3.45, 3.22 [13:59:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.74, 3.27, 3.18 [13:59:48] PROBLEM - mw122 Puppet on mw122 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 25 minutes ago with 0 failures [14:00:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/425b9c624bdd [14:00:15] [02miraheze/puppet] 07paladox 03425b9c6 - jobrunner: Increase time limits for refreshLinks [14:00:16] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [14:00:19] [02puppet] 07paladox opened pull request 03#2470: jobrunner: Increase time limits for refreshLinks - 13https://github.com/miraheze/puppet/pull/2470 [14:00:21] PROBLEM - mw121 Puppet on mw121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 19 minutes ago with 0 failures [14:00:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/425b9c624bdd...8919764436a0 [14:00:46] [02miraheze/puppet] 07paladox 038919764 - Update jobrunner-hi.json.erb [14:00:47] [02puppet] 07paladox synchronize pull request 03#2470: jobrunner: Increase time limits for refreshLinks - 13https://github.com/miraheze/puppet/pull/2470 [14:00:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.84, 9.18, 8.85 [14:00:58] PROBLEM - mw112 Puppet on mw112 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 minutes ago with 0 failures [14:01:10] PROBLEM - mw111 Puppet on mw111 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 25 minutes ago with 0 failures [14:01:26] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:02:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.96, 9.31, 8.97 [14:02:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.47, 8.83, 8.76 [14:03:26] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.735 second response time [14:03:31] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [14:04:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.94, 8.94, 8.89 [14:04:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.42, 9.56, 9.39 [14:05:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.74, 4.00, 3.92 [14:05:41] [02puppet] 07RhinosF1 reviewed pull request 03#2470 commit - 13https://github.com/miraheze/puppet/pull/2470#discussion_r845176130 [14:05:43] [02puppet] 07RhinosF1 reviewed pull request 03#2470 commit - 13https://github.com/miraheze/puppet/pull/2470#discussion_r845177325 [14:05:44] [02puppet] 07RhinosF1 reviewed pull request 03#2470 commit - 13https://github.com/miraheze/puppet/pull/2470#discussion_r845177926 [14:05:46] [02puppet] 07RhinosF1 reviewed pull request 03#2470 commit - 13https://github.com/miraheze/puppet/pull/2470#discussion_r845178363 [14:06:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.62, 9.30, 9.01 [14:06:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.89, 9.15, 9.26 [14:07:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.26, 4.50, 4.13 [14:08:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.48, 8.47, 8.73 [14:08:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.59, 9.32, 9.39 [14:08:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.12, 9.33, 9.30 [14:09:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.99, 9.09, 8.55 [14:10:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.85, 9.10, 9.30 [14:10:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.48, 8.99, 9.18 [14:12:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.05, 9.34, 8.97 [14:13:57] [02puppet] 07paladox closed pull request 03#2470: jobrunner: Increase time limits for refreshLinks - 13https://github.com/miraheze/puppet/pull/2470 [14:13:59] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [14:14:01] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [14:14:27] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [14:14:56] RECOVERY - mw112 Puppet on mw112 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:14:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.74, 9.14, 8.95 [14:15:10] RECOVERY - mw111 Puppet on mw111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:15:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.57, 7.97, 8.26 [14:16:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.35, 9.36, 9.27 [14:18:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.29, 8.94, 9.13 [14:18:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.97, 9.65, 9.28 [14:20:00] .in 10mins check graph [14:20:00] RhinosF1: Okay, will remind at 2022-04-07 - 15:30:00BST [14:22:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.29, 9.47, 9.24 [14:22:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.13, 9.24, 9.25 [14:23:43] paladox: is that bad ^ [14:23:54] they just had jobs PR deployed [14:23:55] what's bad? [14:24:08] paladox: the load on mw111/2 that just alerted [14:24:20] it's been alerting for years [14:24:24] and yes that's bad [14:24:42] but not that we can do much about it, it needs more cpu resources [14:24:54] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.64, 7.95, 8.37 [14:24:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.93, 9.59, 9.32 [14:25:00] paladox: so not a new issue to worry about [14:25:06] nope [14:26:19] paladox: it's had some impact so far i think [14:26:26] ok [14:26:38] but running twice as fast when it's going at a snails pace it's fantastic [14:26:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.23, 7.65, 8.33 [14:27:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.66, 3.49, 3.93 [14:28:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.42, 9.70, 9.38 [14:29:16] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [14:30:00] RhinosF1: check graph [14:30:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.91, 9.43, 9.32 [14:30:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.81, 9.11, 9.04 [14:33:50] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.99, 8.51, 8.48 [14:34:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.52, 8.05, 8.09 [14:34:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.07, 10.27, 9.63 [14:34:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.13, 9.84, 9.41 [14:35:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.43, 3.49, 3.96 [14:35:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.73, 9.28, 8.77 [14:36:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.17, 9.70, 9.52 [14:38:26] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.71, 8.73, 7.94 [14:38:53] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.04, 6.80, 5.98 [14:38:54] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.28, 7.74, 8.00 [14:38:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.33, 9.68, 9.40 [14:40:26] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.06, 7.76, 7.67 [14:40:44] [02puppet] 07RhinosF1 commented on pull request 03#2465: jobs: move around - 13https://github.com/miraheze/puppet/pull/2465#issuecomment-1091823716 [14:40:47] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.99, 6.29, 5.88 [14:40:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.47, 9.65, 9.43 [14:41:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/56c6977b3453 [14:41:12] [02miraheze/puppet] 07paladox 0356c6977 - Switch elasticsearch to rsyslog [14:41:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/56c6977b3453...70669e38ed56 [14:41:40] [02miraheze/puppet] 07paladox 0370669e3 - Update es111.yaml [14:41:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/70669e38ed56...413bf7e7faea [14:41:51] [02miraheze/puppet] 07paladox 03413bf7e - Update es121.yaml [14:42:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/e65b242b2bbc...b0d36def03e1 [14:42:03] [02miraheze/puppet] 07paladox 03b0d36de - Switch elasticsearch to rsyslog (#2471) [14:42:04] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [14:42:11] [02puppet] 07paladox opened pull request 03#2471: Switch elasticsearch to rsyslog - 13https://github.com/miraheze/puppet/pull/2471 [14:42:34] PROBLEM - grayravens.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'grayravens.com' expires in 15 day(s) (Sat 23 Apr 2022 14:38:02 GMT +0000). [14:42:44] [02puppet] 07paladox closed pull request 03#2471: Switch elasticsearch to rsyslog - 13https://github.com/miraheze/puppet/pull/2471 [14:42:50] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [14:43:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.74, 3.04, 3.39 [14:43:15] [02puppet] 07paladox synchronize pull request 03#2471: Switch elasticsearch to rsyslog - 13https://github.com/miraheze/puppet/pull/2471 [14:43:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.20, 3.59, 3.70 [14:43:25] [02puppet] 07paladox synchronize pull request 03#2471: Switch elasticsearch to rsyslog - 13https://github.com/miraheze/puppet/pull/2471 [14:43:33] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [14:43:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e5cb0b58d533...9bd39626bb6e [14:43:38] [02miraheze/ssl] 07MirahezeSSLBot 039bd3962 - Bot: Update SSL cert for grayravens.com [14:46:25] PROBLEM - es101 Puppet on es101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:46:31] PROBLEM - es121 Puppet on es121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:48:32] PROBLEM - es111 Puppet on es111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:50:25] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [14:52:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/f4caf9405497 [14:52:47] [02miraheze/puppet] 07paladox 03f4caf94 - Make ssl::wildcard a define rather then a class [14:52:48] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [14:52:50] [02puppet] 07paladox opened pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472 [14:53:11] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.80, 9.93, 9.81 [14:54:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f4caf9405497...a9c04718d046 [14:54:45] [02miraheze/puppet] 07paladox 03a9c0471 - Update elasticsearch.pp [14:54:47] [02puppet] 07paladox synchronize pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472 [14:55:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.83, 3.82, 4.00 [14:55:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a9c04718d046...5f8fa3ab72e9 [14:55:58] [02miraheze/puppet] 07paladox 035f8fa3a - Update server.pp [14:56:00] [02puppet] 07paladox synchronize pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472 [14:56:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5f8fa3ab72e9...882a636327c6 [14:56:30] [02miraheze/puppet] 07paladox 03882a636 - Update nginx.pp [14:56:32] [02puppet] 07paladox synchronize pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472 [14:56:45] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [14:56:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/882a636327c6...9bddcf251eed [14:56:55] [02miraheze/puppet] 07paladox 039bddcf2 - Update nginx.pp [14:56:57] [02puppet] 07paladox synchronize pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472 [14:57:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9bddcf251eed...f959ea9f21c8 [14:57:07] [02miraheze/puppet] 07paladox 03f959ea9 - Update nginx.pp [14:57:09] [02puppet] 07paladox synchronize pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472 [14:57:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f959ea9f21c8...bb55b960070e [14:57:34] [02miraheze/puppet] 07paladox 03bb55b96 - Update init.pp [14:57:36] [02puppet] 07paladox synchronize pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472 [14:58:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bb55b960070e...1642fd51f36e [14:58:03] [02miraheze/puppet] 07paladox 031642fd5 - Update init.pp [14:58:05] [02puppet] 07paladox synchronize pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472 [14:58:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1642fd51f36e...565598007c7c [14:58:37] [02miraheze/puppet] 07paladox 035655980 - Update init.pp [14:58:38] [02puppet] 07paladox synchronize pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472 [14:59:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.03, 4.53, 3.70 [14:59:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/565598007c7c...1b568094798b [14:59:20] [02miraheze/puppet] 07paladox 031b56809 - Update init.pp [14:59:21] [02puppet] 07paladox synchronize pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472 [14:59:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1b568094798b...1dd33e310e30 [14:59:44] [02miraheze/puppet] 07paladox 031dd33e3 - Update init.pp [14:59:45] [02puppet] 07paladox synchronize pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472 [15:00:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1dd33e310e30...7a0477299649 [15:00:09] [02miraheze/puppet] 07paladox 037a04772 - Update graylog.pp [15:00:10] [02puppet] 07paladox synchronize pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472 [15:00:21] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:00:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7a0477299649...0ab25fb5df0b [15:00:32] [02miraheze/puppet] 07paladox 030ab25fb - Update init.pp [15:00:34] [02puppet] 07paladox synchronize pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472 [15:00:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0ab25fb5df0b...b6a8c2cdd2c2 [15:00:50] [02miraheze/puppet] 07paladox 03b6a8c2c - Update db.pp [15:00:52] [02puppet] 07paladox synchronize pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472 [15:01:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b6a8c2cdd2c2...98b070bebaa0 [15:01:15] [02miraheze/puppet] 07paladox 0398b070b - Update init.pp [15:01:16] [02puppet] 07paladox synchronize pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472 [15:01:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.95, 3.96 [15:01:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/98b070bebaa0...0ef126fa84bc [15:01:44] [02miraheze/puppet] 07paladox 030ef126f - Update init.pp [15:01:45] [02puppet] 07paladox synchronize pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472 [15:01:48] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [15:01:54] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.38, 7.93, 6.82 [15:02:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.58, 7.69, 8.34 [15:02:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.09, 8.97, 8.95 [15:03:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0ef126fa84bc...8774f4ae2b39 [15:03:32] [02miraheze/puppet] 07paladox 038774f4a - Update syslog.pp [15:03:33] [02puppet] 07paladox synchronize pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472 [15:03:48] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.69, 7.72, 6.87 [15:03:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.51, 8.13, 7.82 [15:04:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:04:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8774f4ae2b39...ce62fdbc5915 [15:04:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.42, 9.14, 9.02 [15:05:00] [02miraheze/puppet] 07paladox 03ce62fdb - Update syslog.pp [15:05:02] [02puppet] 07paladox synchronize pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472 [15:05:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.92, 3.71 [15:05:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.89, 3.69, 3.87 [15:05:42] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 10.98, 9.02, 7.45 [15:05:52] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/logrotate.d/nginx] [15:05:53] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.84, 8.10, 7.86 [15:06:25] RECOVERY - es101 Puppet on es101 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [15:07:01] PROBLEM - wiki.parappa.dev - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.parappa.dev' expires in 15 day(s) (Sat 23 Apr 2022 14:36:28 GMT +0000). [15:07:10] PROBLEM - mw111 Puppet on mw111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:07:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.94, 4.13, 3.80 [15:07:49] [02puppet] 07paladox closed pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472 [15:07:50] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [15:07:52] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [15:07:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±16] 13https://github.com/miraheze/puppet/compare/b0d36def03e1...b43333b2bf79 [15:07:55] [02miraheze/puppet] 07paladox 03b43333b - Make ssl::wildcard a define rather then a class (#2472) [15:08:32] RECOVERY - es111 Puppet on es111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:10:31] RECOVERY - es121 Puppet on es121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:10:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.92, 7.36, 8.44 [15:11:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.51, 3.62, 3.72 [15:11:20] PROBLEM - db101 APT on db101 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [15:11:29] RECOVERY - grayravens.com - LetsEncrypt on sslhost is OK: OK - Certificate 'grayravens.com' will expire on Wed 06 Jul 2022 13:43:28 GMT +0000. [15:12:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9bd39626bb6e...be6a6dd882a4 [15:12:12] [02miraheze/ssl] 07MirahezeSSLBot 03be6a6dd - Bot: Update SSL cert for wiki.parappa.dev [15:12:21] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:12:55] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [15:12:55] PROBLEM - mw102 Puppet on mw102 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:12:57] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:12:59] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.15, 7.28, 8.24 [15:13:12] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:16] i'm looking at puppet [15:13:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.38, 3.35, 3.63 [15:13:21] PROBLEM - ldap111 Puppet on ldap111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:13:29] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:43] paladox: Cannot alias File[wildcard.miraheze.org-2020-2] to ["/etc/ssl/localcerts/wildcard.miraheze.org-2020-2.crt"] [15:13:50] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:50] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:59] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:14:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:14:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/b3f7f78044f9 [15:14:26] [02miraheze/puppet] 07paladox 03b3f7f78 - cloud: switch to rsyslog [15:14:28] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [15:14:29] [02puppet] 07paladox opened pull request 03#2473: cloud: switch to rsyslog - 13https://github.com/miraheze/puppet/pull/2473 [15:14:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b3f7f78044f9...22b5e3e627c0 [15:14:44] [02miraheze/puppet] 07paladox 0322b5e3e - Update cloud11.yaml [15:14:45] [02puppet] 07paladox synchronize pull request 03#2473: cloud: switch to rsyslog - 13https://github.com/miraheze/puppet/pull/2473 [15:14:49] [02puppet] 07RhinosF1 commented on pull request 03#2472: Make ssl::wildcard a define rather then a class - 13https://github.com/miraheze/puppet/pull/2472#issuecomment-1091865341 [15:14:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/22b5e3e627c0...f285d975e4d4 [15:14:52] [02miraheze/puppet] 07paladox 03f285d97 - Update cloud12.yaml [15:14:53] [02puppet] 07paladox synchronize pull request 03#2473: cloud: switch to rsyslog - 13https://github.com/miraheze/puppet/pull/2473 [15:15:01] [02puppet] 07paladox closed pull request 03#2473: cloud: switch to rsyslog - 13https://github.com/miraheze/puppet/pull/2473 [15:15:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/b43333b2bf79...e13b6736496f [15:15:04] [02miraheze/puppet] 07paladox 03e13b673 - cloud: switch to rsyslog (#2473) [15:15:05] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.597 second response time [15:15:06] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [15:15:07] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [15:15:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.63, 3.69 [15:15:57] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.428 second response time [15:15:57] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.186 second response time [15:15:59] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23332 bytes in 7.039 second response time [15:16:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e13b6736496f...d7cb6aa7dd10 [15:17:00] [02miraheze/puppet] 07paladox 03d7cb6aa - openldap: Fix ssl::wildcard [15:17:01] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [15:17:23] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.991 second response time [15:17:27] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:17:38] RhinosF1: what file [15:17:42] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.074 second response time [15:18:13] paladox: that's exactly what the error says [15:18:24] https://github.com/miraheze/puppet/pull/2472#issuecomment-1091865341 [15:18:25] [url] Make ssl::wildcard a define rather then a class by paladox · Pull Request #2472 · miraheze/puppet · GitHub | github.com [15:18:52] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:19:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.32, 3.57, 3.64 [15:19:53] RECOVERY - db101 APT on db101 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [15:19:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/831eab9fe09e [15:20:00] [02miraheze/puppet] 07paladox 03831eab9 - Fix ssl::hiera::certs [15:20:01] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [15:20:03] [02puppet] 07paladox opened pull request 03#2474: Fix ssl::hiera::certs - 13https://github.com/miraheze/puppet/pull/2474 [15:20:56] PROBLEM - mw112 Puppet on mw112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:20:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.48, 8.74, 8.40 [15:21:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.77, 3.83, 3.72 [15:21:28] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [15:21:38] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:22:20] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:22:21] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [15:22:26] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:22:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.68, 9.06, 8.17 [15:23:30] [02puppet] 07JohnFLewis commented on pull request 03#2465: jobs: move around - 13https://github.com/miraheze/puppet/pull/2465#issuecomment-1091874578 [15:24:55] [02puppet] 07paladox closed pull request 03#2474: Fix ssl::hiera::certs - 13https://github.com/miraheze/puppet/pull/2474 [15:24:56] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [15:25:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d7cb6aa7dd10...f2d9ad58334c [15:25:01] [02miraheze/puppet] 07paladox 03f2d9ad5 - Fix ssl::hiera::certs (#2474) [15:25:03] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [15:25:45] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [15:26:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/d62f78f5dac4 [15:26:45] [02miraheze/puppet] 07paladox 03d62f78f - rsyslog::input::file: Fix syslog_tag_prefix param [15:26:47] [02puppet] 07paladox created branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [15:26:48] [02puppet] 07paladox opened pull request 03#2475: rsyslog::input::file: Fix syslog_tag_prefix param - 13https://github.com/miraheze/puppet/pull/2475 [15:27:57] [02puppet] 07RhinosF1 commented on pull request 03#2465: jobs: move around - 13https://github.com/miraheze/puppet/pull/2465#issuecomment-1091879291 [15:28:12] [02puppet] 07paladox synchronize pull request 03#2475: rsyslog::input::file: Fix syslog_tag_prefix param - 13https://github.com/miraheze/puppet/pull/2475 [15:28:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d62f78f5dac4...988d2f81af61 [15:28:15] [02miraheze/puppet] 07paladox 03988d2f8 - Update file.pp [15:30:39] [02puppet] 07paladox synchronize pull request 03#2475: rsyslog::input::file: Fix syslog_tag_prefix param - 13https://github.com/miraheze/puppet/pull/2475 [15:30:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/988d2f81af61...78b7103a54d9 [15:30:42] [02miraheze/puppet] 07paladox 0378b7103 - Update file.pp [15:31:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.50, 3.73, 3.89 [15:31:54] [02puppet] 07paladox synchronize pull request 03#2475: rsyslog::input::file: Fix syslog_tag_prefix param - 13https://github.com/miraheze/puppet/pull/2475 [15:31:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/78b7103a54d9...4ee5ccb690b9 [15:31:57] [02miraheze/puppet] 07paladox 034ee5ccb - Update file.pp [15:33:27] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [15:33:32] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:34:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.40, 9.73, 9.24 [15:35:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4ee5ccb690b9...952d9ed82671 [15:35:08] [02miraheze/puppet] 07paladox 03952d9ed - Update file.erb [15:35:09] [02puppet] 07paladox synchronize pull request 03#2475: rsyslog::input::file: Fix syslog_tag_prefix param - 13https://github.com/miraheze/puppet/pull/2475 [15:35:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.14, 4.07, 3.95 [15:35:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.00, 8.63, 8.21 [15:35:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/952d9ed82671...f011c183f902 [15:35:54] [02miraheze/puppet] 07paladox 03f011c18 - Update file.erb [15:35:55] [02puppet] 07paladox synchronize pull request 03#2475: rsyslog::input::file: Fix syslog_tag_prefix param - 13https://github.com/miraheze/puppet/pull/2475 [15:35:58] RECOVERY - wiki.parappa.dev - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.parappa.dev' will expire on Wed 06 Jul 2022 14:12:03 GMT +0000. [15:36:15] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [15:36:42] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 3.86, 4.05, 7.91 [15:36:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.32, 10.35, 9.52 [15:36:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.20, 9.88, 9.71 [15:37:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.68, 3.82 [15:38:39] [02puppet] 07paladox edited pull request 03#2475: rsyslog::input::file: Fix define - 13https://github.com/miraheze/puppet/pull/2475 [15:38:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 9.56, 10.02, 9.80 [15:40:42] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 2.55, 3.31, 6.73 [15:40:43] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [15:41:22] RECOVERY - ldap111 Puppet on ldap111 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [15:42:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.29, 8.63, 8.17 [15:42:21] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:43:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.57, 3.62, 3.70 [15:46:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.10, 9.69, 9.82 [15:47:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.70, 9.52, 8.68 [15:48:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.41, 9.80, 9.78 [15:49:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.72, 9.14, 8.63 [15:50:20] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [15:51:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f011c183f902...08cc33c97863 [15:51:29] [02miraheze/puppet] 07paladox 0308cc33c - Update syslog.pp [15:51:30] [02puppet] 07paladox synchronize pull request 03#2475: rsyslog::input::file: Fix define - 13https://github.com/miraheze/puppet/pull/2475 [15:51:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.25, 8.25, 8.38 [15:52:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.30, 9.76, 9.75 [15:53:19] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [15:53:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/08cc33c97863...5a75fd75f7ee [15:53:40] [02miraheze/puppet] 07paladox 035a75fd7 - Create remote_syslog_rule.conf.erb [15:53:42] [02puppet] 07paladox synchronize pull request 03#2475: rsyslog::input::file: Fix define - 13https://github.com/miraheze/puppet/pull/2475 [15:54:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:54:21] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:54:42] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [15:54:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.01, 8.17, 8.43 [15:54:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.98, 9.06, 9.49 [15:55:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/5a75fd75f7ee...6759fe805184 [15:55:16] [02miraheze/puppet] 07paladox 036759fe8 - Create remote_syslog_rule_parse_json.conf.erb [15:55:17] [02puppet] 07paladox synchronize pull request 03#2475: rsyslog::input::file: Fix define - 13https://github.com/miraheze/puppet/pull/2475 [15:55:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.46, 9.13, 8.75 [15:55:57] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.76, 1.71, 1.13 [15:56:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.20, 10.15, 9.82 [15:56:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.96, 10.13, 9.85 [15:57:35] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:57:46] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [15:57:56] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.49, 1.24, 1.03 [15:58:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6759fe805184...50bc7ee2bec0 [15:58:21] [02miraheze/puppet] 07paladox 0350bc7ee - Update rsyslog_udp_localhost.conf.erb [15:58:22] [02puppet] 07paladox synchronize pull request 03#2475: rsyslog::input::file: Fix define - 13https://github.com/miraheze/puppet/pull/2475 [15:59:01] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:00:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.66, 9.33, 9.58 [16:00:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.03, 9.48, 9.69 [16:01:54] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [16:02:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/50bc7ee2bec0...9bcab9f63441 [16:02:55] [02miraheze/puppet] 07paladox 039bcab9f - Update file.erb [16:02:56] [02puppet] 07paladox synchronize pull request 03#2475: rsyslog::input::file: Fix define - 13https://github.com/miraheze/puppet/pull/2475 [16:02:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.88, 8.64, 8.49 [16:02:57] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.91, 9.47, 8.97 [16:02:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.18, 10.16, 9.91 [16:03:56] [02puppet] 07paladox synchronize pull request 03#2475: rsyslog::input::file: Fix define - 13https://github.com/miraheze/puppet/pull/2475 [16:03:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9bcab9f63441...d6929fe3a1dc [16:03:59] [02miraheze/puppet] 07paladox 03d6929fe - Update syslog.pp [16:04:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:04:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.02, 9.12, 8.90 [16:04:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.43, 9.58, 9.73 [16:05:04] [02puppet] 07paladox synchronize pull request 03#2475: rsyslog::input::file: Fix define - 13https://github.com/miraheze/puppet/pull/2475 [16:05:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d6929fe3a1dc...a5aa47aaf515 [16:05:07] [02miraheze/puppet] 07paladox 03a5aa47a - Update file.pp [16:05:33] [02puppet] 07paladox synchronize pull request 03#2475: rsyslog::input::file: Fix define - 13https://github.com/miraheze/puppet/pull/2475 [16:05:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a5aa47aaf515...52224637a612 [16:05:36] [02miraheze/puppet] 07paladox 035222463 - Update file.pp [16:06:20] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [16:06:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.46, 9.71, 9.55 [16:06:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.16, 9.46, 8.81 [16:08:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.65, 9.62, 8.96 [16:08:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.42, 9.71, 9.70 [16:10:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.35, 9.60, 9.15 [16:10:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.93, 9.85, 9.64 [16:10:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.16, 11.04, 9.56 [16:12:38] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [16:12:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.44, 9.15, 9.04 [16:12:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.99, 10.54, 9.91 [16:12:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.90, 9.83, 9.75 [16:13:00] [02puppet] 07paladox closed pull request 03#2475: rsyslog::input::file: Fix define - 13https://github.com/miraheze/puppet/pull/2475 [16:13:01] [02puppet] 07paladox deleted branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [16:13:03] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-9 [16:13:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±4] 13https://github.com/miraheze/puppet/compare/f2d9ad58334c...9b615dee3037 [16:13:06] [02miraheze/puppet] 07paladox 039b615de - rsyslog::input::file: Fix define (#2475) [16:14:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.91, 9.70, 9.39 [16:14:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.76, 10.43, 9.97 [16:17:01] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [16:17:04] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [16:18:21] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [16:18:55] RECOVERY - mw102 Puppet on mw102 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [16:18:56] RECOVERY - mw112 Puppet on mw112 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [16:19:10] RECOVERY - mw111 Puppet on mw111 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [16:19:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.86, 3.42, 3.93 [16:19:32] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:19:59] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:20:11] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.85, 8.26, 7.86 [16:20:52] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:21:38] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:22:11] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.33, 8.15, 7.86 [16:22:21] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:22:26] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [16:23:22] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [16:25:35] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [16:26:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.50, 9.18, 9.76 [16:27:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.22, 3.98, 3.89 [16:27:48] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [16:28:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.40, 9.67, 9.88 [16:28:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.14, 10.45, 9.65 [16:28:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.30, 3.38, 3.00 [16:30:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.09, 8.80, 9.54 [16:30:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.90, 3.19, 2.98 [16:34:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.24, 9.49, 9.59 [16:34:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.80, 9.68, 9.66 [16:38:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.34, 9.13, 9.44 [16:40:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.27, 9.27, 9.43 [16:40:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.75, 9.52, 9.51 [16:42:44] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+1/-0/±1] 13https://github.com/miraheze/mediawiki/compare/deeade00bb92...c90ed54a3a0e [16:42:46] [02miraheze/mediawiki] 07Universal-Omega 03c90ed54 - Install HitCounters [16:42:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.41, 8.92, 9.31 [16:43:58] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+1/-0/±1] 13https://github.com/miraheze/mediawiki/compare/33ca2e8525e8...0bd8bbe30daa [16:43:59] [02miraheze/mediawiki] 07Universal-Omega 030bd8bbe - Install HitCounters [16:44:42] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [16:44:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.71, 9.97, 9.73 [16:46:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.20, 9.69, 9.08 [16:46:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.72, 9.47, 9.42 [16:46:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.37, 10.02, 9.77 [16:47:03] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+1/-0/±1] 13https://github.com/miraheze/mediawiki/compare/c90ed54a3a0e...8c551728482f [16:47:04] [02miraheze/mediawiki] 07Universal-Omega 038c55172 - Install UploadWizard [16:48:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.40, 9.37, 9.05 [16:48:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.33, 9.49, 9.61 [16:48:58] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+1/-0/±1] 13https://github.com/miraheze/mediawiki/compare/0bd8bbe30daa...6d944d9b305b [16:49:00] [02miraheze/mediawiki] 07Universal-Omega 036d944d9 - Install UploadWizard [16:49:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.51, 8.20, 7.98 [16:49:09] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [16:50:19] !log [universalomega@mwtask111] starting deploy of {'world': True, 'l10n': True, 'extensionlist': True} to all [16:50:50] !log [universalomega@test101] starting deploy of {'world': True, 'extensionlist': True} to all [16:50:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.74, 9.78, 9.23 [16:50:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.23, 9.19, 9.34 [16:50:57] !log [universalomega@mwtask111] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [16:50:59] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.77, 8.04, 7.95 [16:51:03] !log [universalomega@mwtask111] starting deploy of {'world': True, 'extensionlist': True} to all [16:51:15] [02puppet] 07JohnFLewis commented on pull request 03#2465: jobs: move around - 13https://github.com/miraheze/puppet/pull/2465#issuecomment-1091978006 [16:52:47] [02mw-config] 07Universal-Omega closed pull request 03#4552: set repoSiteId for benpediawiki - 13https://github.com/miraheze/mw-config/pull/4552 [16:52:49] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/64a15cd227ac...18a1de3b528b [16:52:50] [02miraheze/mw-config] 07ugochimobi 0318a1de3 - set repoSiteId for benpediawiki (#4552) [16:52:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.13, 9.58, 9.23 [16:52:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.36, 9.68, 9.64 [16:54:02] miraheze/mw-config - Universal-Omega the build passed. [16:54:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.81, 9.73, 9.66 [16:54:58] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [16:54:58] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:56:17] !log [universalomega@test101] finished deploy of {'world': True, 'extensionlist': True} to all - SUCCESS in 326s [16:56:20] !log [@mwtask111] starting deploy of {'config': True} to all [16:56:46] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 26s [16:56:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.02, 9.43, 9.21 [16:56:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.74, 9.91, 9.72 [16:56:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.29, 8.55, 8.12 [16:58:53] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [16:58:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.41, 9.42, 9.22 [16:58:55] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [16:58:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.98, 9.38, 9.55 [16:58:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.29, 8.16, 8.04 [17:00:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.28, 9.76, 9.67 [17:00:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.67, 9.23, 9.89 [17:01:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.62, 3.98 [17:01:49] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [17:02:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.44, 8.98, 8.48 [17:02:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.19, 9.11, 9.14 [17:02:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.45, 9.83, 9.72 [17:02:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.38, 10.27, 10.18 [17:04:22] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.10, 3.63, 3.21 [17:04:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.93, 8.37, 8.30 [17:06:18] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.42, 3.45, 3.19 [17:06:32] !log [universalomega@mwtask111] finished deploy of {'world': True, 'extensionlist': True} to all - SUCCESS in 928s [17:06:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.79, 9.44, 9.25 [17:06:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.62, 9.83, 9.47 [17:06:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.58, 9.88, 9.72 [17:06:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.09, 9.65, 9.97 [17:08:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.60, 3.18, 3.13 [17:08:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.15, 8.89, 8.54 [17:08:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.09, 10.33, 9.69 [17:08:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.99, 8.92, 9.38 [17:08:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.56, 9.76, 9.95 [17:09:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.59, 3.73, 3.74 [17:10:51] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.51, 9.21, 8.68 [17:10:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.30, 10.04, 9.72 [17:10:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.04, 9.63, 9.90 [17:12:11] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.13, 3.61, 3.30 [17:12:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.89, 9.63, 9.56 [17:12:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.10, 9.64, 9.86 [17:14:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.26, 3.60, 3.34 [17:14:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.56, 9.28, 8.86 [17:14:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.78, 9.43, 9.58 [17:14:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.85, 9.66, 9.84 [17:16:04] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.48, 3.19, 3.22 [17:16:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.57, 9.81, 9.86 [17:18:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.79, 9.48, 9.63 [17:20:00] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.20, 3.83, 3.49 [17:20:48] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.87, 8.11, 8.47 [17:21:02] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:21:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.76, 3.52, 3.93 [17:22:59] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23332 bytes in 0.517 second response time [17:23:55] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [17:24:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.70, 8.50, 8.59 [17:25:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.59, 3.78, 3.92 [17:25:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.11, 3.71, 3.60 [17:26:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.32, 7.64, 8.26 [17:27:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.49, 4.63, 3.93 [17:28:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.73, 9.56, 9.90 [17:29:43] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.81, 3.90, 3.75 [17:30:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.37, 9.37, 9.29 [17:30:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.52, 7.65, 8.47 [17:31:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.64, 3.67, 3.90 [17:32:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.17, 9.24, 9.25 [17:32:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.76, 9.88, 9.88 [17:35:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.95, 9.30, 9.66 [17:35:32] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.03, 2.90, 3.39 [17:37:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.79, 3.40, 3.92 [17:38:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.74, 9.50, 9.19 [17:39:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.01, 3.42, 3.55 [17:39:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 3.97, 4.08 [17:40:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.62, 9.33, 9.18 [17:41:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.79, 8.75, 8.56 [17:43:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 3.71, 3.65 [17:43:39] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.28, 9.69, 8.92 [17:44:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.29, 9.13, 9.13 [17:44:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.29, 9.52, 9.42 [17:45:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.34, 8.76, 8.67 [17:46:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.58, 8.68, 8.97 [17:46:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.68, 9.32, 9.34 [17:47:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.19, 2.77, 3.27 [17:48:46] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.29, 1.15, 1.93 [17:48:54] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.17, 7.86, 8.47 [17:48:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.26, 9.66, 9.46 [17:51:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.06, 2.99 [17:51:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.02, 4.33, 3.80 [17:52:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.56, 9.19, 9.35 [17:53:04] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.35, 3.43, 3.13 [17:53:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.09, 8.03, 8.45 [17:54:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:54:42] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/1df29d67c4d4 [17:54:43] [02miraheze/mw-config] 07Universal-Omega 031df29d6 - add HitCounters and UploadWizard to extension-list [17:54:45] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [17:54:46] [02mw-config] 07Universal-Omega opened pull request 03#4554: add HitCounters and UploadWizard to extension-list - 13https://github.com/miraheze/mw-config/pull/4554 [17:55:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.78, 3.07, 3.03 [17:55:59] miraheze/mw-config - Universal-Omega the build has errored. [17:56:05] [02mw-config] 07Universal-Omega synchronize pull request 03#4554: add HitCounters and UploadWizard to extension-list - 13https://github.com/miraheze/mw-config/pull/4554 [17:56:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1df29d67c4d4...90c5fec1dfe3 [17:56:08] [02miraheze/mw-config] 07Universal-Omega 0390c5fec - Update extension-list [17:56:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.41, 9.52, 9.42 [17:57:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.39, 3.49, 3.67 [17:57:13] [02mw-config] 07Universal-Omega synchronize pull request 03#4554: add HitCounters and UploadWizard to extension-list - 13https://github.com/miraheze/mw-config/pull/4554 [17:57:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/90c5fec1dfe3...743c2395cbf3 [17:57:16] miraheze/mw-config - Universal-Omega the build has errored. [17:57:16] [02miraheze/mw-config] 07Universal-Omega 03743c239 - Update extension-list [17:58:21] miraheze/mw-config - Universal-Omega the build passed. [17:58:22] [02mw-config] 07Universal-Omega edited pull request 03#4554: Add HitCounters and UploadWizard to extension-list - 13https://github.com/miraheze/mw-config/pull/4554 [17:58:34] [02mw-config] 07Universal-Omega closed pull request 03#4554: Add HitCounters and UploadWizard to extension-list - 13https://github.com/miraheze/mw-config/pull/4554 [17:58:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/18a1de3b528b...3035adc8caef [17:58:37] [02miraheze/mw-config] 07Universal-Omega 033035adc - Add HitCounters and UploadWizard to extension-list (#4554) [17:58:38] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [17:58:40] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [17:58:46] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.35, 1.34, 1.69 [17:58:48] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [17:58:49] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [17:58:59] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [17:59:09] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.25, 9.00, 8.68 [17:59:13] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 13s [17:59:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.14, 3.39, 3.92 [17:59:33] !log [universalomega@mwtask111] starting deploy of {'l10n': True} to all [17:59:37] !log [universalomega@test101] starting deploy of {'l10n': True} to all [17:59:37] miraheze/mw-config - Universal-Omega the build passed. [18:01:05] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.64, 9.41, 8.89 [18:01:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.20, 9.46, 8.86 [18:02:46] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.46, 1.98, 1.87 [18:02:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.81, 9.59, 9.55 [18:03:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 4.28, 3.96 [18:03:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.66, 4.18, 4.09 [18:03:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.72, 8.72, 8.63 [18:04:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:04:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.54, 9.07, 8.92 [18:06:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.84, 9.12, 8.97 [18:06:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.60, 7.87, 8.37 [18:07:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.16, 3.47, 3.71 [18:07:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.80, 3.98 [18:08:06] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [18:09:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:09:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.88, 4.02, 4.05 [18:10:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.71, 9.43, 9.07 [18:10:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.50, 8.52, 8.56 [18:10:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.83, 10.21, 9.66 [18:11:02] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.62, 8.21, 8.47 [18:14:05] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/1d3d0065b5dd [18:14:07] [02miraheze/mw-config] 07Universal-Omega 031d3d006 - Add HitCounters to ManageWikiExtensions [18:14:08] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [18:14:11] [02mw-config] 07Universal-Omega opened pull request 03#4555: Add HitCounters to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4555 [18:14:22] !log depool mw101 and mw102, increase cpu by 1 and reboot and repool [18:14:53] !log [universalomega@test101] finished deploy of {'l10n': True} to all - SUCCESS in 915s [18:14:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.70, 9.39, 9.16 [18:15:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.15, 2.61, 3.26 [18:15:23] miraheze/mw-config - Universal-Omega the build passed. [18:15:35] [02mw-config] 07Universal-Omega edited pull request 03#4555: T8966: Add HitCounters to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4555 [18:16:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.29, 7.87, 8.33 [18:17:58] what happened to MirahezeLogbot? it does not seem to be logging. [18:18:53] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:19:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:19:07] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:19:09] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:19:10] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:19:23] !log set aes flag to enabled on mw101/102 [18:19:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.86, 3.05, 2.91 [18:19:53] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:42] PROBLEM - mw101 Check Gluster Clients on mw101 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [18:20:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.91, 8.48, 8.45 [18:20:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.87, 9.32, 9.12 [18:22:37] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw101 [18:22:48] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw101 [18:22:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.71, 9.06, 9.05 [18:22:57] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw101 [18:22:58] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [18:23:16] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw101 [18:23:25] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.72, 2.62, 2.79 [18:24:37] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [18:24:42] RECOVERY - mw101 Check Gluster Clients on mw101 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [18:24:48] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [18:24:58] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [18:24:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.74, 9.41, 9.76 [18:25:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [18:25:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.69, 3.26, 4.00 [18:25:20] !log depool mw111 and mw112, increase cpu by 1, set aes flag to enabled and reboot and repool [18:25:45] !log depool mwtask111, increase cpu by 1, set aes flag to enabled and reboot and repool [18:26:05] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23332 bytes in 1.745 second response time [18:26:54] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.78, 7.87, 8.27 [18:26:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 1.68, 0.35, 0.11 [18:27:08] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [18:27:12] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.473 second response time [18:27:19] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.141 second response time [18:27:23] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.792 second response time [18:27:28] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [18:28:46] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.20, 0.30, 0.10 [18:28:59] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 2.80, 0.67, 0.22 [18:29:14] !log depool mw121 and mw122, increase cpu by 1, set aes flag to enabled and reboot and repool [18:30:39] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [18:37:46] !log restarted logbot as it wasn't logging [18:39:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.04, 2.70, 3.29 [18:39:49] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/5f6b69c04b65 [18:39:51] [02miraheze/mw-config] 07Universal-Omega 035f6b69c - Add UploadWizard to ManageWikiExtensions [18:39:52] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [18:39:54] [02mw-config] 07Universal-Omega opened pull request 03#4556: Add UploadWizard to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4556 [18:41:00] miraheze/mw-config - Universal-Omega the build passed. [18:41:14] paladox: restart was 100% ineffective :P [18:41:26] :P [18:49:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:50:32] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5f6b69c04b65...790af6921511 [18:50:34] [02miraheze/mw-config] 07Universal-Omega 03790af69 - Update LocalSettings.php [18:50:35] [02mw-config] 07Universal-Omega synchronize pull request 03#4556: Add UploadWizard to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4556 [18:50:53] [02mw-config] 07Universal-Omega edited pull request 03#4556: T8967: Add UploadWizard to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4556 [18:51:53] miraheze/mw-config - Universal-Omega the build passed. [18:52:07] [02mw-config] 07Universal-Omega synchronize pull request 03#4556: T8967: Add UploadWizard to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4556 [18:52:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/790af6921511...31313d6afecd [18:52:10] [02miraheze/mw-config] 07Universal-Omega 0331313d6 - Update ManageWikiExtensions.php [18:53:24] miraheze/mw-config - Universal-Omega the build passed. [18:55:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.28, 3.60, 3.49 [18:56:35] !log [universalomega@mwtask111] starting deploy of {'l10n': True} to all [18:58:15] !log [macfan@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/resetAuthenticationThrottle.php --wiki=metawiki --user=MirahezeLogbot --ip=2a10:6740::6:205 (END - exit=256) [18:58:38] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.22, 3.62, 3.12 [18:58:40] !log [macfan@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/resetAuthenticationThrottle.php --wiki=metawiki --user=MirahezeLogbot --ip=2a10:6740::6:205 --login (END - exit=0) [18:58:53] !log test [18:59:21] anyway, throttle is reset so should start working at some point [19:00:17] paladox: ^ [19:00:21] !log test [19:00:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.87, 3.21, 3.02 [19:00:39] hmm [19:00:53] that ip that is used is for mon111 to clarify [19:01:09] https://beta.betaheze.org/wiki/ seems to be failing [19:01:10] [url] Betaheze | beta.betaheze.org [19:01:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.65, 4.08, 3.69 [19:01:32] oh damn [19:01:44] paladox: i just loaded it without issue [19:01:46] it really failed because of that var either though it outputed [19:02:30] !log [universalomega@test101] starting deploy of {'config': True} to all [19:02:31] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [19:03:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.19, 4.21, 3.52 [19:05:14] !log [universalomega@mwtask111] finished deploy of {'l10n': True} to all - SUCCESS in 518s [19:05:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.71, 3.95, 3.74 [19:07:17] CosmicAlpha: i can use RequestContext in LS [19:07:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.46, 4.49, 3.95 [19:07:33] Great! [19:07:53] Yeah I was not sure if it would give warnings if used at that stage. [19:08:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.84, 3.85, 3.36 [19:08:52] !log [universalomega@test101] starting deploy of {'config': True} to all [19:08:53] !log [universalomega@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [19:09:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.95, 3.81 [19:10:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 3.91, 3.44 [19:10:10] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/f1e471f018bf [19:10:12] [02miraheze/mw-config] 07paladox 03f1e471f - Exempt MirahezeLogbot from login throttle [19:10:14] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [19:10:15] [02mw-config] 07paladox opened pull request 03#4557: Exempt MirahezeLogbot from login throttle - 13https://github.com/miraheze/mw-config/pull/4557 [19:10:20] CosmicAlpha: MacFan4000 ^ [19:11:30] miraheze/mw-config - paladox the build passed. [19:12:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.74, 3.62, 3.39 [19:13:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.14, 3.87, 3.83 [19:14:06] ok : [RESOLVED] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:14:26] gonna merge and deploy [19:14:32] [02mw-config] 07paladox closed pull request 03#4557: Exempt MirahezeLogbot from login throttle - 13https://github.com/miraheze/mw-config/pull/4557 [19:14:34] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [19:14:35] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3035adc8caef...c1ef4f698405 [19:14:37] [02miraheze/mw-config] 07paladox 03c1ef4f6 - Exempt MirahezeLogbot from login throttle (#4557) [19:14:38] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [19:14:58] !log [paladox@test101] starting deploy of {'pull': 'config', 'config': True} to test101 [19:15:01] !log [paladox@test101] finished deploy of {'pull': 'config', 'config': True} to test101 - SUCCESS in 3s [19:15:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.39, 3.72, 3.71 [19:15:43] miraheze/mw-config - paladox the build passed. [19:15:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.31, 3.02, 3.21 [19:15:53] I got logged out of beta now and can't log in [19:16:09] !log [paladox@test101] starting deploy of {'pull': 'config', 'config': True} to test101 [19:16:10] !log [paladox@test101] finished deploy of {'pull': 'config', 'config': True} to test101 - SUCCESS in 1s [19:17:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.68, 3.70 [19:17:29] !log [paladox@mwtask111] starting deploy of {'config': True} to all [19:17:39] !log [paladox@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 9s [19:18:21] [discord] [dfda73f053d442807b9ff6be] 2022-04-07 19:17:55: Fatal exception of type "LogicException" [19:18:21] [discord] On Meta [19:18:29] [discord] https://meta.miraheze.org/wiki/Meta:Administrators%27_noticeboard#MediaWiki:Blockedtext [19:18:29] [url] Meta:Administrators' noticeboard - Miraheze Meta | meta.miraheze.org [19:18:33] !log test [19:18:50] I can't log in now to prod or beta [19:19:04] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 0.98, 2.56, 3.34 [19:19:09] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c1ef4f698405...52e3223f81a5 [19:19:11] [02miraheze/mw-config] 07paladox 0352e3223 - Revert "Exempt MirahezeLogbot from login throttle (#4557)" [19:19:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.21, 2.80, 3.38 [19:19:21] !log [paladox@mwtask111] starting deploy of {'config': True} to all [19:19:28] !log [paladox@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 6s [19:19:38] !log [@test101] starting deploy of {'config': True} to all [19:19:39] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [19:20:12] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.83, 1.95, 1.25 [19:20:15] miraheze/mw-config - paladox the build passed. [19:24:08] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.93, 1.40 [19:27:06] !log [macfan@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/resetAuthenticationThrottle.php --wiki=metawiki --user=MirahezeLogbot --ip=2a10:6740::6:205 --login (END - exit=0) [19:27:08] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.70, 4.36, 2.30 [19:27:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:22] paladox:: its fixed [19:27:33] oh [19:27:37] i ran the script a second time [19:27:54] didn't work the first time for whatever reason [19:29:20] [02mw-config] 07Universal-Omega closed pull request 03#4555: T8966: Add HitCounters to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4555 [19:29:22] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [19:29:23] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:29:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/52e3223f81a5...8b8ee3975e73 [19:29:26] [02miraheze/mw-config] 07Universal-Omega 038b8ee39 - T8966: Add HitCounters to ManageWikiExtensions (#4555) [19:29:36] [02mw-config] 07Universal-Omega closed pull request 03#4556: T8967: Add UploadWizard to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4556 [19:29:37] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [19:29:39] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 [19:29:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/8b8ee3975e73...783ce0b14463 [19:29:42] [02miraheze/mw-config] 07Universal-Omega 03783ce0b - T8967: Add UploadWizard to ManageWikiExtensions (#4556) [19:30:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.31, 1.66, 1.44 [19:30:31] miraheze/mw-config - Universal-Omega the build passed. [19:30:44] miraheze/mw-config - Universal-Omega the build passed. [19:31:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/9fd6d64c07a2 [19:31:47] [02miraheze/mw-config] 07Universal-Omega 039fd6d64 - Temporarily restrict HitCounters [19:31:49] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [19:31:50] [02mw-config] 07Universal-Omega opened pull request 03#4558: Temporarily restrict HitCounters - 13https://github.com/miraheze/mw-config/pull/4558 [19:32:23] [02mw-config] 07Universal-Omega closed pull request 03#4558: Temporarily restrict HitCounters - 13https://github.com/miraheze/mw-config/pull/4558 [19:32:25] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [19:32:26] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:32:27] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [19:32:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/783ce0b14463...b0c524cf2b99 [19:32:29] [02miraheze/mw-config] 07Universal-Omega 03b0c524c - Temporarily restrict HitCounters (#4558) [19:32:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:32:37] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [19:32:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:32:56] miraheze/mw-config - Universal-Omega the build passed. [19:33:08] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.62, 1.86, 1.82 [19:33:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.41, 3.46, 3.21 [19:33:35] miraheze/mw-config - Universal-Omega the build passed. [19:33:50] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [19:33:52] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [19:33:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:34:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:34:13] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [19:34:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:34:23] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [19:35:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.93, 3.37, 3.22 [19:37:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.13, 3.95, 3.50 [19:39:08] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.64, 1.26, 1.58 [19:39:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.91, 3.51 [19:42:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:43:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 3.84, 3.61 [19:45:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.70, 4.59, 3.91 [19:46:42] !log [universalomega@test101] starting deploy of {'folders': 'w/extensions/ManageWiki'} to all [19:46:43] !log [universalomega@test101] finished deploy of {'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 0s [19:47:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.11, 3.17, 3.38 [19:47:43] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.19, 1.89, 1.57 [19:49:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.78, 3.73 [19:51:39] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.14, 1.65, 1.57 [19:52:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:52:47] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/735d10bb4589...ea657e78f6e4 [19:52:48] [02miraheze/ManageWiki] 07Universal-Omega 03ea657e7 - Update ManageWikiTypes.php [19:52:50] [02ManageWiki] 07Universal-Omega synchronize pull request 03#347: Use texts type for `$wgNamespaceAliases` - 13https://github.com/miraheze/ManageWiki/pull/347 [19:53:00] !log [universalomega@test101] starting deploy of {'folders': 'w/extensions/ManageWiki'} to all [19:53:01] !log [universalomega@test101] finished deploy of {'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 0s [19:53:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.16, 3.93, 3.76 [19:55:42] miraheze/ManageWiki - Universal-Omega the build has errored. [19:57:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.35, 3.58, 3.41 [19:59:09] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/ea657e78f6e4...45498676aa66 [19:59:11] [02miraheze/ManageWiki] 07Universal-Omega 034549867 - Update en.json [19:59:12] [02ManageWiki] 07Universal-Omega synchronize pull request 03#347: Use texts type for `$wgNamespaceAliases` - 13https://github.com/miraheze/ManageWiki/pull/347 [19:59:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.67, 3.21, 3.29 [19:59:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.83, 3.60, 3.73 [19:59:40] !log [universalomega@test101] starting deploy of {'folders': 'w/extensions/ManageWiki'} to all [19:59:41] !log [universalomega@test101] finished deploy of {'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 0s [20:02:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:02:11] !log [universalomega@test101] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki=betawiki --lang=en (END - exit=0) [20:03:07] miraheze/ManageWiki - Universal-Omega the build has errored. [20:03:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.61, 3.51, 3.42 [20:03:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.20, 3.76, 3.76 [20:05:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.41, 4.50, 3.77 [20:05:17] [02ManageWiki] 07Universal-Omega synchronize pull request 03#347: Use texts type for `$wgNamespaceAliases` - 13https://github.com/miraheze/ManageWiki/pull/347 [20:05:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.15, 3.77, 3.78 [20:05:18] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/45498676aa66...225524eeda3c [20:05:20] [02miraheze/ManageWiki] 07Universal-Omega 03225524e - Update ManageWikiFormFactoryBuilder.php [20:06:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.94, 3.43, 3.17 [20:07:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.46, 3.92, 3.83 [20:07:53] !log [universalomega@test101] starting deploy of {'folders': 'w/extensions/ManageWiki'} to all [20:07:54] !log [universalomega@test101] finished deploy of {'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 0s [20:08:07] !log [universalomega@test101] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki=betawiki --lang=en (END - exit=0) [20:08:39] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [20:09:15] [02miraheze/mw-config] 07Universal-Omega pushed 0314 commits to 03Universal-Omega-patch-2 [+0/-0/±19] 13https://github.com/miraheze/mw-config/compare/39b9f6c7851b...d4354610395a [20:09:16] [02miraheze/mw-config] 07Universal-Omega 03d435461 - Merge branch 'master' into Universal-Omega-patch-2 [20:09:18] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [20:09:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.40, 3.80, 3.80 [20:09:52] !log [universalomega@test101] starting deploy of {'config': True} to all [20:09:53] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [20:10:23] miraheze/mw-config - Universal-Omega the build passed. [20:11:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 4.02, 3.88 [20:12:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.91, 3.28, 3.20 [20:12:39] PROBLEM - test101 MediaWiki Rendering on test101 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2550 bytes in 0.122 second response time [20:13:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/f6785593b7b9 [20:13:09] [02miraheze/puppet] 07paladox 03f678559 - logbot: Use OAUTH for login [20:13:11] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [20:13:12] [02puppet] 07paladox opened pull request 03#2476: logbot: Use OAUTH for login - 13https://github.com/miraheze/puppet/pull/2476 [20:13:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d4354610395a...43596c2a9b7c [20:13:30] [02miraheze/mw-config] 07Universal-Omega 0343596c2 - Update Wikibase.php [20:13:31] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [20:13:50] !log [universalomega@test101] starting deploy of {'config': True} to all [20:13:51] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [20:14:35] miraheze/mw-config - Universal-Omega the build passed. [20:15:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.06, 3.61, 3.77 [20:15:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f6785593b7b9...524fedcae87e [20:15:42] [02miraheze/puppet] 07paladox 03524fedc - Update config.py [20:15:43] [02puppet] 07paladox synchronize pull request 03#2476: logbot: Use OAUTH for login - 13https://github.com/miraheze/puppet/pull/2476 [20:17:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:17:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/524fedcae87e...b61fffebfc0d [20:17:03] [02miraheze/puppet] 07paladox 03b61fffe - Update irclogbot.pp [20:17:05] [02puppet] 07paladox synchronize pull request 03#2476: logbot: Use OAUTH for login - 13https://github.com/miraheze/puppet/pull/2476 [20:17:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.60, 3.96, 3.93 [20:18:55] !log [@test101] starting deploy of {'config': True} to all [20:18:56] !log [@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [20:19:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.01, 3.89, 3.81 [20:19:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.89, 5.25, 4.39 [20:19:22] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.34, 1.89, 1.53 [20:19:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.14, 3.62, 3.33 [20:21:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.36, 3.82, 3.80 [20:21:22] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.20, 1.63, 1.48 [20:21:47] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.66, 3.18, 3.19 [20:23:05] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [20:23:24] [02puppet] 07paladox synchronize pull request 03#2476: logbot: Use OAUTH for login - 13https://github.com/miraheze/puppet/pull/2476 [20:23:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b61fffebfc0d...0ce67102c3b0 [20:23:27] [02miraheze/puppet] 07paladox 030ce6710 - Update adminlog.py [20:23:51] [02puppet] 07paladox closed pull request 03#2476: logbot: Use OAUTH for login - 13https://github.com/miraheze/puppet/pull/2476 [20:23:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/9b615dee3037...ae7114426af8 [20:23:54] [02miraheze/puppet] 07paladox 03ae71144 - logbot: Use OAUTH for login (#2476) [20:23:56] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [20:23:57] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [20:24:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:25:53] !log test [20:25:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:26:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/43596c2a9b7c...d67ce4317d8a [20:26:12] [02miraheze/mw-config] 07Universal-Omega 03d67ce43 - Update MirahezeFunctions.php [20:26:13] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [20:26:19] !log [universalomega@test101] starting deploy of {'config': True} to all [20:26:20] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [20:26:22] Works [20:26:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:26:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:26:39] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.096 second response time [20:26:51] CosmicAlpha: you ^ [20:27:02] well I just caused a segmentation fault.. [20:27:07] RhinosF1: yes [20:27:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.15, 2.85, 3.37 [20:27:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.17, 3.46, 3.85 [20:27:19] good CosmicAlpha [20:27:20] miraheze/mw-config - Universal-Omega the build passed. [20:27:29] And I love segfaults [20:27:39] how do I debug this...? [20:28:11] not easy lol [20:28:18] gdb maybe i dunno tho [20:30:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d67ce4317d8a...4e8f059bd069 [20:30:15] [02miraheze/mw-config] 07Universal-Omega 034e8f059 - Update MirahezeFunctions.php [20:30:17] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [20:31:17] miraheze/mw-config - Universal-Omega the build passed. [20:31:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.02, 3.71, 3.83 [20:31:46] CosmicAlpha: with a lot of screaming and crying [20:31:52] Or ask TimStarling [20:32:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4e8f059bd069...94c0a66373f7 [20:32:56] [02miraheze/mw-config] 07Universal-Omega 0394c0a66 - Update MirahezeFunctions.php [20:32:57] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [20:34:03] miraheze/mw-config - Universal-Omega the build passed. [20:34:28] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.08, 4.03, 3.55 [20:34:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:35:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.86, 3.65 [20:35:54] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/commit/83458efa1724 [20:35:55] [02miraheze/ssl] 07MacFan4000 0383458ef - remove conlang.tk [20:35:57] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [20:36:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.09, 3.71, 3.50 [20:37:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 3.79, 3.64 [20:37:39] [02ssl] 07MacFan4000 opened pull request 03#501: remove 4 domains - 13https://github.com/miraheze/ssl/pull/501 [20:38:21] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.84, 4.01, 3.62 [20:39:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.96, 3.61, 3.60 [20:39:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.33, 3.78, 4.00 [20:39:57] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/83458efa1724...c8b3e533cad1 [20:39:58] [02miraheze/ssl] 07MacFan4000 03c8b3e53 - remove pastport.org [20:40:00] [02ssl] 07MacFan4000 synchronize pull request 03#501: remove 4 domains - 13https://github.com/miraheze/ssl/pull/501 [20:40:18] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.08, 3.42, 3.43 [20:40:31] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/c8b3e533cad1...4848226e8c4c [20:40:32] [02miraheze/ssl] 07MacFan4000 034848226 - remove wiki.emsoc.ie [20:40:34] [02ssl] 07MacFan4000 synchronize pull request 03#501: remove 4 domains - 13https://github.com/miraheze/ssl/pull/501 [20:40:59] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/4848226e8c4c...36909a2f58b2 [20:41:01] [02miraheze/ssl] 07MacFan4000 0336909a2 - remove wiki.xysspon.com [20:41:02] [02ssl] 07MacFan4000 synchronize pull request 03#501: remove 4 domains - 13https://github.com/miraheze/ssl/pull/501 [20:42:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.16, 3.18, 3.33 [20:42:30] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/c2ccaa402c99 [20:42:32] [02miraheze/mw-config] 07paladox 03c2ccaa4 - Set all wikis into read only [20:42:33] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [20:42:35] [02mw-config] 07paladox opened pull request 03#4559: Set all wikis into read only - 13https://github.com/miraheze/mw-config/pull/4559 [20:43:48] miraheze/mw-config - paladox the build passed. [20:45:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.95, 3.75 [20:45:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.72, 4.89, 4.34 [20:45:32] PROBLEM - test101 Disk Space on test101 is WARNING: DISK WARNING - free space: / 1935 MB (10% inode=56%); [20:46:44] Well that does help (I think) got that with gdb, running on WebStart.php https://www.irccloud.com/pastebin/MhoJ7q9a/ [20:47:21] RhinosF1: ^ [20:48:48] CosmicAlpha: it's kinda useful [20:49:01] If you can trace where that's called [20:49:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.93, 3.84, 3.74 [20:49:31] I know the issue is in MirahezeFunctions.php, thats the only new file causing it in my PR. [20:49:35] RhinosF1 [20:49:41] !log [@test101] starting deploy of {'config': True} to all [20:49:42] !log [@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [20:49:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:53] CosmicAlpha: where calls that function then [20:49:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:51:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.73, 4.28, 3.89 [20:51:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:51:32] RECOVERY - test101 Disk Space on test101 is OK: DISK OK - free space: / 1949 MB (11% inode=56%); [20:52:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.27, 4.49, 3.83 [20:55:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.27, 3.72, 3.80 [20:55:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.19, 3.90, 3.76 [21:00:15] [02mw-config] 07paladox closed pull request 03#4559: Set all wikis into read only - 13https://github.com/miraheze/mw-config/pull/4559 [21:00:17] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [21:00:18] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [21:00:20] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b0c524cf2b99...743d81fe9eb5 [21:00:21] [02miraheze/mw-config] 07paladox 03743d81f - Set all wikis into read only (#4559) [21:00:37] !log [paladox@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [21:00:49] !log [paladox@test101] starting deploy of {'pull': 'config', 'config': True} to test101 [21:00:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:00:52] !log [paladox@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [21:00:53] !log [paladox@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 16s [21:01:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/94c0a66373f7...b960d7e6f96d [21:01:08] [02miraheze/mw-config] 07Universal-Omega 03b960d7e - Update MirahezeFunctions.php [21:01:09] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [21:01:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.10, 2.78, 3.35 [21:01:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.81, 3.64, 3.96 [21:01:31] miraheze/mw-config - paladox the build passed. [21:01:46] !log increasing memory on db101/111/121 by 3gb [21:02:22] miraheze/mw-config - Universal-Omega the build passed. [21:03:22] !log reboot db101/111/121 [21:05:14] [02mw-config] 07Universal-Omega edited pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [21:05:39] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.30, 2.42, 3.14 [21:05:56] PROBLEM - db101 Disk Space on db101 is CRITICAL: connect to address 2a10:6740::6:102 port 5666: Connection refusedconnect to host 2a10:6740::6:102 port 5666: Connection refused [21:05:56] PROBLEM - db101 PowerDNS Recursor on db101 is CRITICAL: connect to address 2a10:6740::6:102 port 5666: Connection refusedconnect to host 2a10:6740::6:102 port 5666: Connection refused [21:05:56] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1051 bytes in 0.108 second response time [21:05:57] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1051 bytes in 0.088 second response time [21:06:10] PROBLEM - db111 Puppet on db111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: Connection refusedconnect to host 2a10:6740::6:201 port 5666: Connection refused [21:06:14] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1051 bytes in 0.119 second response time [21:06:19] PROBLEM - db121 Disk Space on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: Connection refusedconnect to host 2a10:6740::6:302 port 5666: Connection refused [21:06:21] PROBLEM - db121 conntrack_table_size on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: Connection refusedconnect to host 2a10:6740::6:302 port 5666: Connection refused [21:06:30] PROBLEM - db121 ferm_active on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: Connection refusedconnect to host 2a10:6740::6:302 port 5666: Connection refused [21:06:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:06:32] PROBLEM - db121 Current Load on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: Connection refusedconnect to host 2a10:6740::6:302 port 5666: Connection refused [21:06:33] PROBLEM - db101 APT on db101 is CRITICAL: connect to address 2a10:6740::6:102 port 5666: Connection refusedconnect to host 2a10:6740::6:102 port 5666: Connection refused [21:06:38] PROBLEM - db111 SSH on db111 is CRITICAL: connect to address 2a10:6740::6:201 and port 22: Connection refused [21:06:42] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1051 bytes in 0.103 second response time [21:06:42] PROBLEM - db101 Current Load on db101 is CRITICAL: connect to address 2a10:6740::6:102 port 5666: Connection refusedconnect to host 2a10:6740::6:102 port 5666: Connection refused [21:06:43] PROBLEM - db121 PowerDNS Recursor on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: Connection refusedconnect to host 2a10:6740::6:302 port 5666: Connection refused [21:06:47] PROBLEM - db101 conntrack_table_size on db101 is CRITICAL: connect to address 2a10:6740::6:102 port 5666: Connection refusedconnect to host 2a10:6740::6:102 port 5666: Connection refused [21:06:47] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1051 bytes in 0.125 second response time [21:06:50] PROBLEM - db111 conntrack_table_size on db111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: Connection refusedconnect to host 2a10:6740::6:201 port 5666: Connection refused [21:06:54] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1051 bytes in 0.091 second response time [21:06:59] PROBLEM - db101 SSH on db101 is CRITICAL: connect to address 2a10:6740::6:102 and port 22: Connection refused [21:07:01] PROBLEM - db121 Puppet on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: Connection refusedconnect to host 2a10:6740::6:302 port 5666: Connection refused [21:07:02] PROBLEM - db111 NTP time on db111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: Connection refusedconnect to host 2a10:6740::6:201 port 5666: Connection refused [21:07:04] PROBLEM - db101 MariaDB on db101 is CRITICAL: Can't connect to MySQL server on 'db101.miraheze.org' (115) [21:07:06] PROBLEM - db121 NTP time on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: Connection refusedconnect to host 2a10:6740::6:302 port 5666: Connection refused [21:07:08] PROBLEM - db111 Disk Space on db111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: Connection refusedconnect to host 2a10:6740::6:201 port 5666: Connection refused [21:07:13] PROBLEM - db111 APT on db111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: Connection refusedconnect to host 2a10:6740::6:201 port 5666: Connection refused [21:07:16] PROBLEM - db111 Current Load on db111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: Connection refusedconnect to host 2a10:6740::6:201 port 5666: Connection refused [21:07:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 0.65, 1.99, 3.16 [21:07:20] PROBLEM - db101 Puppet on db101 is CRITICAL: connect to address 2a10:6740::6:102 port 5666: Connection refusedconnect to host 2a10:6740::6:102 port 5666: Connection refused [21:07:20] PROBLEM - db121 MariaDB on db121 is CRITICAL: Can't connect to MySQL server on 'db121.miraheze.org' (115) [21:07:26] PROBLEM - db111 MariaDB on db111 is CRITICAL: Can't connect to MySQL server on 'db111.miraheze.org' (115) [21:07:28] PROBLEM - db111 ferm_active on db111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: Connection refusedconnect to host 2a10:6740::6:201 port 5666: Connection refused [21:07:42] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1053 bytes in 6.111 second response time [21:07:45] PROBLEM - db121 SSH on db121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:07:54] PROBLEM - db101 ferm_active on db101 is CRITICAL: connect to address 2a10:6740::6:102 port 5666: No route to hostconnect to host 2a10:6740::6:102 port 5666: No route to host [21:07:54] PROBLEM - db101 NTP time on db101 is CRITICAL: connect to address 2a10:6740::6:102 port 5666: No route to hostconnect to host 2a10:6740::6:102 port 5666: No route to host [21:08:04] PROBLEM - db121 APT on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: No route to hostconnect to host 2a10:6740::6:302 port 5666: No route to host [21:08:30] RECOVERY - db121 ferm_active on db121 is OK: OK ferm input default policy is set [21:08:32] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 0.71, 0.16, 0.05 [21:08:33] RECOVERY - db101 APT on db101 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [21:08:38] RECOVERY - db111 SSH on db111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [21:08:42] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 1.22, 0.33, 0.11 [21:08:43] RECOVERY - db121 PowerDNS Recursor on db121 is OK: DNS OK: 0.211 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [21:08:47] RECOVERY - db101 conntrack_table_size on db101 is OK: OK: nf_conntrack is 0 % full [21:08:48] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 23341 bytes in 1.759 second response time [21:08:50] RECOVERY - db111 conntrack_table_size on db111 is OK: OK: nf_conntrack is 0 % full [21:08:56] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23332 bytes in 3.570 second response time [21:08:59] RECOVERY - db101 SSH on db101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [21:09:03] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 20 minutes ago with 0 failures [21:09:04] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset 0.02383184433 secs [21:09:04] RECOVERY - db101 MariaDB on db101 is OK: Uptime: 52 Threads: 53 Questions: 6634 Slow queries: 335 Opens: 1112 Open tables: 1106 Queries per second avg: 127.576 [21:09:06] [02miraheze/mw-config] 07paladox pushed 031 commit to 03revert-4559-paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/8e6925a32988 [21:09:08] RECOVERY - db111 Disk Space on db111 is OK: DISK OK - free space: / 329148 MB (52% inode=98%); [21:09:08] [02miraheze/mw-config] 07paladox 038e6925a - Revert "Set all wikis into read only (#4559)" [21:09:09] RECOVERY - db121 NTP time on db121 is OK: NTP OK: Offset -0.000356644392 secs [21:09:09] [02mw-config] 07paladox created branch 03revert-4559-paladox-patch-1 - 13https://github.com/miraheze/mw-config [21:09:11] [02mw-config] 07paladox opened pull request 03#4560: Revert "Set all wikis into read only" - 13https://github.com/miraheze/mw-config/pull/4560 [21:09:15] RECOVERY - db111 APT on db111 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [21:09:15] [02mw-config] 07paladox closed pull request 03#4560: Revert "Set all wikis into read only" - 13https://github.com/miraheze/mw-config/pull/4560 [21:09:16] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 1.76, 0.54, 0.19 [21:09:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/743d81fe9eb5...746299ce1fc9 [21:09:18] [02miraheze/mw-config] 07paladox 03746299c - Revert "Set all wikis into read only (#4559)" (#4560) [21:09:19] [02mw-config] 07paladox deleted branch 03revert-4559-paladox-patch-1 - 13https://github.com/miraheze/mw-config [21:09:20] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 18 minutes ago with 0 failures [21:09:20] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 60 Threads: 130 Questions: 3782 Slow queries: 269 Opens: 1101 Open tables: 1095 Queries per second avg: 63.033 [21:09:21] [02miraheze/mw-config] 07paladox deleted branch 03revert-4559-paladox-patch-1 [21:09:26] !log [paladox@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [21:09:26] RECOVERY - db111 MariaDB on db111 is OK: Uptime: 73 Threads: 6 Questions: 4470 Slow queries: 490 Opens: 697 Open tables: 691 Queries per second avg: 61.232 [21:09:28] RECOVERY - db111 ferm_active on db111 is OK: OK ferm input default policy is set [21:09:29] !log [paladox@test101] starting deploy of {'pull': 'config', 'config': True} to test101 [21:09:32] !log [paladox@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [21:09:32] PROBLEM - test101 Disk Space on test101 is WARNING: DISK WARNING - free space: / 1915 MB (10% inode=56%); [21:09:45] RECOVERY - db121 SSH on db121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [21:09:48] RECOVERY - db101 ferm_active on db101 is OK: OK ferm input default policy is set [21:09:54] RECOVERY - db101 PowerDNS Recursor on db101 is OK: DNS OK: 0.128 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [21:09:56] RECOVERY - db101 Disk Space on db101 is OK: DISK OK - free space: / 319849 MB (50% inode=98%); [21:09:58] paladox: ignore beta [21:09:59] RECOVERY - db101 NTP time on db101 is OK: NTP OK: Offset -0.0003831982613 secs [21:10:02] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23332 bytes in 2.599 second response time [21:10:05] RECOVERY - db121 APT on db121 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [21:10:07] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23332 bytes in 3.951 second response time [21:10:12] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:10:12] RECOVERY - db111 Puppet on db111 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [21:10:13] miraheze/mw-config - paladox the build passed. [21:10:18] miraheze/mw-config - paladox the build passed. [21:10:18] !log [paladox@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 52s [21:10:18] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23332 bytes in 0.298 second response time [21:10:19] RECOVERY - db121 Disk Space on db121 is OK: DISK OK - free space: / 295424 MB (47% inode=98%); [21:10:21] RECOVERY - db121 conntrack_table_size on db121 is OK: OK: nf_conntrack is 0 % full [21:10:36] CosmicAlpha: I assume you can handle beta [21:10:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:10:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:10:50] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23332 bytes in 9.816 second response time [21:11:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:11:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:11:39] .tell paladox ignore beta [21:11:39] RhinosF1: I'll pass that on when paladox is around. [21:11:40] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23332 bytes in 0.458 second response time [21:12:07] paladox: ignore beta [21:12:07] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.017 second response time [21:12:18] CosmicAlpha is breaking that and can deploy himself [21:12:19] ok [21:12:20] [tell] paladox: 2022-04-07 - 22:11:39BST tell paladox ignore beta [21:12:42] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03revert-4553-paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/089ae63be371 [21:12:42] things have recovered and things look good my end [21:12:43] [02miraheze/mw-config] 07RhinosF1 03089ae63 - Revert "Add site notice for emergency database maintenance (#4553)" [21:12:45] [02mw-config] 07RhinosF1 created branch 03revert-4553-paladox-patch-1 - 13https://github.com/miraheze/mw-config [21:12:46] [02mw-config] 07RhinosF1 opened pull request 03#4561: Revert "Add site notice for emergency database maintenance" - 13https://github.com/miraheze/mw-config/pull/4561 [21:12:55] paladox: ye, can you do ^ [21:13:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03revert-4553-paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/089ae63be371...8d20d5d2e886 [21:13:17] [02miraheze/mw-config] 07paladox 038d20d5d - Update Sitenotice.php [21:13:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.76, 3.58, 3.36 [21:13:19] [02mw-config] 07paladox synchronize pull request 03#4561: Revert "Add site notice for emergency database maintenance" - 13https://github.com/miraheze/mw-config/pull/4561 [21:13:24] [02mw-config] 07paladox closed pull request 03#4561: Revert "Add site notice for emergency database maintenance" - 13https://github.com/miraheze/mw-config/pull/4561 [21:13:24] !log [paladox@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [21:13:25] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/746299ce1fc9...e361b7266cc1 [21:13:27] [02miraheze/mw-config] 07RhinosF1 03e361b72 - Revert "Add site notice for emergency database maintenance" (#4561) [21:13:28] [02mw-config] 07paladox deleted branch 03revert-4553-paladox-patch-1 - 13https://github.com/miraheze/mw-config [21:13:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:13:30] [02miraheze/mw-config] 07paladox deleted branch 03revert-4553-paladox-patch-1 [21:13:33] !log [paladox@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [21:13:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:13:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.92, 3.24, 3.05 [21:13:49] miraheze/mw-config - RhinosF1 the build passed. [21:14:12] paladox: perfect! [21:14:20] miraheze/mw-config - paladox the build passed. [21:14:25] I think you can call that a success [21:14:30] miraheze/mw-config - paladox the build passed. [21:14:41] RhinosF1: What do I need to do? [21:15:04] CosmicAlpha: deploy config to beta once it's not messed up to bring it out of read only [21:15:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.59, 2.85, 2.93 [21:16:32] RhinosF1: MirahezeFunctions may not be causeing segfault afterall... [21:16:45] CosmicAlpha: that's cool [21:16:48] What could be [21:16:57] `/home/paladox/php-xhprof-extension/timer.h:222` looks like it is [21:17:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:17:06] oh [21:17:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.57, 3.80, 3.51 [21:17:20] that's interesting lol [21:17:25] CosmicAlpha: fun! [21:17:27] why is it trying to load /home/paladox? [21:17:38] very good question lol [21:17:53] That'll be perf team if you need upstream [21:18:08] paladox: probably weird issue from when it was built [21:18:23] maybe [21:18:35] !log [@test101] starting deploy of {'config': True} to all [21:18:36] !log [@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [21:18:38] https://www.irccloud.com/pastebin/vKAq5Ria/ [21:18:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:18:42] paladox: ^ [21:18:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:18:55] hmm [21:18:59] we can rebuild it [21:19:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.73, 4.44, 3.81 [21:19:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.52, 3.33, 3.08 [21:19:39] maybe it doesn't support php 7.4? [21:20:04] we have a php7.4 version of that I thought? [21:20:13] maybe that one is using php7.3 version? [21:20:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.71, 3.64, 3.13 [21:21:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.09, 3.26, 3.08 [21:21:31] !log sudo apt-get install gdb on test101 - will remove when debugging is done [21:21:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:22:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:22:21] https://github.com/miraheze/puppet/commit/9619c210ac09fc9da6ac11bc679db12223fad7fb oh nvm [21:22:21] [url] Remove tideways_xhprof for PHP 7.3 (#2139) · miraheze/puppet@9619c21 · GitHub | github.com [21:24:10] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.60, 3.91, 3.34 [21:24:13] paladox: should I remove gdb from test101 when done with it or leave it? It can be extremally useful it seems, but we also don't want packages we don't need. I do plan on removing but was just wondering what you thought. [21:25:50] but yeah it is weird it is loading from your home directory here... [21:26:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.36, 3.65, 3.32 [21:26:22] [02ManageWiki] 07Universal-Omega closed pull request 03#347: Use texts type for `$wgNamespaceAliases` - 13https://github.com/miraheze/ManageWiki/pull/347 [21:26:23] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/ManageWiki/compare/366a5d49fc65...9780e2117136 [21:26:25] [02miraheze/ManageWiki] 07Universal-Omega 039780e21 - Use texts type for `$wgNamespaceAliases` (#347) [21:26:26] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ManageWiki [21:26:28] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:27:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.09, 3.97, 3.89 [21:27:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:28:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03tideways [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/90ace6bb5b96 [21:28:19] [02miraheze/puppet] 07paladox 0390ace6b - mediawiki: Update tideways xhprof so file [21:28:21] [02puppet] 07paladox created branch 03tideways - 13https://github.com/miraheze/puppet [21:28:29] [02puppet] 07paladox opened pull request 03#2477: mediawiki: Update tideways xhprof so file - 13https://github.com/miraheze/puppet/pull/2477 [21:28:37] CosmicAlpha: i've built it ^ [21:28:42] and it's installed on test101 [21:28:47] does it work for you? [21:29:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.96, 4.44, 4.08 [21:30:07] CosmicAlpha: also you can leave gdb if you want it's a test server after all [21:30:19] paladox: no, backtrace still shows it uses your home directory. And sounds good, will do then. [21:30:30] !log leaving gdb on test101 [21:30:32] oh ok [21:30:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:30:53] miraheze/ManageWiki - Universal-Omega the build passed. [21:31:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.36, 3.41, 3.19 [21:32:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:33:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.98, 3.21, 3.13 [21:33:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.82, 3.90 [21:34:35] hmm CosmicAlpha it seems to be able to find the lines example /home/paladox/php-xhprof-extension/tideways_xhprof.c:61 [21:34:44] so i don't think it's because it's doing /home/paladox? [21:35:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.03, 4.37, 4.10 [21:36:20] paladox: oh, weird... can we turn the php extension off for now to see if that fixes the segfault then? It is still weird to me that it even uses /home/paladox at all though. [21:36:31] sure [21:36:37] you can delete the extension [21:36:42] in /etc/php [21:37:17] done for you CosmicAlpha [21:37:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.49, 3.63, 3.86 [21:37:19] does it work? [21:37:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.03, 2.96, 3.18 [21:38:34] paladox: fixed segfault, now another error, related to the MirahezeFunctions this time, which is better. [21:38:44] ok [21:38:50] probably you want to disable puppet [21:38:56] to prevent it reinstalling it [21:40:02] done paladox [21:40:07] ok [21:40:07] thanks! [21:40:41] PROBLEM - test101 Puppet on test101 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 21 minutes ago with 1 failures [21:40:57] Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 262144 bytes) in /srv/mediawiki/config/initialise/MirahezeFunctions.php on line 90 [21:41:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 3.97, 3.93 [21:41:20] well that is not right, seeing as how this is supposed to improve performance here... [21:41:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.95, 3.45, 3.22 [21:42:33] [02puppet] 07paladox closed pull request 03#2477: mediawiki: Update tideways xhprof so file - 13https://github.com/miraheze/puppet/pull/2477 [21:42:35] [02miraheze/puppet] 07paladox deleted branch 03tideways [21:42:36] [02puppet] 07paladox deleted branch 03tideways - 13https://github.com/miraheze/puppet [21:44:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:44:55] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [21:44:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b960d7e6f96d...935021a186c3 [21:44:58] [02miraheze/mw-config] 07Universal-Omega 03935021a - Update MirahezeFunctions.php [21:45:25] !log [universalomega@test101] starting deploy of {'config': True} to all [21:45:26] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [21:45:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:45:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:45:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.97, 3.35, 3.25 [21:45:56] miraheze/mw-config - Universal-Omega the build passed. [21:49:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:51:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.99, 3.73, 3.87 [21:53:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.47, 3.89, 3.90 [21:53:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.07, 3.69, 3.45 [21:55:23] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.86, 3.65, 3.46 [21:56:19] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/36909a2f58b2...c66fed4ca8b6 [21:56:20] [02miraheze/ssl] 07MacFan4000 03c66fed4 - Update certs.yaml [21:56:22] [02ssl] 07MacFan4000 synchronize pull request 03#501: remove 4 domains - 13https://github.com/miraheze/ssl/pull/501 [21:57:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.96, 4.52, 3.81 [21:57:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:58:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.40, 3.57, 3.32 [21:59:35] [02ssl] 07MacFan4000 closed pull request 03#501: remove 4 domains - 13https://github.com/miraheze/ssl/pull/501 [21:59:36] [02miraheze/ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [21:59:38] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-4/±1] 13https://github.com/miraheze/ssl/compare/be6a6dd882a4...8edf88b9b11c [21:59:39] [02miraheze/ssl] 07MacFan4000 038edf88b - remove 4 domains (#501) [21:59:41] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [22:00:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.57, 3.19, 3.21 [22:01:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.93, 3.61, 3.59 [22:01:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.45, 3.72, 3.95 [22:02:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:03:36] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/pastport.org.key],File[/etc/ssl/private/wiki.xysspon.com.key],File[/etc/ssl/private/conlang.tk.key],File[/etc/ssl/private/wiki.emsoc.ie.key] [22:03:51] That should resolve itself next puppet run [22:04:04] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.42, 1.77, 1.42 [22:05:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.34, 2.87, 3.29 [22:05:53] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/pastport.org.key],File[/etc/ssl/private/wiki.xysspon.com.key],File[/etc/ssl/private/conlang.tk.key],File[/etc/ssl/private/wiki.emsoc.ie.key] [22:06:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.69, 1.75, 1.46 [22:07:12] PROBLEM - mw111 Puppet on mw111 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/pastport.org.key],File[/etc/ssl/private/wiki.xysspon.com.key],File[/etc/ssl/private/conlang.tk.key],File[/etc/ssl/private/wiki.emsoc.ie.key] [22:08:00] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.31, 1.62, 1.44 [22:09:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.86, 3.89 [22:11:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.83, 3.45, 3.73 [22:14:25] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [22:14:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/935021a186c3...3cfaf73fc63c [22:14:28] [02miraheze/mw-config] 07Universal-Omega 033cfaf73 - Update MirahezeFunctions.php [22:14:39] !log [universalomega@test101] starting deploy of {'config': True} to all [22:14:40] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [22:14:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:14:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:15:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 3.73, 3.75 [22:15:30] miraheze/mw-config - Universal-Omega the build passed. [22:15:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.42, 3.34, 3.16 [22:15:39] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8edf88b9b11c...ec5c79ce13fc [22:15:40] [02miraheze/ssl] 07MacFan4000 03ec5c79c - renew wiki.autocountsoft.com [22:17:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.40, 3.62 [22:17:27] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.43, 3.13, 3.11 [22:21:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:23:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.39, 3.64, 3.61 [22:25:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.80, 3.30, 3.48 [22:27:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.56, 2.78, 3.28 [22:27:31] CosmicAlpha: i'm sure you're aware but test101 looks to be a bit low on space [22:27:38] (icinga alert) [22:27:42] I know, will look soon [22:27:55] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.56, 2.00, 1.27 [22:30:59] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.60, 1.31, 1.13 [22:31:15] RECOVERY - wiki.autocountsoft.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.autocountsoft.com' will expire on Wed 06 Jul 2022 21:13:42 GMT +0000. [22:31:30] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [22:35:07] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:36:40] RECOVERY - mw111 Puppet on mw111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:42:32] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.87, 9.80, 9.16 [22:44:32] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.51, 9.20, 9.02 [22:46:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:54:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 3.26, 2.91 [22:55:52] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.14, 3.53, 3.27 [22:56:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.81, 2.77, 2.78 [22:57:50] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.59, 3.83, 3.40 [22:59:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.58, 3.35 [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:03] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:25] !log [@test101] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [23:00:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:00:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.39, 4.70, 3.77 [23:04:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.84, 3.27 [23:05:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.47, 3.62, 3.22 [23:05:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:07:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.71, 4.24, 3.48 [23:10:16] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [23:10:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3cfaf73fc63c...c3ddcecfac67 [23:10:19] [02miraheze/mw-config] 07Universal-Omega 03c3ddcec - improve performance [23:10:44] !log [universalomega@test101] starting deploy of {'config': True} to all [23:10:45] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [23:10:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:10:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:11:37] miraheze/mw-config - Universal-Omega the build has errored. [23:11:46] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [23:11:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c3ddcecfac67...0c1c1e2ae758 [23:11:49] [02miraheze/mw-config] 07Universal-Omega 030c1c1e2 - Update MirahezeFunctions.php [23:12:04] !log [universalomega@test101] starting deploy of {'config': True} to all [23:12:05] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [23:12:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:12:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:12:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:12:55] miraheze/mw-config - Universal-Omega the build passed. [23:14:35] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 873s [23:14:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:18:40] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [23:18:42] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0c1c1e2ae758...3f27f76c1b96 [23:18:43] [02miraheze/mw-config] 07Universal-Omega 033f27f76 - Update MirahezeFunctions.php [23:19:47] miraheze/mw-config - Universal-Omega the build passed. [23:20:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.60, 3.87, 3.89 [23:20:33] !log [universalomega@test101] starting deploy of {'config': True} to all [23:20:34] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [23:20:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:20:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:20:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.61, 3.40, 3.67 [23:22:10] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [23:22:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3f27f76c1b96...38484ac50565 [23:22:13] [02miraheze/mw-config] 07Universal-Omega 0338484ac - Update MirahezeFunctions.php [23:22:24] !log [universalomega@test101] starting deploy of {'config': True} to all [23:22:25] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [23:22:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:22:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:23:32] miraheze/mw-config - Universal-Omega the build passed. [23:26:30] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [23:26:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/38484ac50565...fe0f7beb8bc9 [23:26:33] [02miraheze/mw-config] 07Universal-Omega 03fe0f7be - Update MirahezeFunctions.php [23:26:42] !log [universalomega@test101] starting deploy of {'config': True} to all [23:26:43] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [23:26:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:26:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.90, 3.73, 3.70 [23:26:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:29] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [23:27:29] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:27:43] miraheze/mw-config - Universal-Omega the build passed. [23:27:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.35, 3.79, 3.71 [23:28:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.15, 3.44, 3.59 [23:29:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.23, 3.39, 3.57 [23:30:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:33:48] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.97, 4.12, 3.81 [23:34:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.92, 4.08, 3.77 [23:35:13] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [23:35:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/fe0f7beb8bc9...18188025d9c1 [23:35:16] [02miraheze/mw-config] 07Universal-Omega 031818802 - Update MirahezeFunctions.php [23:35:34] !log [universalomega@test101] starting deploy of {'config': True} to all [23:35:35] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [23:35:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:35:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:36:20] miraheze/mw-config - Universal-Omega the build passed. [23:36:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.59, 3.50, 3.60 [23:39:48] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.70, 3.79, 3.87 [23:40:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:41:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.55, 3.48, 3.92 [23:42:50] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.89, 2.85, 3.32 [23:45:01] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [23:45:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/18188025d9c1...f63f158356ef [23:45:04] [02miraheze/mw-config] 07Universal-Omega 03f63f158 - Update MirahezeFunctions.php [23:45:16] !log [universalomega@test101] starting deploy of {'config': True} to all [23:45:17] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [23:45:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:45:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:46:11] miraheze/mw-config - Universal-Omega the build passed. [23:47:03] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [23:47:05] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f63f158356ef...18119460b2e4 [23:47:06] [02miraheze/mw-config] 07Universal-Omega 031811946 - Update MirahezeFunctions.php [23:47:16] !log [universalomega@test101] starting deploy of {'config': True} to all [23:47:17] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [23:47:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:47:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:48:13] miraheze/mw-config - Universal-Omega the build passed. [23:48:27] PROBLEM - test101 MediaWiki Rendering on test101 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2550 bytes in 0.342 second response time [23:53:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.76, 2.95, 3.40 [23:54:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:55:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.02, 2.74, 3.24 [23:59:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki