[00:00:54] !log [rhinos@test101] finished deploy of {'config': True, 'world': True, 'l10n': True, 'extensionlist': True} to skip - SUCCESS in 1065s [00:01:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:01:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.15, 9.38, 7.72 [00:01:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:04:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.19, 8.44, 7.68 [00:04:06] miraheze/ManageWiki - Universal-Omega the build passed. [00:04:50] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/d5d9124e6e94 [00:04:51] [02miraheze/mw-config] 07RhinosF1 03d5d9124 - Sitenotice: add notice about ES [00:04:53] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-3 - 13https://github.com/miraheze/mw-config [00:05:04] [02mw-config] 07RhinosF1 opened pull request 03#4437: Sitenotice: add notice about ES - 13https://github.com/miraheze/mw-config/pull/4437 [00:05:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.38, 9.89, 9.36 [00:05:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.12, 9.33, 9.53 [00:05:15] [02mw-config] 07RhinosF1 assigned pull request 03#4436: Betawiki: configure guarded ES cluster - 13https://github.com/miraheze/mw-config/pull/4436 [00:05:18] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.53, 1.73, 1.44 [00:05:19] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 410 failures. Last run 4 minutes ago with 410 failures. Failed resources (up to 3 shown): File[www.bluepageswiki.org_private],File[wiki.mobilityengineer.com],File[wiki.mobilityengineer.com_private],File[www.mcpk.wiki] [00:05:46] looking [00:06:08] miraheze/mw-config - RhinosF1 the build passed. [00:06:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:08:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.35, 9.12, 9.42 [00:08:08] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [00:08:13] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.47, 1.54, 1.41 [00:10:30] !log [rhinos@mwtask111] DEPLOY ABORTED: Canary check failed for mw101 [00:10:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:10:36] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.40, 9.06, 8.25 [00:10:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.78, 8.61, 8.29 [00:10:45] [02mw-config] 07Universal-Omega reviewed pull request 03#4436 commit - 13https://github.com/miraheze/mw-config/pull/4436#discussion_r791251187 [00:12:13] [02mw-config] 07Universal-Omega commented on pull request 03#4437: Sitenotice: add notice about ES - 13https://github.com/miraheze/mw-config/pull/4437#issuecomment-1020677798 [00:12:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.86, 9.54, 8.90 [00:13:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.35, 10.44, 8.89 [00:13:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.03, 8.43, 8.27 [00:14:01] !log [rhinos@mwtask111] starting deploy of {'config': True, 'l10n': True, 'extensionlist': True, 'force': True} to scsvg [00:14:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:14:54] CosmicAlpha: ? [00:15:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.50, 10.31, 9.27 [00:16:04] RhinosF1: yes? [00:16:14] CosmicAlpha: how formatting [00:16:34] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.48, 9.66, 8.58 [00:16:45] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.15, 10.00, 9.56 [00:16:52] RhinosF1: To much tabbing, poor structure. See how it is in GlobalExtensions.php, that's how it should look here. [00:18:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.49, 9.89, 9.30 [00:18:55] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ca1fa92fdc07...60850c03ed38 [00:18:56] [02miraheze/mw-config] 07RhinosF1 0360850c0 - Update LocalWiki.php [00:18:58] CosmicAlpha: ^ [00:18:58] [02mw-config] 07RhinosF1 synchronize pull request 03#4436: Betawiki: configure guarded ES cluster - 13https://github.com/miraheze/mw-config/pull/4436 [00:19:33] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.61, 9.14, 8.58 [00:20:04] miraheze/mw-config - RhinosF1 the build passed. [00:20:50] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.34, 1.75, 1.49 [00:22:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.20, 8.71, 8.90 [00:22:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.69, 9.29, 9.71 [00:22:23] [02mw-config] 07RhinosF1 commented on pull request 03#4437: Sitenotice: add notice about ES - 13https://github.com/miraheze/mw-config/pull/4437#issuecomment-1020683077 [00:22:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.18, 7.89, 8.21 [00:22:50] CosmicAlpha: the sitenotice will only show that way while the wiki is being migrated [00:23:04] i will likely do it by like letter or something [00:23:45] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.43, 1.66, 1.50 [00:23:57] !log [rhinos@mwtask111] finished deploy of {'config': True, 'l10n': True, 'extensionlist': True, 'force': True} to scsvg - SUCCESS in 596s [00:24:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:24:12] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.56, 7.49, 8.43 [00:25:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.31, 9.80, 9.77 [00:27:48] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.57, 7.23, 8.20 [00:28:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.33, 9.05, 9.36 [00:28:36] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.38, 10.37, 9.95 [00:30:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.39, 1.78, 1.64 [00:31:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.35, 8.84, 9.20 [00:31:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.95, 9.69, 9.07 [00:33:30] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.63, 1.35, 1.51 [00:33:47] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.96, 9.26, 9.26 [00:34:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.64, 9.84, 9.91 [00:36:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:36:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.97, 9.49, 9.42 [00:39:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.40, 1.77, 1.70 [00:42:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.61, 7.61, 8.28 [00:42:56] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.81, 1.42, 1.58 [00:43:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:45:53] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.38, 10.13, 9.70 [00:46:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.66, 8.74, 8.47 [00:48:00] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.37, 9.30, 9.17 [00:48:44] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.92, 1.88, 1.49 [00:51:43] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.39, 2.14, 1.67 [00:52:21] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.54, 7.81, 8.20 [00:54:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.16, 3.80, 3.13 [00:54:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.57, 4.08, 3.51 [00:56:59] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.60, 3.14, 2.98 [00:57:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.70, 3.36, 3.32 [01:00:32] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.98, 1.99, 1.82 [01:00:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.36, 8.99, 9.72 [01:02:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.76, 9.00, 9.52 [01:03:29] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.00, 2.02, 1.87 [01:05:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.53, 10.46, 9.98 [01:06:23] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.92, 1.98, 1.89 [01:08:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.21, 9.96, 9.88 [01:09:16] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.23, 9.63, 9.71 [01:09:17] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.11, 2.05, 1.94 [01:15:19] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.94, 1.97, 1.95 [01:17:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.08, 9.78, 8.63 [01:18:21] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.12, 2.08, 2.00 [01:19:51] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.56, 8.69, 8.38 [01:20:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.64, 8.70, 8.46 [01:23:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.65, 9.90, 9.72 [01:23:17] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.03, 8.28, 8.33 [01:26:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.81, 9.90, 9.76 [01:27:49] !log [rhinos@mwtask111] END fixRC on all wikis [01:28:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:29:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.79, 11.04, 10.25 [01:30:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.90, 8.67, 8.64 [01:30:37] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.53, 1.91, 1.99 [01:32:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.64, 9.87, 9.94 [01:33:36] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.49, 2.14, 2.06 [01:34:46] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.16, 8.07, 8.48 [01:35:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.76, 10.61, 10.24 [01:36:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.46, 8.64, 8.56 [01:41:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.14, 8.96, 8.76 [01:42:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.69, 10.15, 9.29 [01:45:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.33, 9.74, 9.28 [01:46:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.74, 8.48, 7.92 [01:48:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.93, 3.83, 3.35 [01:49:49] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.44, 7.97, 7.82 [01:50:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.13, 7.56, 8.21 [01:51:40] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.41, 3.16, 3.18 [02:02:05] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.88, 8.64, 8.26 [02:03:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.62, 9.68, 8.94 [02:03:42] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.50, 9.78, 9.14 [02:06:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.02, 8.89, 8.71 [02:08:03] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.96, 8.34, 8.30 [02:09:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.76, 7.87, 8.37 [02:09:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.41, 9.58, 9.37 [02:09:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.64, 8.81, 8.70 [02:12:40] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.02, 10.46, 9.34 [02:18:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.50, 9.41, 9.28 [02:18:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.11, 8.47, 8.55 [02:21:18] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.31, 0.90, 1.89 [02:21:35] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.27, 8.21, 8.45 [02:24:07] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.06, 0.53, 1.58 [02:25:16] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.47, 8.94, 8.43 [02:28:12] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.75, 8.27, 8.28 [02:33:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.68, 9.40, 9.96 [02:35:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.17, 8.83, 9.99 [02:36:11] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.10, 10.32, 10.22 [02:37:09] [02ManageWiki] 07lens0021 commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1020751125 [02:37:52] [02ManageWiki] 07lens0021 edited a comment on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1020751125 [02:38:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.13, 4.20, 3.33 [02:38:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.48, 9.24, 9.03 [02:41:03] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.24, 9.31, 9.75 [02:43:51] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.71, 3.36 [02:43:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.04, 9.40, 9.73 [02:44:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.77, 8.83, 8.53 [02:44:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.74, 9.77, 9.35 [02:44:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.88, 9.21, 9.81 [02:44:51] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.77, 2.64, 1.90 [02:46:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.57, 3.34 [02:47:08] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.78, 8.40, 8.40 [02:47:39] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.12, 1.79, 1.68 [02:49:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.66, 9.94, 9.79 [02:49:32] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.92, 2.87, 3.11 [02:50:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.41, 2.22, 1.84 [02:52:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.46, 9.40, 9.47 [02:53:19] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.64, 1.85, 1.75 [02:55:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.75, 9.40, 9.45 [02:55:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.17, 7.74, 8.42 [02:56:09] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.70, 1.36, 1.58 [02:58:08] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.52, 7.64, 8.31 [02:58:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.54, 10.18, 9.72 [02:58:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:01:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.70, 9.52, 9.55 [03:04:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.63, 9.68, 9.55 [03:04:57] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.59, 9.77, 9.02 [03:05:18] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.86, 2.37, 1.25 [03:07:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.60, 9.80, 9.67 [03:09:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:10:17] [02ManageWiki] 07Universal-Omega commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1020765998 [03:10:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.40, 9.18, 8.99 [03:13:04] [02mw-config] 07Universal-Omega commented on pull request 03#4437: Sitenotice: add notice about ES - 13https://github.com/miraheze/mw-config/pull/4437#issuecomment-1020767387 [03:15:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.08, 9.83, 9.66 [03:17:19] [02ManageWiki] 07lens0021 synchronize pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329 [03:18:05] [02ManageWiki] 07lens0021 edited pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329 [03:18:12] [02ManageWiki] 07lens0021 edited pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329 [03:18:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.33, 9.54, 9.62 [03:19:08] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/c400776a97a6...30c2ce8b92af [03:19:09] [02miraheze/RottenLinks] 07Universal-Omega 0330c2ce8 - blacklist -> exclude [03:21:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 9.79, 10.23, 9.89 [03:22:20] miraheze/ManageWiki - lens0021 the build passed. [03:22:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.30, 8.25, 8.50 [03:23:59] [02ManageWiki] 07Universal-Omega reviewed pull request 03#329 commit - 13https://github.com/miraheze/ManageWiki/pull/329#discussion_r791326880 [03:24:43] miraheze/RottenLinks - Universal-Omega the build passed. [03:30:48] [02puppet] 07Universal-Omega opened pull request 03#2392: Add Elastica composer - 13https://github.com/miraheze/puppet/pull/2392 [03:32:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.75, 9.26, 9.70 [03:38:02] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [03:40:33] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [03:41:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.80, 3.26 [03:41:34] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [03:43:46] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [03:44:06] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [03:44:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.59, 3.27 [03:46:13] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [03:47:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.19, 3.00, 3.10 [03:47:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.88, 9.50, 9.99 [03:48:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.84, 8.87, 8.50 [03:48:58] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [03:50:21] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [03:50:23] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.87, 11.71, 10.77 [03:51:00] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [03:51:24] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [03:52:27] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [03:52:45] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [03:52:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.03, 9.28, 9.22 [03:53:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.63, 9.68, 8.96 [03:54:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:55:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.86, 8.45, 8.93 [03:56:05] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [03:56:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.72, 9.19, 8.89 [03:57:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:58:02] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [04:01:11] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [04:01:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.06, 9.17, 9.97 [04:01:58] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [04:07:20] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [04:11:45] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.10, 1.39, 1.90 [04:12:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.94, 7.90, 8.46 [04:13:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.61, 8.23, 8.50 [04:15:58] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.49, 9.90, 9.41 [04:16:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.60, 3.26 [04:17:28] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.11, 1.17, 1.63 [04:19:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.81, 9.78, 9.01 [04:21:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.30, 9.63, 9.08 [04:24:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.12, 2.99, 3.16 [04:27:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:33:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.99, 8.48, 9.54 [04:33:08] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.71, 8.06, 8.45 [04:35:24] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [04:39:13] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [04:41:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.65, 8.64, 8.96 [04:44:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.63, 7.68, 8.54 [04:44:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:44:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.84, 8.03, 7.68 [04:45:44] [02puppet] 07Universal-Omega opened pull request 03#2393: base::packages: use ensure_packages - 13https://github.com/miraheze/puppet/pull/2393 [04:46:44] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.22, 10.41, 9.12 [04:47:02] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.52, 7.69, 8.40 [04:47:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.47, 9.24, 8.14 [04:47:32] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.75, 7.94, 7.70 [04:47:45] [02puppet] 07Universal-Omega synchronize pull request 03#2393: base::packages: use ensure_packages - 13https://github.com/miraheze/puppet/pull/2393 [04:48:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.96, 8.25, 7.48 [04:50:04] [02puppet] 07Universal-Omega synchronize pull request 03#2393: base::packages: use ensure_packages - 13https://github.com/miraheze/puppet/pull/2393 [04:50:43] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.56, 8.83, 7.68 [04:51:41] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.63, 9.70, 8.16 [04:52:28] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw111 [04:52:36] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [04:53:59] [02ManageWiki] 07lens0021 synchronize pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329 [04:54:17] [02ManageWiki] 07lens0021 reviewed pull request 03#329 commit - 13https://github.com/miraheze/ManageWiki/pull/329#discussion_r791355382 [04:55:05] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 11 backends are healthy [04:55:23] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 11 backends are healthy [04:56:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.71, 9.34, 8.29 [04:56:34] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.53, 8.24, 7.88 [04:57:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.27, 9.96, 8.81 [04:57:30] [02ManageWiki] 07Universal-Omega commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1020812191 [04:57:56] [02ManageWiki] 07Universal-Omega reviewed pull request 03#329 commit - 13https://github.com/miraheze/ManageWiki/pull/329#discussion_r791356689 [04:59:06] [02ManageWiki] 07Universal-Omega commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1020812804 [04:59:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.58, 7.99, 7.85 [05:00:09] [02ManageWiki] 07Universal-Omega commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1020813175 [05:00:19] miraheze/ManageWiki - lens0021 the build passed. [05:01:30] [02ManageWiki] 07Universal-Omega commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1020813695 [05:04:24] [02ManageWiki] 07Universal-Omega commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1020814985 [05:05:36] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.95, 10.17, 9.35 [05:06:55] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.92, 10.11, 9.15 [05:13:14] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.78, 2.18, 1.60 [05:14:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.25, 9.92, 8.77 [05:14:24] [02ManageWiki] 07Universal-Omega closed pull request 03#328: Bump nanoid from 3.1.25 to 3.2.0 - 13https://github.com/miraheze/ManageWiki/pull/328 [05:14:26] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/a0b7168b5c98...e664c960b3f5 [05:14:27] [02miraheze/ManageWiki] 07dependabot[bot] 03e664c96 - Bump nanoid from 3.1.25 to 3.2.0 (#328) [05:14:29] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/nanoid-3.2.0 [05:14:30] [02ManageWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/nanoid-3.2.0 - 13https://github.com/miraheze/ManageWiki [05:15:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.76, 1.64, 1.50 [05:16:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.34, 9.16, 9.53 [05:16:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.80, 8.61, 8.47 [05:17:38] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.85, 9.13, 9.44 [05:19:23] miraheze/ManageWiki - Universal-Omega the build passed. [05:19:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.13, 8.24, 8.34 [05:20:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.18, 9.05, 9.87 [05:21:34] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.68, 9.64, 8.41 [05:24:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:25:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 0.70, 2.55, 2.36 [05:27:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.30, 9.30, 8.65 [05:30:40] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.27, 7.62, 8.41 [05:32:39] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.48, 8.16, 8.41 [05:33:32] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.33, 1.65, 1.97 [05:34:05] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.58, 7.46, 8.36 [05:36:28] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 7.71, 3.21, 2.44 [05:37:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:39:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 9.50, 10.09, 9.38 [05:39:27] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [05:41:36] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [05:42:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.64, 8.87, 9.00 [05:42:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:42:48] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [05:44:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.34, 9.27, 9.86 [05:44:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.84, 1.54, 1.92 [05:47:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 9.86, 9.95, 10.05 [05:47:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:47:44] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.01, 8.89, 8.85 [05:48:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.93, 7.76, 8.32 [05:50:52] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.99, 1.12, 1.61 [05:57:44] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [05:58:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.19, 8.50, 8.52 [06:00:10] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [06:00:48] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [06:00:59] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.96, 8.32, 8.45 [06:02:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.02, 9.95, 9.97 [06:02:41] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [06:07:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.21, 8.49, 9.55 [06:08:36] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [06:08:50] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [06:09:12] [02ManageWiki] 07lens0021 edited pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329 [06:11:21] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.70, 3.30, 2.03 [06:16:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.98, 7.72, 8.48 [06:17:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:17:40] [02ManageWiki] 07lens0021 commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1020848082 [06:22:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.48, 8.37, 8.58 [06:23:01] [02ManageWiki] 07Universal-Omega commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1020851136 [06:24:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:29:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:34:02] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.37, 9.12, 8.78 [06:34:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.66, 9.68, 9.11 [06:36:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.26, 8.29, 8.52 [06:37:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:37:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.38, 8.38, 8.72 [06:39:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.02, 7.76, 8.29 [06:42:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:47:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:48:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.43, 8.78, 8.64 [06:51:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.63, 9.58, 8.94 [06:52:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:54:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.15, 9.16, 8.91 [06:56:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:57:17] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.60, 9.46, 9.01 [07:00:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.85, 8.68, 8.78 [07:01:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:03:45] [02mw-config] 07RhinosF1 commented on pull request 03#4437: Sitenotice: add notice about ES - 13https://github.com/miraheze/mw-config/pull/4437#issuecomment-1020870841 [07:08:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.08, 7.75, 8.37 [07:18:38] [02ManageWiki] 07lens0021 synchronize pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329 [07:19:21] [02ManageWiki] 07lens0021 synchronize pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329 [07:19:27] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.70, 9.04, 8.84 [07:19:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:22:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.54, 8.37, 8.61 [07:22:49] miraheze/ManageWiki - lens0021 the build passed. [07:26:25] miraheze/ManageWiki - lens0021 the build passed. [07:29:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:30:19] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.62, 7.80, 8.31 [07:40:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:45:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:02:50] [02ManageWiki] 07Universal-Omega commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1020907026 [08:25:58] PROBLEM - wiki.yapsavun.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.yapsavun.xyz could not be found [08:26:45] Reception123: can you merge elastica composer [08:30:57] [02puppet] 07Reception123 closed pull request 03#2392: Add Elastica composer - 13https://github.com/miraheze/puppet/pull/2392 [08:30:58] RhinosF1: done [08:30:59] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/35067f247159...f9cb682aa7e1 [08:31:00] [02miraheze/puppet] 07Universal-Omega 03f9cb682 - Add Elastica composer (#2392) [08:31:31] Reception123: someone will need to deploy at some point [08:31:43] No rush as it's not deployed anywhere [08:31:48] I plan to do beta later [08:32:08] sounds good, I see we've got some wikis volunteering for CS [08:36:47] Yes [08:36:55] Ideally we need some more [08:45:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 3.58, 2.63 [08:45:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:47:55] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.74, 3.03, 2.57 [08:50:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:04:26] !log inserted content_models and slot_roles for vilemaximwiki [09:04:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:16:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:21:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:23:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.91, 8.95, 8.43 [09:24:12] [02ManageWiki] 07lens0021 edited pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329 [09:25:49] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.92, 8.16, 8.22 [09:32:26] [02ManageWiki] 07lens0021 commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1020984633 [09:35:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:40:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:45:26] [02ManageWiki] 07Universal-Omega commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1020995380 [09:46:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:51:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.78, 9.85, 8.25 [09:53:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.52, 9.47, 8.39 [09:54:11] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.01, 1.14, 1.84 [09:57:14] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.02, 10.12, 8.82 [09:57:34] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.00, 1.07, 1.67 [09:58:00] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.64, 9.89, 8.52 [10:00:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.47, 8.94, 8.57 [10:00:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.10, 8.13, 8.07 [10:01:00] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 2.30, 4.14, 3.16 [10:01:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.68, 3.43, 3.01 [10:01:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:02:49] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.64, 7.58, 8.11 [10:03:40] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.61, 3.18, 2.96 [10:03:51] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 0.98, 2.58, 2.77 [10:12:07] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.75, 1.64, 1.31 [10:12:25] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.55, 2.75, 2.07 [10:14:52] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.70, 1.24, 1.21 [10:34:39] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.01, 1.39, 1.85 [10:37:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.11, 8.82, 8.20 [10:40:09] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.04, 1.18, 1.61 [10:42:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:45:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.53, 8.63, 7.86 [10:46:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.87, 8.30, 8.26 [10:47:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:48:09] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.38, 8.01, 7.74 [10:52:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.15, 8.81, 8.11 [10:53:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:54:56] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.95, 1.79, 1.23 [10:57:45] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.53, 1.22, 1.10 [11:00:39] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.80, 8.15, 8.11 [11:03:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:10:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:15:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:23:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.84, 9.60, 8.26 [11:24:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.64, 8.32, 7.99 [11:25:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:26:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.92, 9.27, 8.37 [11:29:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.29, 9.29, 8.53 [11:30:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:32:17] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.27, 8.45, 8.33 [11:34:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.63, 8.21, 8.36 [11:42:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:43:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.05, 8.80, 8.57 [11:46:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.41, 7.30, 8.03 [11:47:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:53:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.03, 9.82, 8.83 [11:54:08] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.84, 1.66, 1.27 [11:56:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.77, 8.61, 8.54 [11:58:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:59:35] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.59, 1.66, 1.42 [12:01:35] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.14, 8.92, 8.66 [12:03:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:04:18] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.16, 8.05, 8.37 [12:06:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.12, 8.28, 7.85 [12:08:49] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.37, 7.88, 7.77 [12:09:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:22:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.26, 9.27, 8.64 [12:24:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:27:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.53, 8.80, 8.33 [12:27:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:30:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.01, 10.56, 9.13 [12:31:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.14, 10.74, 9.56 [12:33:31] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.33, 2.84, 2.03 [12:33:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.45, 9.87, 9.09 [12:36:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.37, 10.16, 9.34 [12:36:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.62, 9.31, 9.35 [12:39:11] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.59, 1.61, 1.76 [12:39:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.08, 9.85, 9.57 [12:41:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.32, 9.85, 9.06 [12:42:02] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.66, 1.16, 1.55 [12:42:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.81, 9.18, 9.14 [12:42:18] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.85, 8.83, 7.95 [12:44:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.88, 9.97, 9.25 [12:45:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.10, 9.57, 9.26 [12:46:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.30, 10.64, 9.56 [12:48:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.75, 9.77, 8.56 [12:50:26] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.59, 10.42, 8.82 [12:50:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.46, 9.93, 9.62 [12:51:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.14, 9.72, 8.78 [12:52:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.98, 9.11, 9.30 [12:53:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.02, 8.99, 8.59 [12:56:16] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.30, 8.42, 8.40 [12:58:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.04, 9.60, 9.38 [12:59:36] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.41, 9.78, 9.28 [12:59:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.14, 9.64, 10.00 [13:00:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.39, 9.46, 8.96 [13:01:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.51, 9.50, 9.37 [13:02:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.37, 10.04, 10.10 [13:02:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.64, 9.03, 8.88 [13:05:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.61, 9.51, 9.06 [13:06:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.60, 8.95, 8.28 [13:08:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.09, 9.64, 9.56 [13:08:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.44, 9.69, 9.97 [13:08:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.51, 8.95, 8.91 [13:09:28] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.52, 8.39, 8.19 [13:09:46] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.15, 10.02, 9.72 [13:11:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.56, 10.35, 9.47 [13:12:38] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.38, 1.77, 1.44 [13:14:11] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.71, 9.21, 9.59 [13:14:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.31, 9.41, 9.29 [13:15:30] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.23, 1.70, 1.47 [13:22:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.63, 7.28, 8.43 [13:23:13] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:24:12] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.66, 8.81, 9.66 [13:26:15] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.499 second response time [13:26:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.75, 7.51, 8.40 [13:29:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.90, 9.41, 9.65 [13:32:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.35, 8.88, 9.40 [13:35:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.86, 9.52, 9.53 [13:38:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.68, 9.25, 9.41 [13:42:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:43:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.37, 8.84, 8.48 [13:43:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.47, 9.70, 9.51 [13:47:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:47:56] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.17, 9.97, 8.96 [13:49:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.00, 9.46, 9.52 [13:50:44] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.78, 8.95, 8.46 [13:50:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.33, 9.47, 8.93 [13:53:38] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.09, 9.74, 8.84 [13:53:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.16, 10.51, 9.41 [13:54:51] PROBLEM - docse.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'docse.tk' expires in 15 day(s) (Thu 10 Feb 2022 13:45:14 GMT +0000). [13:56:33] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.62, 9.37, 8.83 [13:58:23] PROBLEM - id.altilunium.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'id.altilunium.xyz' expires in 15 day(s) (Thu 10 Feb 2022 13:52:57 GMT +0000). [13:59:28] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.69, 10.07, 9.17 [13:59:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.27, 9.25, 8.53 [13:59:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.30, 9.31, 9.35 [13:59:59] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bdfa239e1dd5...9845e948729f [14:00:00] [02miraheze/ssl] 07MirahezeSSLBot 039845e94 - Bot: Update SSL cert for docse.tk [14:00:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9845e948729f...6087266375ae [14:00:42] [02miraheze/ssl] 07MirahezeSSLBot 036087266 - Bot: Update SSL cert for id.altilunium.xyz [14:01:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.08, 10.07, 9.82 [14:02:16] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.07, 9.40, 9.09 [14:02:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.14, 9.80, 9.53 [14:07:57] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.14, 9.00, 8.93 [14:08:20] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.02, 9.26, 8.84 [14:10:43] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.70, 8.69, 8.88 [14:12:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.16, 9.74, 9.08 [14:14:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.04, 9.88, 9.83 [14:14:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.91, 9.35, 9.12 [14:15:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.93, 8.73, 8.82 [14:15:21] PROBLEM - conlang.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'conlang.tk' expires in 15 day(s) (Thu 10 Feb 2022 13:48:47 GMT +0000). [14:16:56] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 9.93, 10.34, 10.05 [14:17:37] PROBLEM - notes.mridulpm.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'notes.mridulpm.in' expires in 15 day(s) (Thu 10 Feb 2022 13:50:05 GMT +0000). [14:18:29] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6087266375ae...3c84fd719bc4 [14:18:30] [02miraheze/ssl] 07MirahezeSSLBot 033c84fd7 - Bot: Update SSL cert for conlang.tk [14:21:47] PROBLEM - wiki.alathramc.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.alathramc.com' expires in 15 day(s) (Thu 10 Feb 2022 13:53:33 GMT +0000). [14:22:50] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.18, 9.03, 8.97 [14:23:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3c84fd719bc4...2db5b9351bb6 [14:23:35] [02miraheze/ssl] 07MirahezeSSLBot 032db5b93 - Bot: Update SSL cert for notes.mridulpm.in [14:24:04] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.18, 9.32, 8.95 [14:24:10] PROBLEM - diamowiki.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'diamowiki.ga' expires in 15 day(s) (Thu 10 Feb 2022 13:54:32 GMT +0000). [14:25:50] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.38, 8.53, 8.81 [14:27:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.50, 8.67, 8.77 [14:27:54] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.37, 9.50, 9.00 [14:28:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.48, 9.21, 9.83 [14:28:44] RECOVERY - id.altilunium.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'id.altilunium.xyz' will expire on Mon 25 Apr 2022 13:00:35 GMT +0000. [14:28:50] PROBLEM - vrcdev.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vrcdev.wiki' expires in 15 day(s) (Thu 10 Feb 2022 13:56:08 GMT +0000). [14:29:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.02, 9.31, 9.97 [14:29:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.14, 9.21, 8.92 [14:30:44] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.39, 9.19, 8.95 [14:31:40] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.89, 7.81, 8.39 [14:31:44] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2db5b9351bb6...e44404f4ff43 [14:31:46] [02miraheze/ssl] 07MirahezeSSLBot 03e44404f - Bot: Update SSL cert for wiki.alathramc.com [14:32:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:32:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.93, 8.12, 8.55 [14:34:16] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.16, 10.28, 10.06 [14:35:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.56, 8.24, 9.77 [14:35:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.12, 9.23, 9.68 [14:35:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.64, 7.85, 8.37 [14:36:19] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.91, 7.84, 8.40 [14:37:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.06, 9.41, 9.74 [14:37:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:37:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.76, 8.70, 9.38 [14:39:11] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e44404f4ff43...1f32b6e0efd7 [14:39:12] [02miraheze/ssl] 07MirahezeSSLBot 031f32b6e - Bot: Update SSL cert for vrcdev.wiki [14:40:44] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.10, 9.48, 9.81 [14:41:07] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1f32b6e0efd7...dd2378f73242 [14:41:08] [02miraheze/ssl] 07MirahezeSSLBot 03dd2378f - Bot: Update SSL cert for diamowiki.ga [14:41:12] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.88, 9.78, 8.96 [14:43:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.37, 9.44, 9.75 [14:44:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.99, 9.38, 8.94 [14:44:54] RECOVERY - conlang.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'conlang.tk' will expire on Mon 25 Apr 2022 13:18:23 GMT +0000. [14:45:18] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.68, 8.93, 8.82 [14:46:27] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.40, 9.62, 9.46 [14:46:29] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.91, 9.85, 9.83 [14:49:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.64, 8.92, 9.24 [14:49:53] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.69, 7.39, 8.24 [14:54:24] RECOVERY - docse.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'docse.tk' will expire on Mon 25 Apr 2022 12:59:53 GMT +0000. [14:54:57] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.81, 9.47, 9.70 [14:56:49] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.10, 8.02, 8.37 [14:57:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:57:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.64, 9.57, 9.40 [14:58:19] RECOVERY - vrcdev.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'vrcdev.wiki' will expire on Mon 25 Apr 2022 13:39:05 GMT +0000. [15:00:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.44, 9.34, 9.35 [15:08:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.64, 7.69, 8.19 [15:11:51] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.45, 8.49, 7.92 [15:12:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:12:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.30, 10.06, 9.03 [15:13:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.94, 8.62, 8.08 [15:15:11] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.11, 8.92, 8.61 [15:16:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.32, 8.18, 8.01 [15:17:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.10, 8.81, 8.83 [15:18:06] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.44, 10.14, 9.15 [15:18:07] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.11, 9.81, 9.49 [15:18:23] RECOVERY - notes.mridulpm.in - LetsEncrypt on sslhost is OK: OK - Certificate 'notes.mridulpm.in' will expire on Mon 25 Apr 2022 13:23:29 GMT +0000. [15:18:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.64, 3.10 [15:18:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.94, 9.22, 9.00 [15:21:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.82, 9.75, 9.17 [15:21:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.52, 3.26, 3.05 [15:22:14] RECOVERY - wiki.alathramc.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.alathramc.com' will expire on Mon 25 Apr 2022 13:31:38 GMT +0000. [15:23:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.98, 9.51, 9.22 [15:23:52] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.25, 10.15, 9.40 [15:25:01] RECOVERY - diamowiki.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'diamowiki.ga' will expire on Mon 25 Apr 2022 13:41:01 GMT +0000. [15:26:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.86, 9.90, 9.37 [15:26:43] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.52, 1.84, 1.49 [15:26:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.59, 9.77, 9.41 [15:29:45] !log [rhinos@test101] starting deploy of {'world': True} to skip [15:29:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:30:09] !log [rhinos@mwtask111] starting deploy of {'world': True, 'force': True} to scsvg [15:30:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:31:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.60, 3.28 [15:32:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.92, 9.37, 8.85 [15:33:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.10, 9.82, 9.99 [15:33:04] Reception123: I'm gonna enable ES on beta soon [15:33:35] !log [rhinos@test101] finished deploy of {'world': True} to skip - SUCCESS in 230s [15:33:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:34:01] [02mw-config] 07RhinosF1 closed pull request 03#4436: Betawiki: configure guarded ES cluster - 13https://github.com/miraheze/mw-config/pull/4436 [15:34:03] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 [15:34:04] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 - 13https://github.com/miraheze/mw-config [15:34:06] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f9d63fb01031...0e926054e560 [15:34:07] [02miraheze/mw-config] 07RhinosF1 030e92605 - Betawiki: configure guarded ES cluster (#4436) [15:34:22] !log [rhinos@test101] starting deploy of {'config': True} to skip [15:34:23] !log [rhinos@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [15:34:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.65, 3.18, 3.18 [15:34:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:34:49] !log [rhinos@test101] sudo -u www-data php /srv/mediawiki/w/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=betawiki (END - exit=256) [15:34:54] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.43, 8.85, 8.73 [15:35:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:35:13] miraheze/mw-config - RhinosF1 the build passed. [15:35:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:35:19] !log [rhinos@test101] starting deploy of {'config': True} to skip [15:35:20] !log [rhinos@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [15:35:24] !log [rhinos@test101] sudo -u www-data php /srv/mediawiki/w/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=betawiki (END - exit=65280) [15:35:31] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.98, 1.50, 1.52 [15:35:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:35:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:35:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:36:33] !log [rhinos@mwtask111] finished deploy of {'world': True, 'force': True} to scsvg - SUCCESS in 384s [15:36:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:37:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:38:25] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0e926054e560...56a75e62ff11 [15:38:26] [02miraheze/mw-config] 07RhinosF1 0356a75e6 - Update LocalWiki.php [15:38:34] !log [rhinos@test101] starting deploy of {'config': True} to skip [15:38:35] !log [rhinos@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [15:38:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:38:47] !log [rhinos@test101] starting deploy of {'config': True} to skip [15:38:48] !log [rhinos@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [15:38:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.14, 10.12, 10.08 [15:38:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:38:55] !log [rhinos@test101] sudo -u www-data php /srv/mediawiki/w/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=betawiki (END - exit=256) [15:39:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:39:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:39:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:39:37] miraheze/mw-config - RhinosF1 the build passed. [15:39:43] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/56a75e62ff11...8e3651de2693 [15:39:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.43, 9.02, 8.82 [15:39:45] [02miraheze/mw-config] 07RhinosF1 038e3651d - Update LocalWiki.php [15:39:51] !log [rhinos@test101] starting deploy of {'config': True} to skip [15:39:52] !log [rhinos@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [15:39:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:40:01] !log [rhinos@test101] sudo -u www-data php /srv/mediawiki/w/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=betawiki (END - exit=256) [15:40:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:40:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:40:29] paladox: hey, i'm getting operation timed out connecting to ES [15:40:53] miraheze/mw-config - RhinosF1 the build passed. [15:40:58] firewall and we only use https for es. https://es101.miraheze.org:9200 [15:41:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:41:46] paladox: is firewall open? [15:41:54] Not to mediawiki [15:42:09] paladox: how is it ready for us to setup then? [15:42:24] I wasn't the one that said it was. [15:42:36] i don't know who told you that it was? [15:43:06] oh hold on [15:43:09] paladox: "The current es* cluster set up should be able to handle a limited deploy of CirrusSearch to wikis. Note it is on HDDs though. " - John [15:43:18] https://github.com/miraheze/puppet/blob/master/modules/role/manifests/elasticsearch.pp#L70 [15:43:19] [url] puppet/elasticsearch.pp at master · miraheze/puppet · GitHub | github.com [15:43:21] it's open [15:43:22] also the docs for mediawiki don't mention TLS [15:43:29] you have to use https://es101.miraheze.org [15:44:14] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8e3651de2693...d24e9eb931eb [15:44:16] [02miraheze/mw-config] 07RhinosF1 03d24e9eb - Update LocalWiki.php [15:44:26] !log [rhinos@test101] starting deploy of {'config': True} to skip [15:44:27] !log [rhinos@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [15:44:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:44:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:45:25] RhinosF1: no, you direct all quaries through es101 [15:45:30] and no it doesn't use port 9200 [15:45:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.95, 9.85, 9.30 [15:45:35] miraheze/mw-config - RhinosF1 the build passed. [15:45:37] the only port open is https [15:47:44] https://github.com/wikimedia/operations-mediawiki-config/blob/091bb3a695612ea67f86dfd935716ed2ecce14f3/wmf-config/CirrusSearch-labs.php#L16 [15:47:44] [url] operations-mediawiki-config/CirrusSearch-labs.php at 091bb3a695612ea67f86dfd935716ed2ecce14f3 · wikimedia/operations-mediawiki-config · GitHub | github.com [15:48:35] paladox: when i asked i was told it's running on 9200 [15:48:39] !log [rhinos@test101] sudo -u www-data php /srv/mediawiki/w/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=betawiki (END - exit=2) [15:48:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:52:57] paladox: what port is actually open for mediawiki to connect via [15:53:17] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.94, 9.42, 9.01 [15:53:37] it's https there is no other port. https://es101.miraheze.org. [15:53:48] see https://github.com/miraheze/puppet/blob/master/modules/role/manifests/elasticsearch.pp#L70 [15:53:49] [url] puppet/elasticsearch.pp at master · miraheze/puppet · GitHub | github.com [15:54:28] paladox: what version are we running? [15:54:35] version 7 [15:54:57] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.51, 7.93, 8.33 [15:55:19] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d24e9eb931eb...639ce6971192 [15:55:20] [02miraheze/mw-config] 07RhinosF1 03639ce69 - Update LocalWiki.php [15:55:29] !log [rhinos@test101] starting deploy of {'config': True} to skip [15:55:30] !log [rhinos@test101] DEPLOY ABORTED: Canary check failed for localhost [15:55:34] !log [rhinos@test101] starting deploy of {'config': True, 'force': True} to skip [15:55:35] !log [rhinos@test101] finished deploy of {'config': True, 'force': True} to skip - SUCCESS in 0s [15:55:41] !log [rhinos@test101] sudo -u www-data php /srv/mediawiki/w/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=betawiki (END - exit=65280) [15:55:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:56:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:56:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:56:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.87, 9.07, 8.94 [15:56:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:56:16] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/639ce6971192...625e20dfeb5b [15:56:17] RhinosF1: i just said do not use 'https://es111.miraheze.org', 'https://es121.miraheze.org'. I said put everything through es101 as it's the master. [15:56:18] [02miraheze/mw-config] 07RhinosF1 03625e20d - Update LocalWiki.php [15:56:24] you won't be able to connect to 'https://es111.miraheze.org', 'https://es121.miraheze.org' [15:56:24] !log [rhinos@test101] starting deploy of {'config': True, 'force': True} to skip [15:56:25] !log [rhinos@test101] finished deploy of {'config': True, 'force': True} to skip - SUCCESS in 0s [15:56:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:56:30] miraheze/mw-config - RhinosF1 the build has errored. [15:56:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:56:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:57:13] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/625e20dfeb5b...338c71ce2a6a [15:57:15] [02miraheze/mw-config] 07RhinosF1 03338c71c - Update LocalWiki.php [15:57:21] miraheze/mw-config - RhinosF1 the build has errored. [15:57:37] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/338c71ce2a6a...d379b39e2be9 [15:57:38] [02miraheze/mw-config] 07RhinosF1 03d379b39 - Update LocalWiki.php [15:58:04] paladox: we can't even use version 7 [15:58:09] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d379b39e2be9...18a711d783da [15:58:10] [02miraheze/mw-config] 07github-actions 0318a711d - CI: lint code to MediaWiki standards [15:58:17] oh, i guess we cannot use ES for mw then [15:58:20] miraheze/mw-config - RhinosF1 the build has errored. [15:58:49] miraheze/mw-config - RhinosF1 the build passed. [15:59:26] [02miraheze/mw-config] 07RhinosF1 pushed 039 commits to 03master [+0/-0/±9] 13https://github.com/miraheze/mw-config/compare/18a711d783da...b61b69ab5e9f [15:59:28] [02miraheze/mw-config] 07RhinosF1 03cdc05dd - Revert "CI: lint code to MediaWiki standards" [15:59:29] [02miraheze/mw-config] 07RhinosF1 03d89562c - Revert "Update LocalWiki.php" [15:59:31] [02miraheze/mw-config] 07RhinosF1 03a24cb49 - Revert "Update LocalWiki.php" [15:59:32] [02miraheze/mw-config] ... and 6 more commits. [15:59:54] !log [rhinos@test101] starting deploy of {'config': True, 'force': True} to skip [15:59:55] !log [rhinos@test101] finished deploy of {'config': True, 'force': True} to skip - SUCCESS in 0s [16:00:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:00:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:00:38] miraheze/mw-config - RhinosF1 the build passed. [16:00:50] paladox: https://phabricator.miraheze.org/T7740#175701 [16:00:52] [url] ⚓ T7740 Reconsider production usage of Elastic/CirrusSearch | phabricator.miraheze.org [16:01:04] ok [16:02:14] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.27, 9.76, 9.31 [16:04:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.71, 8.72, 8.50 [16:04:54] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.62, 8.90, 8.54 [16:07:38] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.21, 8.42, 8.41 [16:08:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.51, 9.57, 9.41 [16:09:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.88, 9.65, 9.03 [16:10:46] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.76, 10.22, 9.16 [16:14:06] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.12, 9.94, 9.62 [16:18:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.99, 9.56, 9.46 [16:19:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.27, 9.98, 9.81 [16:22:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.25, 9.73, 9.54 [16:25:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.34, 10.10, 9.82 [16:28:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.35, 9.49, 9.65 [16:33:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.06, 9.88, 9.34 [16:34:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.18, 9.78, 9.71 [16:36:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.65, 8.84, 9.03 [16:37:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.98, 9.10, 9.44 [16:43:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.53, 9.40, 9.41 [16:49:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.72, 9.91, 9.62 [16:51:10] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.39, 8.81, 8.39 [16:52:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.23, 7.91, 8.38 [16:54:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.60, 9.77, 9.11 [16:55:17] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/43b651b816d0 [16:55:19] [02miraheze/mw-config] 07RhinosF1 0343b651b - UserMerge: undeploy; no longer supported [16:55:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.38, 10.02, 9.70 [16:55:20] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-2 - 13https://github.com/miraheze/mw-config [16:55:22] [02mw-config] 07RhinosF1 opened pull request 03#4438: UserMerge: undeploy; no longer supported - 13https://github.com/miraheze/mw-config/pull/4438 [16:55:50] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/43b651b816d0...a3f6d6a55133 [16:55:51] [02miraheze/mw-config] 07RhinosF1 03a3f6d6a - Update GlobalExtensions.php [16:55:53] [02mw-config] 07RhinosF1 synchronize pull request 03#4438: UserMerge: undeploy; no longer supported - 13https://github.com/miraheze/mw-config/pull/4438 [16:56:31] miraheze/mw-config - RhinosF1 the build passed. [16:56:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.03, 9.72, 9.23 [16:57:12] miraheze/mw-config - RhinosF1 the build passed. [16:58:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.97, 9.92, 9.74 [17:00:49] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/a69bb1e21875 [17:00:51] [02miraheze/mw-config] 07RhinosF1 03a69bb1e - LS: disable usermerge [17:00:52] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-4 - 13https://github.com/miraheze/mw-config [17:00:54] [02mw-config] 07RhinosF1 opened pull request 03#4439: LS: disable usermerge - 13https://github.com/miraheze/mw-config/pull/4439 [17:01:28] JohnLewis: can we remove ES as a goal [17:02:00] Based on my chat with search team, we are highly unlikely to get it this period [17:02:07] miraheze/mw-config - RhinosF1 the build passed. [17:02:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.07, 8.84, 8.54 [17:02:30] Reception is the one to talk to about goals [17:02:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.36, 9.52, 9.99 [17:02:52] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.57, 9.79, 9.04 [17:04:08] Reception123: ^ [17:04:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.18, 9.83, 9.61 [17:04:42] JohnLewis: also fyi as a steward to the 2 PRs I just sent [17:05:38] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.62, 10.12, 10.11 [17:05:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.76, 9.14, 8.91 [17:06:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.39, 9.56, 9.27 [17:06:49] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.86, 1.90, 1.52 [17:07:26] RhinosF1: yeah if it's upstream we can remove as we can't realistically do anything about it [17:08:00] hi Reception123 :D [17:08:09] Hi [17:08:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.84, 9.28, 8.77 [17:09:43] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.37, 2.17, 1.69 [17:10:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.01, 9.22, 9.48 [17:11:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.90, 9.13, 8.81 [17:12:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.53, 1.79, 1.63 [17:14:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.42, 9.42, 9.37 [17:15:25] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.43, 1.58, 1.57 [17:16:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:16:52] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.60, 9.29, 8.98 [17:17:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.66, 9.78, 9.92 [17:17:20] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.47, 2.59, 2.18 [17:17:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.26, 9.61, 9.43 [17:19:59] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.38, 8.68, 8.80 [17:20:04] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 9.28, 10.01, 10.01 [17:20:28] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.63, 1.71, 1.91 [17:20:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.23, 9.24, 9.31 [17:23:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.07, 7.97, 8.39 [17:23:29] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.46, 1.19, 1.67 [17:23:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:23:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.51, 9.67, 9.46 [17:26:00] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.92, 7.70, 8.35 [17:26:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.87, 9.55, 9.44 [17:36:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.95, 10.25, 9.51 [17:41:34] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.37, 10.15, 9.49 [17:42:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.71, 9.69, 9.08 [17:45:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.33, 4.14, 3.42 [17:46:35] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.69, 9.96, 9.12 [17:48:04] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.29, 10.08, 9.38 [17:48:19] [02mw-config] 07Universal-Omega reviewed pull request 03#4439 commit - 13https://github.com/miraheze/mw-config/pull/4439#discussion_r791974141 [17:49:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.80, 8.61, 8.77 [17:50:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.64, 9.95, 9.72 [17:51:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.08, 8.60, 8.92 [17:51:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.84, 9.61, 9.82 [17:54:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.18, 9.55, 9.74 [17:55:32] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.29, 8.90, 8.78 [17:56:33] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.05, 9.31, 9.45 [17:57:02] PROBLEM - puppet111 APT on puppet111 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [17:58:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.90, 8.67, 8.70 [17:59:48] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.85, 3.74, 2.93 [18:01:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.01, 4.08, 3.29 [18:02:42] PROBLEM - es101 Current Load on es101 is CRITICAL: CRITICAL - load average: 4.76, 3.92, 3.12 [18:02:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.20, 9.30, 9.43 [18:03:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.32, 9.28, 9.55 [18:03:58] PROBLEM - cloud10 SMART on cloud10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:04:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.78, 3.58, 3.24 [18:06:00] [02ManageWiki] 07lens0021 commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1021465791 [18:07:32] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.01, 3.31, 3.19 [18:07:50] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.97, 8.18, 8.47 [18:08:27] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.37, 3.84, 3.37 [18:08:40] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.21, 7.25, 8.28 [18:09:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.40, 8.92, 9.72 [18:10:02] RECOVERY - cloud10 SMART on cloud10 is OK: OK: [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean [18:11:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.69, 9.52, 9.38 [18:12:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.90, 9.37, 9.40 [18:14:24] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 2.49, 3.19, 3.24 [18:14:52] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.31, 9.26, 8.94 [18:15:24] [02ManageWiki] 07lens0021 edited a comment on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1021465791 [18:15:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.61, 8.47, 8.56 [18:15:44] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.25, 10.32, 10.06 [18:17:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.30, 8.25, 9.01 [18:21:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.53, 9.39, 9.42 [18:22:05] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.29, 7.96, 8.32 [18:23:10] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:33] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:48] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:24:02] [02ManageWiki] 07lens0021 commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1021480953 [18:24:43] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.54, 7.49, 8.21 [18:26:30] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.018 second response time [18:26:56] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.082 second response time [18:27:08] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.199 second response time [18:28:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.54, 9.63, 9.43 [18:31:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.14, 8.83, 9.15 [18:32:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b61b69ab5e9f...a053627f61bd [18:32:22] [02miraheze/mw-config] 07Universal-Omega 03a053627 - Update GlobalCache.php [18:33:29] miraheze/mw-config - Universal-Omega the build passed. [18:34:19] [02ManageWiki] 07Universal-Omega commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1021489781 [18:34:39] CosmicAlpha: if I still have puppet locked then feel free to release [18:34:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.32, 10.82, 9.89 [18:35:04] I locked it when testing last 2 days [18:38:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.01, 2.87, 3.76 [18:40:06] RhinosF1: I'm not at PC. I didn't mean to push that to master... I meant to push that to a PR, though I'm sure it won't cause issues so I didn't undo it. [18:40:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.22, 9.53, 8.94 [18:40:36] CosmicAlpha: puppet is disabled I believe still so it's not deployed [18:40:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.62, 9.35, 8.63 [18:40:44] Up to you what to do [18:40:49] I can free it later [18:41:47] RhinosF1: you can enable later if you want. [18:41:54] K [18:43:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.42, 8.60, 8.43 [18:43:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.59, 9.79, 9.17 [18:44:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.68, 2.98, 3.44 [18:46:28] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 4.05, 7.12, 7.94 [18:47:05] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 4.65, 8.30, 9.39 [18:47:17] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 4.67, 7.33, 8.15 [18:48:12] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.31, 2.24, 3.06 [18:48:22] !log [@test101] starting deploy of {'config': True} to skip [18:48:23] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [18:48:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:49:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:50:47] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 5.35, 7.12, 9.24 [18:56:15] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.67, 1.95, 1.57 [18:56:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.20, 10.47, 9.41 [18:56:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.51, 9.72, 9.68 [18:57:41] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.60, 8.85, 8.43 [18:59:22] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.55, 1.43, 1.44 [18:59:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.99, 10.02, 9.58 [18:59:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.71, 9.78, 9.35 [19:00:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.47, 9.91, 8.78 [19:00:33] [02puppet] 07Universal-Omega opened pull request 03#2394: Revert "Add Elastica composer" - 13https://github.com/miraheze/puppet/pull/2394 [19:01:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4435-RhinosF1-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/7be2f2214f1f [19:01:39] [02miraheze/mw-config] 07Universal-Omega 037be2f22 - Revert "ext-list: register ES extensions (#4435)" [19:01:41] [02mw-config] 07Universal-Omega created branch 03revert-4435-RhinosF1-patch-2 - 13https://github.com/miraheze/mw-config [19:01:42] [02mw-config] 07Universal-Omega opened pull request 03#4440: Revert "ext-list: register ES extensions" - 13https://github.com/miraheze/mw-config/pull/4440 [19:02:50] miraheze/mw-config - Universal-Omega the build passed. [19:05:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.87, 9.82, 9.40 [19:05:59] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.90, 9.64, 9.10 [19:08:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.01, 9.76, 9.76 [19:09:53] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.40, 10.01, 9.18 [19:12:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.41, 3.45, 3.02 [19:12:52] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.45, 9.60, 9.17 [19:15:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.69, 3.26, 3.02 [19:17:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.68, 10.83, 10.16 [19:20:51] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.35, 9.42, 9.08 [19:23:44] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.44, 9.66, 9.26 [19:26:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.71, 3.52, 3.22 [19:26:49] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.74, 10.61, 9.73 [19:27:57] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.21, 1.88, 1.57 [19:28:55] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [19:29:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.40, 3.22 [19:29:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.43, 9.85, 9.57 [19:30:56] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.39, 1.65, 1.53 [19:32:49] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 8.88, 10.14, 9.76 [19:33:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.88, 8.99, 9.72 [19:33:55] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [19:35:03] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [19:35:23] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.47, 3.11, 3.18 [19:35:46] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.07, 9.38, 9.52 [19:36:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.05, 9.32, 9.71 [19:36:48] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [19:37:05] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.10, 9.35, 9.03 [19:38:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.15, 10.86, 10.06 [19:39:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.84, 9.70, 9.81 [19:40:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.42, 9.32, 9.07 [19:41:32] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.08, 9.55, 9.70 [19:42:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.98, 10.16, 9.96 [19:42:53] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.71, 9.73, 9.25 [19:44:24] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.05, 9.82, 9.79 [19:47:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 4.05, 3.51 [19:47:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.19, 9.30, 9.58 [19:50:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.76, 3.47 [19:50:08] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.86, 9.48, 9.57 [19:53:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.65, 8.90, 9.34 [19:55:52] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.51, 9.70, 9.58 [19:56:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.59, 3.96, 3.64 [19:57:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.34, 9.83, 9.74 [19:58:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.57, 9.04, 9.38 [19:59:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.14, 3.66, 3.57 [20:00:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.57, 9.55, 9.98 [20:02:04] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.62, 3.09, 3.35 [20:03:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.67, 10.06, 10.11 [20:03:36] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.11, 9.41, 9.43 [20:09:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 4.39, 3.86 [20:09:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.81, 4.21, 3.27 [20:11:24] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.27, 4.10, 3.57 [20:12:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.88, 3.75, 3.70 [20:12:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.98, 3.09, 3.00 [20:12:52] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.44, 9.38, 9.51 [20:14:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.33, 9.12, 9.98 [20:14:21] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.83, 3.44, 3.37 [20:15:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.85, 4.44, 3.98 [20:16:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.40, 9.09, 8.96 [20:17:14] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.13, 2.94, 3.20 [20:19:33] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.50, 8.23, 8.64 [20:21:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.07, 3.56, 3.74 [20:21:43] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.58, 9.11, 9.13 [20:23:06] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.09, 10.02, 9.96 [20:24:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.60, 9.22, 9.21 [20:31:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.02, 8.75, 8.74 [20:33:32] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.53, 4.41, 3.91 [20:34:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.47, 9.33, 8.99 [20:37:57] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.46, 2.18, 1.59 [20:39:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.52, 3.80, 3.79 [20:39:35] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.58, 9.89, 9.41 [20:40:49] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.16, 1.74, 1.52 [20:42:28] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.97, 9.57, 9.41 [20:43:46] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.97, 1.37, 1.41 [20:45:24] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.56, 3.63 [20:53:44] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.81, 1.96, 1.63 [20:54:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.68, 7.62, 8.39 [20:56:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.01, 1.53, 1.52 [20:57:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.47, 3.24, 3.39 [21:01:23] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.51, 12.99, 10.74 [21:03:29] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.74, 10.40, 9.75 [21:06:33] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.84, 9.88, 9.66 [21:10:45] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.86, 1.85, 1.59 [21:11:22] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:11:46] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:12:33] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:12:41] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:12:54] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:13:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.64, 7.95, 9.24 [21:14:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.20, 8.42, 9.56 [21:14:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.18, 1.43, 1.47 [21:14:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.05, 8.07, 9.58 [21:14:50] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [21:15:14] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.730 second response time [21:15:58] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 3.260 second response time [21:16:02] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.420 second response time [21:16:15] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.014 second response time [21:17:23] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.18, 9.31, 9.71 [21:17:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.67, 9.39, 9.84 [21:19:43] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.66, 8.81, 8.81 [21:19:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.28, 10.05, 9.74 [21:22:49] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.97, 9.15, 8.95 [21:22:55] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/commit/4b1595ea8e82 [21:22:57] [02miraheze/ManageWiki] 07Universal-Omega 034b1595e - blacklistSkipSkins -> excludeSkipSkins [21:22:58] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ManageWiki [21:23:00] [02ManageWiki] 07Universal-Omega opened pull request 03#330: blacklistSkipSkins -> excludeSkipSkins - 13https://github.com/miraheze/ManageWiki/pull/330 [21:23:19] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.77, 9.72, 9.77 [21:25:38] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.33, 10.01, 9.31 [21:28:12] miraheze/ManageWiki - Universal-Omega the build passed. [21:28:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:28:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.80, 9.90, 9.40 [21:28:57] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.25, 10.49, 10.02 [21:37:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.61, 9.66, 9.42 [21:43:12] [02ManageWiki] 07Universal-Omega commented on pull request 03#329: Add a new type 'integers' - 13https://github.com/miraheze/ManageWiki/pull/329#issuecomment-1021636823 [21:46:32] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.46, 9.81, 9.76 [21:48:20] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.79, 3.25 [21:51:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.58, 3.27 [21:52:02] [02ManageWiki] 07Universal-Omega closed pull request 03#330: blacklistSkipSkins -> excludeSkipSkins - 13https://github.com/miraheze/ManageWiki/pull/330 [21:52:04] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/e664c960b3f5...951c544fd192 [21:52:05] [02miraheze/ManageWiki] 07Universal-Omega 03951c544 - blacklistSkipSkins -> excludeSkipSkins (#330) [21:52:07] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:52:08] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ManageWiki [21:52:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.98, 8.97, 9.75 [21:54:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.03, 2.89, 3.05 [21:55:52] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.71, 9.68, 9.68 [21:57:01] miraheze/ManageWiki - Universal-Omega the build passed. [21:59:00] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.14, 8.22, 9.13 [21:59:12] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.02, 1.92, 1.79 [22:01:38] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.66, 9.86, 9.75 [22:05:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.28, 1.52, 1.64 [22:08:12] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.77, 7.75, 8.45 [22:13:17] PROBLEM - mw121 APT on mw121 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:14:11] PROBLEM - mw102 APT on mw102 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:14:51] PROBLEM - cloud12 APT on cloud12 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:15:17] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.09, 8.99, 8.76 [22:15:19] PROBLEM - cloud10 APT on cloud10 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:16:32] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.72, 9.72, 9.86 [22:18:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.21, 8.97, 8.82 [22:19:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.09, 11.86, 10.68 [22:22:13] PROBLEM - mw112 APT on mw112 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:22:19] PROBLEM - test101 APT on test101 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:25:56] PROBLEM - cloud11 APT on cloud11 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:27:13] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:28:30] PROBLEM - es111 APT on es111 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [22:34:49] PROBLEM - mw101 APT on mw101 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:36:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.90, 9.18, 9.80 [22:36:51] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.07, 8.75, 8.62 [22:37:25] PROBLEM - mw122 APT on mw122 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:40:31] PROBLEM - mw111 APT on mw111 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:40:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.06, 9.65, 10.00 [22:42:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.04, 9.63, 9.05 [22:45:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.66, 10.52, 10.21 [22:46:27] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.52, 2.37, 1.47 [22:48:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.43, 9.20, 9.77 [22:55:55] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.21, 1.75, 1.63 [22:59:08] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.14, 1.44, 1.52 [23:00:01] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:08] !log [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg [23:00:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:49] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.67, 9.32, 8.99 [23:01:55] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.60, 9.48, 9.24 [23:02:21] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.19, 2.05, 1.61 [23:05:02] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.14, 9.08, 8.99 [23:05:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.93, 8.82, 9.03 [23:05:31] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.72, 1.44, 1.45 [23:10:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.24, 8.81, 8.89 [23:10:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.83, 9.56, 9.93 [23:11:09] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.53, 9.42, 9.15 [23:11:17] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.31, 9.69, 9.26 [23:14:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.38, 10.90, 10.35 [23:14:20] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.46, 9.85, 9.45 [23:15:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.07, 4.77, 3.79 [23:15:48] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.38, 2.05, 1.74 [23:16:09] !log [@test101] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 967s [23:16:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:16:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.30, 9.46, 9.21 [23:17:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.12, 9.92, 9.58 [23:17:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.68, 10.19, 9.65 [23:20:07] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.79, 9.76, 9.34 [23:20:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.90, 9.08, 9.75 [23:20:29] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.61, 9.59, 9.52 [23:21:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.14, 3.48, 3.58 [23:22:17] !log [@mwtask111] DEPLOY ABORTED: Canary check failed for mw101 [23:22:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:23:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.99, 9.68, 9.37 [23:25:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.30, 1.83, 1.86 [23:26:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.01, 9.93, 9.88 [23:27:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.20, 3.04, 3.39 [23:29:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.31, 9.38, 9.68 [23:29:43] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.12, 10.07, 9.68 [23:31:10] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.22, 1.36, 1.62 [23:32:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.69, 9.77, 9.51 [23:32:41] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.15, 9.52, 9.56 [23:35:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.95, 10.60, 9.99 [23:35:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.87, 8.55, 9.08 [23:38:28] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.57, 3.18, 2.32 [23:41:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.31, 10.24, 9.59 [23:43:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.52, 9.61, 9.95 [23:46:57] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.28, 9.48, 9.04 [23:47:33] PROBLEM - wiki.slimeapedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.slimeapedia.org could not be found [23:47:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.92, 9.86, 9.70 [23:47:55] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.45, 7.58, 8.48 [23:50:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.34, 9.27, 9.07 [23:52:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.17, 9.48, 9.49 [23:56:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.05, 9.92, 9.54