[00:03:42] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.26, 19.60, 18.91 [00:05:41] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.30, 22.30, 20.00 [00:07:17] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.93, 5.13, 4.31 [00:09:14] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 11.66, 7.09, 5.10 [00:12:09] PROBLEM - cp15 Puppet on cp15 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.rothwell-leeds.co.uk_private] [00:13:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.68, 3.48, 3.33 [00:15:18] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.47, 3.15, 2.28 [00:17:17] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 3.06, 3.13, 2.38 [00:19:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.34, 3.79, 3.48 [00:22:45] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.67, 6.65, 5.27 [00:24:39] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.90, 6.39, 5.35 [00:27:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.84, 3.66 [00:30:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.71, 3.63 [00:32:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.89, 3.76, 3.66 [00:36:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.18, 2.86, 3.31 [00:40:08] RECOVERY - cp15 Puppet on cp15 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:40:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.97, 3.85, 3.59 [00:44:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 4.00, 3.70 [00:54:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.95, 3.88, 3.66 [00:56:09] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.01, 4.59, 5.91 [00:58:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.68, 3.63 [01:00:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 3.89, 3.71 [01:01:00] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 13.00, 17.88, 23.67 [01:05:58] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.32, 4.10, 5.03 [01:10:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.83, 3.82 [01:10:52] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.91, 15.22, 19.82 [01:14:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.85, 3.82 [01:16:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.70, 3.77 [01:18:50] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.45, 4.17, 3.92 [01:24:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.64, 3.71, 3.86 [01:40:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.88, 4.21, 3.75 [01:44:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.94, 3.80, 3.68 [01:48:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.51, 4.06, 3.79 [01:52:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.92, 3.81 [01:56:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.73, 3.89, 3.79 [01:58:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.98, 3.63, 3.71 [02:00:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.57, 4.10, 3.86 [02:04:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.62, 3.90, 3.83 [02:22:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.24, 3.63, 3.54 [02:26:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.80, 3.59, 3.58 [02:32:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.72, 3.14, 3.39 [02:51:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 4.16, 3.65 [02:59:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.86, 3.84, 3.74 [03:01:01] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.98, 4.85, 4.11 [03:10:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.68, 3.76, 3.93 [03:30:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.98, 3.05, 3.40 [03:39:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.24, 3.47, 3.39 [03:41:35] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.52, 3.42 [03:45:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.47, 3.66, 3.48 [03:47:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.58, 3.48 [03:51:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 3.79, 3.58 [03:53:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.11, 3.60, 3.54 [03:56:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.66, 3.97, 3.70 [03:58:50] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.02, 3.54, 3.57 [04:01:31] PROBLEM - wiki.ct777.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ct777.cf' expires in 15 day(s) (Sat 27 Nov 2021 03:53:12 GMT +0000). [04:04:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.55, 3.12, 3.40 [04:12:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.90, 3.52, 3.41 [04:16:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.96, 3.31, 3.36 [04:22:46] PROBLEM - wmworld.sktz.live - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wmworld.sktz.live' expires in 15 day(s) (Sat 27 Nov 2021 04:14:05 GMT +0000). [04:23:20] PROBLEM - olwest.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'olwest.org' expires in 15 day(s) (Sat 27 Nov 2021 04:18:59 GMT +0000). [04:24:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.11, 3.86, 3.56 [04:26:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.06, 3.40, 3.41 [04:29:29] PROBLEM - you.r-fit.cc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'you.r-fit.cc' expires in 15 day(s) (Sat 27 Nov 2021 04:20:57 GMT +0000). [04:32:45] PROBLEM - www.rangpurpedia.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'rangpurpedia.xyz' expires in 15 day(s) (Sat 27 Nov 2021 04:29:15 GMT +0000). [04:32:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.55, 3.00, 3.24 [04:37:30] PROBLEM - rangpurpedia.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'rangpurpedia.xyz' expires in 15 day(s) (Sat 27 Nov 2021 04:29:15 GMT +0000). [04:51:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.67, 3.36 [04:53:35] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.32, 3.29, 3.26 [04:59:03] PROBLEM - cp15 Current Load on cp15 is WARNING: WARNING - load average: 1.73, 1.64, 0.95 [05:01:02] RECOVERY - cp15 Current Load on cp15 is OK: OK - load average: 0.50, 1.21, 0.88 [05:32:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.46, 3.19 [05:34:50] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.56, 3.19, 3.13 [05:40:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.29, 3.15 [05:44:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.06, 2.81, 3.01 [05:55:07] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48924 MB (10% inode=98%); [06:04:46] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.20, 5.94, 3.76 [06:05:55] PROBLEM - db12 Current Load on db12 is WARNING: WARNING - load average: 6.86, 5.32, 3.37 [06:06:46] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.64, 6.49, 4.23 [06:06:52] [02mw-config] 07AgentIsai opened pull request 03#4216: Add wgCommentStreamsEnableSearch to ManageWikiSettings - 13https://git.io/JXS6e [06:07:53] miraheze/mw-config - AgentIsai the build passed. [06:07:55] RECOVERY - db12 Current Load on db12 is OK: OK - load average: 6.59, 5.72, 3.76 [06:08:05] CosmicAlpha, around to review PR? ^ [06:08:46] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.42, 7.70, 4.96 [06:17:07] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49119 MB (11% inode=98%); [06:20:46] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 5.42, 7.50, 6.70 [06:22:46] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.38, 7.71, 6.86 [06:25:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.32, 2.87 [06:27:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.14, 3.12, 2.84 [06:28:46] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.49, 7.33, 6.98 [06:30:46] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.30, 6.65, 6.78 [06:31:07] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48637 MB (10% inode=98%); [06:32:26] Agent: replied. [06:32:34] seen [06:34:33] [02mw-config] 07AgentIsai synchronize pull request 03#4216: Add wgCommentStreamsEnableSearch to ManageWikiSettings - 13https://git.io/JXS6e [06:35:34] miraheze/mw-config - AgentIsai the build passed. [06:35:47] CosmicAlpha: ^ [06:36:15] [02mw-config] 07Universal-Omega reviewed pull request 03#4216 commit - 13https://git.io/JXSyM [06:37:06] Agent: ^ very minor and non-essential, I can merge now if you want but that'd be good to do I think. [06:38:57] CosmicAlpha: pardon me but my degree in English does is not serving me well here, where do I lack punctuation? [06:39:16] oh wow, "does is not" [06:39:37] Agent: the help message, at the end could be a `?` [06:39:52] oh [06:41:02] [02mw-config] 07AgentIsai synchronize pull request 03#4216: Add wgCommentStreamsEnableSearch to ManageWikiSettings - 13https://git.io/JXS6e [06:41:09] I didn't request changes there though because it's not essential more of a comment. But thanks for fixing. [06:41:14] np [06:41:40] my original intent was for it to be a statement, not a question but if you see it fit to add a question mark then sure :) [06:42:00] [02mw-config] 07Universal-Omega closed pull request 03#4216: Add wgCommentStreamsEnableSearch to ManageWikiSettings - 13https://git.io/JXS6e [06:42:01] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JXS9B [06:42:03] [02miraheze/mw-config] 07AgentIsai 0369fd66d - Add wgCommentStreamsEnableSearch to ManageWikiSettings (#4216) [06:42:07] miraheze/mw-config - AgentIsai the build passed. [06:42:36] Agent: typically for check boxes, which are boolean true or false, then it is a question I think. [06:43:03] miraheze/mw-config - Universal-Omega the build passed. [06:43:21] To be fair alot of the help messages could be improved grammatically though. Even some of the ones I added. [06:43:25] The more you know [06:43:28] thanks though [06:43:37] thanks for your patience too lol [06:44:19] [02puppet] 07Reception123 closed pull request 03#2120: deploy-mediawiki: use `--save-gitinfo` for rebuildVersionCache - 13https://git.io/JX6Oi [06:44:21] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JXSHU [06:44:22] [02miraheze/puppet] 07Universal-Omega 034b6954a - deploy-mediawiki: use `--save-gitinfo` for rebuildVersionCache (#2120) [06:44:28] Agent: Of course! [06:44:39] Thanks Reception123! [06:47:07] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49527 MB (11% inode=98%); [06:52:35] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JXS7Z [06:52:37] [02miraheze/ssl] 07Reception123 03404bb87 - renew SSL certs [06:54:15] !log [@test3] starting deploy of {'config': True} to skip [06:54:16] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [06:54:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:54:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:09:56] PROBLEM - cp14 Puppet on cp14 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:11:20] Reception123: ^ [07:11:28] since you were working with SSL certs [07:11:30] !log [@mw11] starting deploy of {'config': True} to all [07:11:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:11:39] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 8s [07:11:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:12:09] PROBLEM - cp15 Puppet on cp15 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:12:09] PROBLEM - mwtask1 Puppet on mwtask1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:14:12] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:15:03] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:18:57] Ergh will fix that [07:22:02] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:22:39] PROBLEM - mw13 Puppet on mw13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:24:22] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:24:23] PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:25:51] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.74, 7.27, 6.66 [07:27:07] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:27:49] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.59, 7.82, 6.92 [07:29:43] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:29:46] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.61, 7.44, 6.90 [07:31:12] PROBLEM - cp13 Puppet on cp13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:37:55] RECOVERY - cp14 Puppet on cp14 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:40:08] RECOVERY - cp15 Puppet on cp15 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:41:31] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.62, 7.53, 7.10 [07:42:08] RECOVERY - mwtask1 Puppet on mwtask1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:42:11] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:43:03] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:43:29] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.97, 7.06, 6.97 [07:47:24] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.40, 6.60, 6.79 [07:50:02] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [07:50:38] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:51:16] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.91, 6.73, 6.80 [07:52:21] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [07:53:14] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.38, 6.56, 6.73 [07:54:23] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:55:07] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:57:44] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [07:59:11] RECOVERY - cp13 Puppet on cp13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:05:07] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48941 MB (10% inode=98%); [08:08:46] RECOVERY - duepedia.uk.to - LetsEncrypt on sslhost is OK: OK - Certificate 'duepedia.uk.to' will expire on Wed 09 Feb 2022 05:51:34 GMT +0000. [08:11:33] RECOVERY - staratlaswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'staratlaswiki.org' will expire on Wed 09 Feb 2022 05:50:28 GMT +0000. [08:15:07] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49268 MB (11% inode=98%); [09:31:07] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48943 MB (10% inode=98%); [11:23:07] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49233 MB (11% inode=98%); [12:07:07] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48947 MB (10% inode=98%); [12:07:43] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JXHGD [12:07:44] [02miraheze/CreateWiki] 07translatewiki 0365ad601 - Localisation updates from https://translatewiki.net. [12:07:45] [url] Main page - translatewiki.net | translatewiki.net [12:07:46] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JXHGy [12:07:47] [02miraheze/ManageWiki] 07translatewiki 03ac06808 - Localisation updates from https://translatewiki.net. [12:07:48] [url] Main page - translatewiki.net | translatewiki.net [12:07:49] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±7] 13https://git.io/JXHG9 [12:07:50] [02miraheze/MirahezeMagic] 07translatewiki 03e24a8ce - Localisation updates from https://translatewiki.net. [12:07:51] [url] Main page - translatewiki.net | translatewiki.net [12:10:20] miraheze/MirahezeMagic - translatewiki the build passed. [12:17:30] miraheze/ManageWiki - translatewiki the build passed. [12:18:25] miraheze/CreateWiki - translatewiki the build passed. [14:11:07] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49133 MB (11% inode=98%); [14:39:07] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48953 MB (10% inode=98%); [15:17:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.05, 3.41, 3.06 [15:27:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.34, 3.16, 3.16 [16:38:24] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.24, 3.95, 3.46 [16:40:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.88, 3.65, 3.41 [16:42:14] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.75, 3.47 [16:44:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.64, 3.46 [16:49:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.91, 3.26, 3.37 [16:55:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.90, 3.55, 3.45 [16:59:07] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49070 MB (11% inode=98%); [17:05:14] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.79, 3.21, 3.38 [17:21:07] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48944 MB (10% inode=98%); [17:25:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.70, 3.87, 3.38 [17:27:35] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.90, 3.46 [17:29:30] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.84, 3.48 [17:30:19] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.63, 7.57, 5.52 [17:32:19] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.28, 6.58, 5.40 [17:33:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.75, 3.52 [17:35:03] !sre [17:35:15] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 3.35, 3.40 [17:51:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.49, 3.34 [17:55:24] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.78, 3.25, 3.28 [18:08:51] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.04, 3.86, 3.46 [18:12:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.90, 3.26, 3.31 [18:19:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.81, 4.01, 3.65 [18:27:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.81, 3.64, 3.71 [18:36:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.25, 3.08, 3.40 [18:41:30] [02puppet] 07Universal-Omega opened pull request 03#2121: mwscript: use argparse - 13https://git.io/JXQhx [18:42:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.34, 4.14, 3.69 [18:43:57] [02puppet] 07Universal-Omega synchronize pull request 03#2121: mwscript: use argparse - 13https://git.io/JXQhx [18:44:18] [02puppet] 07Universal-Omega edited pull request 03#2121: mwscript: rewrite to use argparse - 13https://git.io/JXQhx [18:58:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.13, 3.49, 3.78 [19:04:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.45, 3.67 [19:06:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.60, 3.14, 3.53 [19:08:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.37, 2.91, 3.40 [19:20:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.48, 3.30 [19:22:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.12, 2.91, 3.11 [19:51:07] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 48992 MB (11% inode=98%); [20:01:07] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48946 MB (10% inode=98%); [20:26:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.86, 3.43, 3.07 [20:28:28] [02puppet] 07Universal-Omega synchronize pull request 03#2121: mwscript: rewrite to use argparse - 13https://git.io/JXQhx [20:38:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.97, 3.37, 3.29 [20:42:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.53, 4.36, 3.69 [20:46:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.64, 3.65, 3.57 [20:58:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.44, 3.58, 3.49 [21:02:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.32, 3.91, 3.69 [21:24:19] [02puppet] 07Universal-Omega synchronize pull request 03#2121: mwscript: rewrite to use argparse - 13https://git.io/JXQhx [21:32:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.40, 3.13, 3.37 [21:49:34] PROBLEM - bacula2 APT on bacula2 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [21:50:10] PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [21:51:10] PROBLEM - mw13 APT on mw13 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [21:51:12] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [21:51:20] PROBLEM - db11 APT on db11 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [21:51:35] PROBLEM - mem2 APT on mem2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [21:51:35] PROBLEM - gluster3 APT on gluster3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [21:52:03] PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [21:52:14] PROBLEM - db12 APT on db12 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [21:53:22] PROBLEM - mon2 APT on mon2 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (1 critical updates). [21:53:56] PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [21:54:05] PROBLEM - puppet3 APT on puppet3 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (2 critical updates). [21:54:09] PROBLEM - db13 APT on db13 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [21:54:38] PROBLEM - cp12 APT on cp12 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [21:56:21] PROBLEM - gluster4 APT on gluster4 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [21:57:21] PROBLEM - cloud5 APT on cloud5 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [21:58:10] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [21:59:43] PROBLEM - mw12 APT on mw12 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [22:00:43] PROBLEM - mail2 APT on mail2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [22:01:35] PROBLEM - cp13 APT on cp13 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [22:03:41] PROBLEM - graylog2 APT on graylog2 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (1 critical updates). [22:04:23] PROBLEM - phab2 APT on phab2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [22:07:35] PROBLEM - mem1 APT on mem1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [22:09:16] PROBLEM - ldap2 APT on ldap2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [22:09:18] PROBLEM - cp15 APT on cp15 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [22:10:34] PROBLEM - cp14 APT on cp14 is CRITICAL: APT CRITICAL: 15 packages available for upgrade (1 critical updates). [22:11:37] [02mw-config] 07CrystalClearGal commented on pull request 03#4216: Add wgCommentStreamsEnableSearch to ManageWikiSettings - 13https://git.io/JX7ix [22:12:57] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [22:13:20] [02mw-config] 07Universal-Omega commented on pull request 03#4216: Add wgCommentStreamsEnableSearch to ManageWikiSettings - 13https://git.io/JX7PL [22:14:53] PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [22:15:16] paladox: can you run upgrades [22:15:57] PROBLEM - cloud4 APT on cloud4 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [22:16:48] RECOVERY - cp12 APT on cp12 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:16:51] !log all hosts: sudo unattended-upgrade -d [22:16:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:17:12] RECOVERY - ns2 APT on ns2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:17:15] RECOVERY - ldap2 APT on ldap2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:17:18] RECOVERY - cp15 APT on cp15 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:17:20] RECOVERY - db11 APT on db11 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:17:22] RECOVERY - cloud5 APT on cloud5 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:17:24] RECOVERY - cp13 APT on cp13 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:17:31] RECOVERY - mon2 APT on mon2 is OK: APT OK: 9 packages available for upgrade (0 critical updates). [22:17:34] RECOVERY - bacula2 APT on bacula2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:17:35] RECOVERY - mem1 APT on mem1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:17:35] RECOVERY - mem2 APT on mem2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:17:36] RECOVERY - gluster3 APT on gluster3 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:17:45] RECOVERY - graylog2 APT on graylog2 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [22:17:57] RECOVERY - mw10 APT on mw10 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:17:57] RECOVERY - cloud4 APT on cloud4 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:18:03] RECOVERY - mw9 APT on mw9 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:18:04] RECOVERY - puppet3 APT on puppet3 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [22:18:10] RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:18:10] RECOVERY - ns1 APT on ns1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:18:11] RECOVERY - db13 APT on db13 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:18:14] RECOVERY - db12 APT on db12 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:18:21] RECOVERY - gluster4 APT on gluster4 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:18:23] RECOVERY - phab2 APT on phab2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:18:37] RECOVERY - cp14 APT on cp14 is OK: APT OK: 14 packages available for upgrade (0 critical updates). [22:18:43] RECOVERY - mail2 APT on mail2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:18:52] RECOVERY - mw11 APT on mw11 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:18:59] RECOVERY - mw8 APT on mw8 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:19:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.06, 4.25, 3.55 [22:19:10] RECOVERY - mw13 APT on mw13 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:19:30] RECOVERY - mw12 APT on mw12 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:24:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.57, 3.83, 3.58 [22:28:26] !log [universalomega@mw11] starting deploy of {'config': True} to all [22:28:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:28:33] !log [universalomega@mw11] finished deploy of {'config': True} to all - SUCCESS in 6s [22:28:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:32:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.60, 3.04, 3.32 [22:33:09] !log [universalomega@mw11] starting deploy of {'pull': 'config', 'config': True} to all [22:33:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:33:16] !log [universalomega@mw11] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 6s [22:33:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:46:32] [02puppet] 07Universal-Omega opened pull request 03#2122: fix deploy-mediawiki - 13https://git.io/JX7D2 [22:47:44] RhinosF1: ^ do you think that could be a cause of what I described there? [22:49:28] CosmicAlpha: very possibly [22:49:34] paladox: can you ^ [22:49:42] CosmicAlpha: this is why I want tests [22:50:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JX7yJ [22:50:01] [02miraheze/puppet] 07Universal-Omega 039c606ab - fix deploy-mediawiki (#2122) [22:50:03] [02puppet] 07paladox closed pull request 03#2122: fix deploy-mediawiki - 13https://git.io/JX7D2 [22:53:17] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.22, 22.93, 17.87 [22:54:19] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.04, 6.52, 5.15 [22:54:29] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.04, 3.42, 2.15 [22:55:15] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.26, 24.29, 18.96 [22:56:29] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.71, 3.42, 2.30 [22:58:19] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 15.57, 9.05, 6.33 [22:58:29] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.12, 3.55, 2.48 [22:59:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.11, 6.96, 5.56 [23:00:01] !log [@mw11] starting deploy of {'l10nupdate': True} to all [23:00:02] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:29] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.86, 3.63, 2.64 [23:00:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:01:58] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.91, 6.76, 4.45 [23:02:29] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.10, 3.90, 2.86 [23:03:19] PROBLEM - mw11 APT on mw11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:03:37] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:37] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.45, 8.25, 6.34 [23:03:51] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 12.59, 10.07, 6.88 [23:03:53] !log [universalomega@mw11] starting deploy of {'config': True} to all [23:04:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:04:18] !log [universalomega@mw11] DEPLOY ABORTED: Canary check failed for mw8 [23:04:29] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.52, 3.76, 2.94 [23:04:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:05:05] ugh... mw8 is working... or at least when proxying. [23:05:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [23:05:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 54.38.211.199/cpweb [23:05:17] RECOVERY - mw11 APT on mw11 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [23:05:20] guess not though... [23:05:35] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20011 bytes in 0.305 second response time [23:06:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:06:30] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 1.94, 3.20, 2.84 [23:06:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.16, 7.60, 6.20 [23:06:36] PROBLEM - mw12 APT on mw12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:06:45] !log [universalomega@mw11] starting deploy of {'config': True} to all [23:06:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:06:55] !log [universalomega@mw11] finished deploy of {'config': True} to all - SUCCESS in 10s [23:07:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:07:08] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:07:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:09:33] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.31, 7.98, 6.80 [23:09:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.39, 7.24, 6.74 [23:10:30] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.14, 6.53, 6.12 [23:10:44] RECOVERY - mw12 APT on mw12 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [23:11:40] !log sudo -u www-data rm -rf /srv/mediawiki/config/config && sudo -u www-data rm -rf /srv/mediawiki/landing/landing && sudo -u www-data rm -rf /srv/mediawiki/ErrorPages/ErrorPages && sudo -u www-data rm -rf /srv/mediawiki/w/w on mw* / mwtask1 [23:11:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:11:51] RhinosF1: It seemed to work now. [23:11:58] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.46, 5.75, 5.39 [23:12:58] CosmicAlpha: ok [23:13:21] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.65, 6.65, 6.52 [23:13:32] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.07, 6.19, 6.44 [23:15:18] !log [universalomega@mw11] starting deploy of {'landing': True} to all [23:15:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:15:45] !log [universalomega@mw11] finished deploy of {'landing': True} to all - SUCCESS in 26s [23:15:58] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.32, 5.25, 5.21 [23:15:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:17:58] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.71, 4.57, 4.96 [23:18:51] !log set email for a user per sre@ request following confirmation content known of private wiki only they can access [23:18:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:19:06] PROBLEM - mw12 APT on mw12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:21:04] RECOVERY - mw12 APT on mw12 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [23:21:58] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 11.01, 7.26, 5.87 [23:22:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:22:46] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 51.222.25.132/cpweb [23:23:10] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.85, 7.27, 6.05 [23:23:27] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.87, 7.96, 7.08 [23:24:43] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:25:10] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.48, 6.69, 5.99 [23:25:26] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.57, 6.80, 6.76 [23:27:36] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:27:58] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.03, 5.91, 5.87 [23:29:58] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.09, 6.06, 5.90 [23:30:19] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.35, 7.33, 7.94 [23:30:35] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 1833s [23:30:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:31:58] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.20, 5.75, 5.81 [23:35:58] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.93, 6.58, 6.07 [23:38:19] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.15, 5.51, 6.66 [23:39:58] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.91, 5.70, 5.82 [23:41:58] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.45, 6.51, 6.11 [23:42:14]