[00:07:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.42, 3.06 [00:09:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.93, 2.93, 2.93 [00:10:45] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.26, 4.86, 3.50 [00:11:51] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 33.12, 24.77, 19.24 [00:16:45] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.01, 5.69, 4.44 [00:18:46] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 11.86, 8.06, 5.46 [00:20:08] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb [00:20:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [00:22:14] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:24:07] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:34:45] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.61, 5.16, 5.83 [00:37:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.35, 3.08 [00:37:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 12.45, 18.82, 23.34 [00:38:45] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.96, 3.62, 5.05 [00:39:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.30, 3.00, 2.99 [00:43:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.58, 3.26 [00:45:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.06, 3.37, 3.22 [00:45:34] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.43, 15.69, 19.79 [00:51:30] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.52, 4.09, 3.56 [00:57:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.33, 3.57, 3.55 [01:03:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.87, 3.06, 3.38 [01:11:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.33, 3.35 [01:13:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.27, 2.97, 3.22 [01:25:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.48, 3.70, 3.46 [01:25:55] [02puppet] 07Universal-Omega edited pull request 03#2142: mediawiki: use deploy repos for mathoid and 3d2png - 13https://git.io/JMQhM [01:29:17] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.49, 3.13, 3.29 [01:48:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JM7bq [01:48:27] [02miraheze/mw-config] 07Universal-Omega 03168b98c - AddThis conflicts with Mirage [01:49:30] miraheze/mw-config - Universal-Omega the build passed. [01:55:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JM7Nt [01:55:13] [02miraheze/mw-config] 07Universal-Omega 03a3c71cb - Update GlobalExtensions.php [01:55:15] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [01:55:34] [02mw-config] 07Universal-Omega opened pull request 03#4258: Don't install GlobalPreferences on ldapwikiwiki - 13https://git.io/JM7Nq [01:56:30] miraheze/mw-config - Universal-Omega the build passed. [01:57:34] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JM7Na [01:57:36] [02miraheze/mw-config] 07Universal-Omega 03016ce90 - Update Defines.php [01:57:37] [02mw-config] 07Universal-Omega synchronize pull request 03#4258: Don't install GlobalPreferences on ldapwikiwiki - 13https://git.io/JM7Nq [01:58:36] miraheze/mw-config - Universal-Omega the build passed. [02:12:10] !log [@mw11] starting deploy of {'config': True} to all [02:12:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:12:33] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 22s [02:12:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:13:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.24, 6.72, 4.99 [02:14:17] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.67, 18.61, 16.25 [02:15:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.50, 5.95, 4.92 [02:15:38] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.34, 6.52, 4.75 [02:17:31] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.77, 6.84, 5.35 [02:17:38] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.29, 5.62, 4.63 [02:18:17] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.37, 20.61, 17.58 [02:19:25] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.59, 6.49, 5.40 [02:22:17] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.12, 21.44, 18.66 [02:24:17] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.92, 19.65, 18.33 [02:45:29] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.26, 5.18, 4.37 [02:47:26] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.07, 4.35, 4.16 [02:49:14] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JM5vN [02:49:15] [02miraheze/ManageWiki] 07Universal-Omega 0379489a5 - Allow all content models to be available in ManageWikiNamespaces [02:49:17] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns [02:49:18] [02ManageWiki] 07Universal-Omega opened pull request 03#318: Allow all content models to be available in ManageWikiNamespaces - 13https://git.io/JM5vA [02:50:44] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.42, 2.97 [02:54:42] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.89, 3.96, 3.27 [02:56:13] miraheze/ManageWiki - Universal-Omega the build passed. [02:56:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.64, 3.42, 3.16 [02:58:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.26, 3.26, 3.12 [04:31:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.24, 3.67, 3.15 [04:33:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.66, 3.20 [04:34:57] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.96, 1.58, 1.08 [04:36:57] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.55, 1.61, 1.15 [04:45:55] PROBLEM - wiki.villagecollaborative.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.villagecollaborative.net could not be found [04:46:59] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.69, 3.20, 3.31 [04:50:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.49, 3.39 [04:52:51] RECOVERY - wiki.villagecollaborative.net - reverse DNS on sslhost is OK: SSL OK - wiki.villagecollaborative.net reverse DNS resolves to cp21.miraheze.org - CNAME OK [04:52:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.30, 2.93, 3.19 [05:01:44] PROBLEM - wiki.villagecollaborative.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.villagecollaborative.net could not be found [05:06:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JM5Bq [05:06:26] [02miraheze/mw-config] 07Universal-Omega 03fd5f7ff - Raise password requirements for global groups [05:06:28] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [05:06:29] [02mw-config] 07Universal-Omega opened pull request 03#4259: Raise password requirements for global groups - 13https://git.io/JM5BY [05:07:32] miraheze/mw-config - Universal-Omega the build passed. [05:11:07] PROBLEM - wiki.jeremyboyd.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.jeremyboyd.org' expires in 15 day(s) (Tue 21 Dec 2021 05:07:51 GMT +0000). [05:17:59] PROBLEM - fanon.polandballwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fanon.polandballwiki.com' expires in 15 day(s) (Tue 21 Dec 2021 05:09:59 GMT +0000). [05:34:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JM5zc [05:34:58] [02miraheze/mw-config] 07Universal-Omega 03bd38b02 - Update LocalSettings.php [05:34:59] [02mw-config] 07Universal-Omega synchronize pull request 03#4259: Raise password requirements for global groups - 13https://git.io/JM5BY [05:35:58] miraheze/mw-config - Universal-Omega the build passed. [05:36:23] [02mw-config] 07Universal-Omega edited pull request 03#4259: Raise password requirements for global groups - 13https://git.io/JM5BY [05:37:03] [02mw-config] 07Universal-Omega edited pull request 03#4259: Raise password requirements for global groups - 13https://git.io/JM5BY [05:37:29] [02mw-config] 07Universal-Omega edited pull request 03#4259: Raise password requirements for global groups - 13https://git.io/JM5BY [06:13:52] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49204 MB (11% inode=98%); [06:17:51] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 46806 MB (10% inode=98%); [06:24:18] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.07, 3.76, 2.85 [06:28:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.98, 3.18 [06:32:15] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.04, 3.28, 3.06 [06:46:38] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.96, 7.46, 6.51 [06:48:34] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 8.00, 7.59, 6.67 [06:52:26] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.31, 7.62, 6.87 [06:54:22] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.46, 7.28, 6.84 [06:58:14] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.54, 7.82, 7.13 [07:00:10] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.22, 7.37, 7.06 [07:02:06] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.06, 7.63, 7.18 [07:04:02] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.51, 7.34, 7.12 [07:09:50] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.51, 6.16, 6.66 [07:15:40] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.28, 7.08, 6.88 [07:17:36] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.09, 6.61, 6.73 [07:33:51] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 58469 MB (13% inode=98%); [09:07:35] [02mw-config] 07Universal-Omega assigned pull request 03#4247: 1.37 upgrade sitenotice - 13https://git.io/JMDu9 [09:24:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JMdsL [09:24:50] [02miraheze/mw-config] 07Universal-Omega 03d5c53f8 - Update SQL for CommentStreams for 1.37 [09:24:51] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-5 - 13https://git.io/vbvb3 [09:24:53] [02mw-config] 07Universal-Omega opened pull request 03#4260: Update SQL for CommentStreams for 1.37 - 13https://git.io/JMdst [09:25:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-1/±0] 13https://git.io/JMdsZ [09:25:17] [02miraheze/mw-config] 07Universal-Omega 03372609f - Delete commentstreams_temp.sql [09:25:18] [02mw-config] 07Universal-Omega synchronize pull request 03#4260: Update SQL for CommentStreams for 1.37 - 13https://git.io/JMdst [09:25:29] [02mw-config] 07Universal-Omega assigned pull request 03#4260: Update SQL for CommentStreams for 1.37 - 13https://git.io/JMdst [09:25:54] miraheze/mw-config - Universal-Omega the build passed. [09:26:19] miraheze/mw-config - Universal-Omega the build passed. [09:27:00] [02mw-config] 07Universal-Omega edited pull request 03#4260: Update SQL for CommentStreams for 1.37 - 13https://git.io/JMdst [09:29:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-1/±0] 13https://git.io/JMdGT [09:29:04] [02miraheze/mw-config] 07Universal-Omega 033c92cf4 - Delete 137updates.sql [09:29:06] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-6 - 13https://git.io/vbvb3 [09:29:28] [02mw-config] 07Universal-Omega opened pull request 03#4261: Drop 137updates.sql - 13https://git.io/JMdGY [09:30:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://git.io/JMdGG [09:30:12] [02miraheze/mw-config] 07Universal-Omega 031f1c50e - Update LocalSettings.php [09:30:14] [02mw-config] 07Universal-Omega synchronize pull request 03#4261: Drop 137updates.sql - 13https://git.io/JMdGY [09:30:27] miraheze/mw-config - Universal-Omega the build passed. [09:30:41] [02mw-config] 07Universal-Omega assigned pull request 03#4261: Drop 137updates.sql - 13https://git.io/JMdGY [09:31:13] miraheze/mw-config - Universal-Omega the build passed. [10:05:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.54, 7.69, 5.42 [10:07:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.36, 6.92, 5.40 [10:09:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.63, 6.27, 5.33 [10:12:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.90, 6.47, 4.75 [10:14:42] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.06, 6.22, 4.86 [10:21:30] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 14.27, 10.82, 6.85 [10:21:40] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [10:22:00] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 15.57, 9.17, 6.42 [10:22:04] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 29.05, 18.00, 9.57 [10:22:15] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [10:22:29] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 32.92, 29.48, 22.28 [10:22:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:22:39] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.27, 11.89, 7.78 [10:22:44] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.37, 12.17, 8.14 [10:23:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.04, 7.91, 6.31 [10:23:39] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:23:59] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.98, 7.83, 6.28 [10:24:14] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:24:25] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/mnt/mediawiki-static] [10:25:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.35, 8.78, 6.86 [10:26:26] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 14.88, 21.45, 20.57 [10:26:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.09, 7.89, 7.05 [10:26:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 3.37, 7.70, 7.24 [10:27:23] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.05, 6.84, 6.41 [10:27:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:27:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.49, 7.23, 6.50 [10:27:55] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.40, 7.54, 6.47 [10:28:26] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.43, 19.40, 19.91 [10:28:28] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.01, 6.76, 6.75 [10:29:20] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.28, 6.46, 6.33 [10:29:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.56, 8.71, 7.17 [10:29:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.52, 7.29, 6.52 [10:31:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.95, 7.81, 6.99 [10:32:00] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 3.58, 6.73, 7.44 [10:32:42] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 2.82, 5.28, 6.40 [10:33:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 2.96, 6.04, 6.44 [10:33:53] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.15, 5.32, 5.91 [10:34:00] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 2.26, 5.14, 6.77 [10:50:19] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:11:25] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.07, 9.57, 6.56 [11:11:52] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.96, 7.73, 5.71 [11:13:23] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.09, 7.40, 6.12 [11:13:50] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.02, 6.02, 5.32 [11:15:21] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 12.26, 9.81, 7.16 [11:16:44] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.36, 7.41, 5.15 [11:17:45] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.47, 7.24, 6.11 [11:18:00] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.26, 6.88, 5.33 [11:20:00] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.66, 8.50, 6.10 [11:21:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.58, 6.89, 5.50 [11:21:19] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.59, 7.67, 7.09 [11:23:14] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.69, 9.31, 6.56 [11:23:17] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 17.82, 11.90, 8.72 [11:23:41] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 16.23, 9.61, 7.24 [11:23:43] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [11:24:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:25:40] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:27:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.47, 7.86, 6.60 [11:28:00] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.87, 7.78, 6.94 [11:28:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 3.60, 7.27, 7.09 [11:29:06] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.39, 6.69, 6.30 [11:29:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:29:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.43, 7.07, 6.87 [11:31:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 3.87, 7.15, 7.77 [11:31:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.40, 5.79, 6.42 [11:32:00] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.42, 6.26, 6.54 [11:32:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.40, 7.07, 6.24 [11:33:08] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.84, 7.80, 7.92 [11:34:01] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.99, 7.69, 6.53 [11:34:43] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.81, 7.16, 7.04 [11:35:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 19.97, 10.32, 7.58 [11:35:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:35:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 16.51, 10.41, 8.07 [11:36:01] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.28, 8.67, 7.53 [11:36:22] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [11:36:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.37, 6.95, 6.99 [11:37:56] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.37, 5.84, 6.09 [11:38:00] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 3.65, 6.80, 6.98 [11:38:22] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:38:42] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.48, 5.63, 6.50 [11:38:56] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 3.32, 6.98, 6.90 [11:39:03] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 3.14, 6.73, 7.66 [11:40:00] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 2.58, 5.33, 6.42 [11:40:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:40:53] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.82, 5.84, 6.48 [11:41:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.16, 7.04, 7.54 [11:42:59] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.38, 4.84, 6.66 [11:45:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.21, 4.97, 6.57 [13:16:24] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [13:30:20] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp20.miraheze.org - CNAME OK [13:57:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.97, 1.59, 1.21 [13:59:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.11, 1.45, 1.21 [15:57:48] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.01, 2.92, 1.84 [15:59:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.76, 1.37, 1.03 [16:01:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.62, 1.42, 1.09 [16:01:48] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.82, 1.72, 1.59 [16:03:48] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.44, 1.46, 1.50 [16:08:45] https://phabricator.miraheze.org/p/Jackthomson/ advertiser? [16:08:46] [url] ♟ Jackthomson | phabricator.miraheze.org [16:08:46] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.17, 3.84, 3.30 [16:08:57] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.75, 1.80, 1.12 [16:10:57] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.55, 1.33, 1.03 [16:12:06] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.47, 7.24, 5.79 [16:12:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.76, 3.39 [16:14:04] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.62, 6.76, 5.81 [16:15:48] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.03, 1.71, 1.53 [16:16:43] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.78, 4.02, 3.56 [16:17:48] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.25, 1.57, 1.50 [16:17:59] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.39, 6.85, 6.06 [16:19:58] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.66, 6.78, 6.12 [16:20:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.09, 6.40, 5.29 [16:21:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.86, 7.96, 5.93 [16:21:04] RECOVERY - mon2 Puppet on mon2 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [16:21:31] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.84, 7.00, 5.83 [16:22:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.32, 6.34, 5.41 [16:22:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 4.00, 3.71 [16:23:00] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.20, 7.19, 5.88 [16:23:26] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.70, 6.74, 5.87 [16:23:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.26, 7.15, 6.45 [16:24:57] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.85, 6.40, 5.76 [16:26:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.71, 1.62, 1.32 [16:26:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.69, 3.61 [16:27:52] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.74, 6.34, 6.31 [16:27:57] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.02, 1.39, 1.27 [16:28:39] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.89, 3.53, 3.57 [16:30:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.93, 2.95, 3.35 [16:34:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.05, 3.58, 3.55 [16:36:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.57, 3.11, 3.38 [16:40:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.76, 3.88, 3.61 [16:42:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.87, 3.56, 3.53 [16:46:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.86, 3.64 [16:47:48] Reception123: ^^ [16:48:30] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.91, 3.70 [16:50:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.32, 4.21, 3.84 [16:51:13] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:54:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.86, 3.88, 3.83 [16:58:47] [02mw-config] 07Universal-Omega edited pull request 03#4257: T8323: remove the Liberty skin - 13https://git.io/JM7VN [17:01:13] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [17:02:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.36, 3.52 [17:03:09] !log [universalomega@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/liberty.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgLibertyMainColor --sourcelist=false --remove (START) [17:03:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:04:24] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.10, 3.15, 3.42 [17:04:25] !log [universalomega@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/liberty.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgLibertyMainColor --sourcelist=false --remove (END - exit=0) [17:04:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:05:02] !log [universalomega@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/liberty.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgLibertySecondColor --sourcelist=false --remove (START) [17:05:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:06:01] !log [universalomega@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/liberty.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --wgsetting=wgLibertySecondColor --sourcelist=false --remove (END - exit=0) [17:06:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:06:23] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.68, 2.95, 3.31 [17:06:38] !log [universalomega@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/liberty.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --disable liberty (START) [17:06:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:07:42] !log [universalomega@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/liberty.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --disable liberty (END - exit=0) [17:07:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:08:00] [02mw-config] 07Universal-Omega closed pull request 03#4257: T8323: remove the Liberty skin - 13https://git.io/JM7VN [17:08:01] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [17:08:03] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [17:08:04] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JMFMi [17:08:06] [02miraheze/mw-config] 07Universal-Omega 038e257f9 - T8323: remove the Liberty skin (#4257) [17:08:17] [02mw-config] 07Universal-Omega edited pull request 03#4247: 1.37 upgrade sitenotice - 13https://git.io/JMDu9 [17:08:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JMFMM [17:08:37] [02miraheze/mw-config] 07Reception123 03d1f8616 - 1.37 upgrade sitenotice (#4247) [17:08:39] [02mw-config] 07Universal-Omega closed pull request 03#4247: 1.37 upgrade sitenotice - 13https://git.io/JMDu9 [17:08:40] [02mw-config] 07Universal-Omega deleted branch 03Reception123-patch-3 - 13https://git.io/vbvb3 [17:08:42] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Reception123-patch-3 [17:09:02] miraheze/mw-config - Universal-Omega the build passed. [17:09:05] !log [universalomega@mw11] starting deploy of {'pull': 'config', 'config': True} to all [17:09:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:09:23] !log [universalomega@mw11] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 18s [17:09:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:09:39] miraheze/mw-config - Universal-Omega the build passed. [17:09:40] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 157.09 ms [17:10:55] !log [universalomega@test3] starting deploy of {'pull': 'config', 'config': True} to skip [17:11:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:11:04] !log [universalomega@test3] finished deploy of {'pull': 'config', 'config': True} to skip - SUCCESS in 9s [17:11:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:11:23] !log [universalomega@test3] starting deploy of {'l10n': True} to skip [17:11:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:11:53] !log [universalomega@mw11] starting deploy of {'l10n': True} to all [17:12:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:14:06] [02mw-config] 07Universal-Omega closed pull request 03#4239: Lingo conflicts with Newsletter in 1.37 (see T8166) - 13https://git.io/JMzOX [17:14:07] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Reception123-patch-1 [17:14:09] [02mw-config] 07Universal-Omega deleted branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [17:14:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JMFDR [17:14:12] [02miraheze/mw-config] 07Reception123 035e8424c - Lingo conflicts with Newsletter in 1.37 (see T8166) (#4239) [17:14:48] [02mw-config] 07Universal-Omega closed pull request 03#4244: Add a new canned response - 13https://git.io/JMXBG [17:14:49] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JMFDw [17:14:51] [02miraheze/mw-config] 07magogre 03a019979 - Add a new canned response (#4244) [17:15:20] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JMFDi [17:15:21] miraheze/mw-config - Universal-Omega the build passed. [17:15:22] [02miraheze/mw-config] 07github-actions 03a06ece1 - CI: lint code to MediaWiki standards [17:15:59] miraheze/mw-config - Universal-Omega the build passed. [17:16:26] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.57, 19.89, 16.95 [17:16:27] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.74, 6.83, 5.50 [17:16:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JMFDA [17:16:40] [02miraheze/mw-config] 07Universal-Omega 03a37f0a3 - fix [17:17:37] miraheze/mw-config - Universal-Omega the build passed. [17:18:05] [02mw-config] 07JohnFLewis commented on pull request 03#4249: LS: blacklist .*betawiki in prep for future usage - 13https://git.io/JMFyk [17:18:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.19, 3.17 [17:18:25] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.03, 7.60, 5.95 [17:18:57] !log [universalomega@test3] finished deploy of {'l10n': True} to skip - SUCCESS in 454s [17:19:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:19:43] [02mw-config] 07RhinosF1 commented on pull request 03#4249: LS: blacklist .*betawiki in prep for future usage - 13https://git.io/JMFyW [17:20:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.59, 3.68, 3.35 [17:20:26] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.23, 20.32, 17.90 [17:21:29] [02mw-config] 07JohnFLewis commented on pull request 03#4249: LS: blacklist .*betawiki in prep for future usage - 13https://git.io/JMFy6 [17:22:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.68, 3.39 [17:22:30] [02mw-config] 07RhinosF1 commented on pull request 03#4249: LS: blacklist .*betawiki in prep for future usage - 13https://git.io/JMFyF [17:24:15] [02mw-config] 07JohnFLewis commented on pull request 03#4249: LS: blacklist .*betawiki in prep for future usage - 13https://git.io/JMFyh [17:24:15] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.29, 3.27, 3.28 [17:24:19] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.78, 7.25, 6.53 [17:25:26] [02mw-config] 07Universal-Omega commented on pull request 03#4249: LS: blacklist .*betawiki in prep for future usage - 13https://git.io/JMFSm [17:26:17] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.52, 19.36, 16.70 [17:26:19] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.84, 7.51, 6.67 [17:26:28] [02mw-config] 07RhinosF1 commented on pull request 03#4249: LS: blacklist .*betawiki in prep for future usage - 13https://git.io/JMFS8 [17:27:41] [02mw-config] 07Reception123 commented on pull request 03#4249: LS: blacklist .*betawiki in prep for future usage - 13https://git.io/JMFS6 [17:28:22] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.04, 6.81, 5.30 [17:28:37] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 64.17, 34.86, 23.93 [17:28:45] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.42, 7.86, 6.43 [17:28:45] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 13.30, 9.68, 6.81 [17:28:50] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:29:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:30:00] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 7.71, 9.17, 7.04 [17:30:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.30, 19.97, 17.54 [17:30:22] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.92, 7.19, 5.65 [17:30:28] !log [universalomega@mw11] finished deploy of {'l10n': True} to all - SUCCESS in 1114s [17:30:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:30:43] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.85, 6.61, 6.14 [17:30:50] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 0.032 second response time [17:32:00] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.40, 7.97, 6.87 [17:32:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.11, 3.41, 3.33 [17:32:13] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 2.77, 7.80, 7.63 [17:32:21] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.04, 6.37, 5.54 [17:32:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.01, 6.81, 6.33 [17:33:19] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±1] 13https://git.io/JMF9D [17:33:21] [02miraheze/mediawiki] 07Universal-Omega 03bc7db05 - Remove Liberty [17:33:22] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [17:33:54] [02mediawiki] 07Universal-Omega opened pull request 03#4384: Remove Liberty - 13https://git.io/JMF9Q [17:34:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.34, 3.38, 3.33 [17:34:23] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JMF9F [17:34:24] [02miraheze/mediawiki] 07Universal-Omega 03f281432 - Update non-rel-extensions [17:34:26] [02mediawiki] 07Universal-Omega synchronize pull request 03#4384: Remove Liberty - 13https://git.io/JMF9Q [17:34:26] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 12.43, 21.88, 21.84 [17:34:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:34:35] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.29, 2.68, 1.81 [17:34:42] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.60, 5.80, 6.02 [17:35:01] [02mediawiki] 07Universal-Omega closed pull request 03#4384: Remove Liberty - 13https://git.io/JMF9Q [17:35:02] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-1/±2] 13https://git.io/JMF9j [17:35:04] [02miraheze/mediawiki] 07Universal-Omega 039ee8a19 - Remove Liberty (#4384) [17:35:05] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [17:35:07] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [17:36:00] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.49, 6.01, 6.31 [17:36:08] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 2.61, 5.32, 6.66 [17:36:34] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.50, 1.89, 1.62 [17:38:26] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.55, 18.40, 20.39 [17:38:34] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.50, 1.42, 1.48 [17:42:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.40, 3.30 [17:42:08] !log [@mw11] starting deploy of {'config': True} to all [17:42:23] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 14s [17:42:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:42:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:44:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.16, 3.23, 3.25 [17:44:31] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 14.52, 9.05, 6.72 [17:44:42] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.50, 7.96, 6.56 [17:46:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.41, 7.52, 6.45 [17:46:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.34, 6.91, 6.35 [17:48:42] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.34, 6.40, 6.23 [17:49:49] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 112.76 ms [17:50:03] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.75, 4.01, 3.52 [17:50:26] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.91, 6.59, 6.30 [17:52:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.72, 3.47 [17:58:00] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.89, 3.36, 3.39 [18:14:50] !log [universalomega@test3] starting deploy of {'world': True, 'extensionlist': True} to skip [18:14:53] [02mw-config] 07RhinosF1 commented on pull request 03#4249: LS: blacklist .*betawiki in prep for future usage - 13https://git.io/JMFbv [18:14:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:14:55]