[00:00:57] RECOVERY - gluster4 Disk Space on gluster4 is OK: DISK OK - free space: / 117635 MB (11% inode=79%); [00:03:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.12, 2.77 [00:05:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.33, 2.94, 2.76 [00:07:17] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.95, 19.39, 17.17 [00:08:57] PROBLEM - gluster4 Disk Space on gluster4 is WARNING: DISK WARNING - free space: / 117628 MB (10% inode=79%); [00:08:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.55, 3.11 [00:09:12] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.74, 21.67, 18.28 [00:10:08] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.38, 4.92, 3.66 [00:10:59] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.31, 4.01, 3.31 [00:12:46] PROBLEM - cp15 Puppet on cp15 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.tulpa.info_private] [00:13:08] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.18, 7.29, 5.27 [00:14:08] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.53, 5.44, 4.24 [00:14:16] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.29, 7.39, 5.54 [00:14:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.62, 3.88, 3.43 [00:15:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.20, 6.43, 5.21 [00:16:13] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.86, 5.95, 5.24 [00:16:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.93, 3.50 [00:18:08] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.94, 4.78, 4.26 [00:18:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.73, 3.48 [00:22:05] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.98, 6.16, 5.44 [00:22:08] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.25, 5.48, 4.69 [00:22:53] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.05, 4.03, 3.65 [00:24:02] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.07, 5.63, 5.33 [00:24:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.58, 3.81, 3.61 [00:28:50] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.33, 3.75, 3.62 [00:32:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.30, 3.35, 3.51 [00:34:08] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.28, 5.83, 5.86 [00:34:31] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JP7Xk [00:34:32] [02miraheze/mw-config] 07paladox 03943e9df - Fix setting wgNoticeProject [00:34:34] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [00:34:35] [02mw-config] 07paladox opened pull request 03#4201: Fix setting wgNoticeProject - 13https://git.io/JP7XI [00:35:42] miraheze/mw-config - paladox the build passed. [00:36:46] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.61, 3.07, 3.36 [00:37:31] [02mw-config] 07paladox closed pull request 03#4201: Fix setting wgNoticeProject - 13https://git.io/JP7XI [00:37:33] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [00:37:34] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [00:40:43] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.32, 3.41 [00:40:44] RECOVERY - cp15 Puppet on cp15 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:42:43] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.77, 3.21, 3.36 [00:44:08] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.83, 3.58, 4.72 [00:48:39] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.00, 3.39, 3.41 [00:50:38] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.59, 4.10, 3.66 [00:52:08] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.44, 4.68, 4.67 [00:54:08] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.68, 4.41, 4.58 [00:55:08] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 14.48, 19.82, 23.17 [00:58:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.25, 3.93, 3.82 [01:00:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 4.03, 3.87 [01:02:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 3.95, 3.87 [01:05:08] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.72, 15.96, 19.74 [01:12:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.77, 2.63, 3.26 [01:49:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.35, 3.63, 3.35 [01:53:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.69, 3.44 [01:57:07] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.00, 3.98, 3.59 [01:59:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.20, 3.33, 3.40 [02:30:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.95, 3.54 [02:32:47] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.68, 3.48 [02:42:43] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.11, 3.12, 3.37 [03:35:59] PROBLEM - wiki.tigcity.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.tigcity.tk' expires in 15 day(s) (Sat 20 Nov 2021 03:31:27 GMT +0000). [03:48:55] PROBLEM - famedata.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'famedata.org' expires in 15 day(s) (Sat 20 Nov 2021 03:44:02 GMT +0000). [03:52:27] PROBLEM - www.famedata.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'famedata.org' expires in 15 day(s) (Sat 20 Nov 2021 03:44:02 GMT +0000). [03:59:24] PROBLEM - reviews.thejoshmeister.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for reviews.thejoshmeister.com could not be found [04:10:39] PROBLEM - www.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.petrawiki.org' expires in 15 day(s) (Sat 20 Nov 2021 04:06:27 GMT +0000). [04:12:59] PROBLEM - files.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'files.petrawiki.org' expires in 15 day(s) (Sat 20 Nov 2021 04:08:07 GMT +0000). [04:13:07] PROBLEM - en.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.petrawiki.org' expires in 15 day(s) (Sat 20 Nov 2021 04:06:27 GMT +0000). [04:15:14] PROBLEM - petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.petrawiki.org' expires in 15 day(s) (Sat 20 Nov 2021 04:06:27 GMT +0000). [04:24:30] PROBLEM - translate.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'translate.petrawiki.org' expires in 15 day(s) (Sat 20 Nov 2021 04:17:44 GMT +0000). [05:16:18] RECOVERY - reviews.thejoshmeister.com - reverse DNS on sslhost is OK: SSL OK - reviews.thejoshmeister.com reverse DNS resolves to cp14.miraheze.org - CNAME OK [06:05:02] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 10.12, 6.39, 3.89 [06:08:12] PROBLEM - cp14 Current Load on cp14 is CRITICAL: CRITICAL - load average: 2.07, 1.92, 1.20 [06:10:12] RECOVERY - cp14 Current Load on cp14 is OK: OK - load average: 0.75, 1.55, 1.15 [06:21:02] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 5.27, 7.33, 6.53 [06:25:02] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.60, 6.66, 6.46 [06:51:02] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.57, 7.37, 6.91 [06:51:39] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.94, 3.07, 2.36 [06:53:38] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.96, 2.82, 2.35 [06:54:45] Passwords aren't stored in plain text bukkit so you wouldn't immediately have the password even if CentralAuth data was leaked. You could brute force them until you get a match but that takes time. [06:55:06] It would be the token that's more dangerous [06:55:23] Which is why we'd log everyone out [06:55:59] Because the token allows account access while it's valid [06:56:01] PROBLEM - hr.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hr.petrawiki.org' expires in 15 day(s) (Sat 20 Nov 2021 06:49:08 GMT +0000). [07:07:12] PROBLEM - db12 APT on db12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:09:02] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 11.11, 8.49, 7.53 [07:09:31] PROBLEM - db12 Current Load on db12 is CRITICAL: CRITICAL - load average: 41.56, 17.02, 8.93 [07:09:51] PROBLEM - db12 Puppet on db12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:11:02] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 5.57, 7.23, 7.18 [07:11:20] RECOVERY - db12 APT on db12 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [07:11:45] RECOVERY - db12 Puppet on db12 is OK: OK: Puppet is currently enabled, last run 22 minutes ago with 0 failures [07:12:42] alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:13:02] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 4.33, 6.00, 6.72 [07:14:57] I'm assuming db related [07:14:59] Reception123: ^ [07:17:41] ok : [RESOLVED] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:19:21] PROBLEM - db12 Current Load on db12 is WARNING: WARNING - load average: 1.59, 5.17, 7.30 [07:21:21] RECOVERY - db12 Current Load on db12 is OK: OK - load average: 1.84, 4.12, 6.66 [07:35:32] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 174.01 ms [07:36:19] PROBLEM - ping4 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 162.54 ms [12:08:01] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JPdp4 [12:08:03] [02miraheze/MirahezeMagic] 07translatewiki 03f4e991b - Localisation updates from https://translatewiki.net. [12:08:03] [url] Main page - translatewiki.net | translatewiki.net [12:08:04] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JPdpB [12:08:06] [02miraheze/ManageWiki] 07translatewiki 03d5a1daa - Localisation updates from https://translatewiki.net. [12:08:06] [url] Main page - translatewiki.net | translatewiki.net [12:11:02] miraheze/MirahezeMagic - translatewiki the build passed. [12:13:31] PROBLEM - diamowiki.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:15:23] miraheze/ManageWiki - translatewiki the build passed. [12:20:23] PROBLEM - diamowiki.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'diamowiki.ga' expires in 11 day(s) (Mon 15 Nov 2021 13:48:35 GMT +0000). [12:42:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.17, 2.69 [12:44:30] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.10, 2.82, 2.63 [13:55:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.43, 3.00 [13:57:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.79, 3.24, 2.98 [14:21:17] [02puppet] 07ugochimobi commented on pull request 03#2104: Add player.vimeo.com and docs.google.com to frame-src - 13https://git.io/JPFgZ [14:41:34] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.83, 3.33, 2.86 [14:43:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.31, 3.21, 2.87 [14:47:30] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.54, 3.09 [14:49:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.83, 3.12, 2.98 [15:46:18] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.45, 2.97 [15:48:18] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.28, 3.10, 2.91 [15:52:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.55, 3.14 [15:58:15] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.50, 3.24, 3.19 [16:22:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.68, 3.42 [16:24:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.54, 3.40 [16:30:07] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.57, 3.09, 3.29 [16:39:47] PROBLEM - wiki.autocountsoft.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:43:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.12, 3.08 [16:45:03] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.25, 3.11, 3.08 [16:46:15] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True} to all [16:46:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:48:31] PROBLEM - wiki.autocountsoft.com - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.query( [16:48:31] ame, 'A')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.001439332962036 seconds [16:55:40] PROBLEM - wiki.autocountsoft.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.autocountsoft.com All nameservers failed to answer the query. [17:00:28] RECOVERY - wiki.autocountsoft.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.autocountsoft.com' will expire on Mon 31 Jan 2022 22:36:14 GMT +0000. [17:02:43] RECOVERY - wiki.autocountsoft.com - reverse DNS on sslhost is OK: SSL OK - wiki.autocountsoft.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [17:05:00] !log [universalomega@mw11] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 1125s [17:05:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:29:00] PROBLEM - wiki.autocountsoft.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.autocountsoft.com All nameservers failed to answer the query. [17:36:18] PROBLEM - wiki.autocountsoft.com - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.query( [17:36:19] ame, 'A')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.00570774078369 seconds [17:43:03] PROBLEM - wiki.autocountsoft.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:17] RECOVERY - wiki.autocountsoft.com - reverse DNS on sslhost is OK: SSL OK - wiki.autocountsoft.com reverse DNS resolves to cp14.miraheze.org - CNAME OK [17:45:45] [02CreateWiki] 07Universal-Omega synchronize pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [17:56:42] RECOVERY - wiki.autocountsoft.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.autocountsoft.com' will expire on Mon 31 Jan 2022 22:36:14 GMT +0000. [18:08:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.01, 2.76 [18:10:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.99, 2.98, 2.78 [18:11:02] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 0.99, 1.79, 1.21 [18:12:58] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.33, 1.28, 1.08 [18:14:39] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.54, 3.04 [18:16:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.68, 3.22, 2.98 [18:20:37] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.90, 3.88, 3.31 [18:22:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.84, 3.80, 3.35 [18:22:46] PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [18:24:22] !log [@test3] starting deploy of {'config': True} to skip [18:24:23] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [18:24:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:45] [02CreateWiki] 07Universal-Omega edited pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [18:25:47] [02CreateWiki] 07Universal-Omega edited pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [18:25:48] [02CreateWiki] 07Universal-Omega edited pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [18:25:50] [02CreateWiki] 07Universal-Omega edited pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [18:25:51] [02CreateWiki] 07Universal-Omega edited pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [18:25:53] [02CreateWiki] 07Universal-Omega edited pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [18:28:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.60, 2.83, 3.11 [18:28:53] [02miraheze/mediawiki] 07Universal-Omega pushed 0311 commits to 03REL1_37 [+0/-0/±22] 13https://git.io/JPb8D [18:28:54] [02miraheze/mediawiki] 07paladox 0311b890c - JobQueueRedis: Replace deprecated zSize with zCard [18:28:56] [02miraheze/mediawiki] 07Ladsgroup 03b3de2da - Update git submodules [18:28:57] [02miraheze/mediawiki] 07urbanecm 03ef08e93 - Update git submodules [18:28:59] [02miraheze/mediawiki] ... and 8 more commits. [18:51:18] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.21, 5.47, 4.30 [18:52:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.82, 5.54, 4.37 [18:54:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.64, 5.30, 4.41 [18:55:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.47, 6.06, 4.82 [18:57:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 3.49, 3.06 [18:59:07] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.82, 3.08, 2.95 [19:16:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.45, 3.13 [19:20:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.26, 4.44, 3.60 [19:22:47] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.73, 3.86, 3.49 [19:28:43] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.31, 4.16, 3.68 [19:30:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.59, 3.64, 3.55 [19:34:28] PROBLEM - cp14 Current Load on cp14 is CRITICAL: CRITICAL - load average: 2.01, 2.47, 1.35 [19:34:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.69, 3.21, 3.39 [19:36:28] PROBLEM - cp14 Current Load on cp14 is WARNING: WARNING - load average: 0.52, 1.71, 1.20 [19:40:28] RECOVERY - cp14 Current Load on cp14 is OK: OK - load average: 0.75, 1.41, 1.21 [19:41:36] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.78, 3.84, 3.57 [19:43:35] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.92, 3.99, 3.67 [19:45:05] [02CreateWiki] 07Universal-Omega synchronize pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [19:45:30] [02CreateWiki] 07Universal-Omega synchronize pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [19:47:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.47, 3.79, 3.64 [19:51:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.67, 3.63 [19:55:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.88, 3.73 [19:56:57] miraheze/CreateWiki - Universal-Omega the build passed. [19:57:11] miraheze/CreateWiki - Universal-Omega the build passed. [19:57:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.06, 3.25, 3.52 [20:01:22] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 2.94, 3.34 [20:01:44] miraheze/CreateWiki - Universal-Omega the build passed. [20:09:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.34, 3.32 [20:11:13] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.55, 3.15, 3.26 [20:15:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.33, 3.30 [20:17:07] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.70, 3.87, 3.50 [20:19:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.85, 3.80, 3.52 [20:21:04] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.89, 4.05, 3.63 [20:23:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.39, 3.37, 3.43 [20:25:01] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.80, 3.08, 3.31 [20:49:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.70, 3.76, 3.32 [20:59:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.89, 3.69 [21:01:01] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.80, 7.07, 6.05 [21:03:01] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.51, 5.99, 5.76 [21:03:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.46, 4.14, 3.84 [21:05:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.42, 3.55, 3.66 [21:11:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.73, 3.93, 3.73 [21:13:24] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.70, 3.67 [21:17:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.59, 4.01, 3.82 [21:23:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.03, 3.70, 3.78 [21:25:25] !log [eval.php on test3] \MediaWiki\MediaWikiServices::getInstance()->get( 'CreateWiki.NotificationsManager' )->sendNotification( ['type' => 'wiki-creation', 'extra' => [ 'wiki-url' => 'https://test3.miraheze.org', 'sitename' => 'test' ], 'subject' => wfMessage( 'createwiki-email-subject', 'test' )->inContentLanguage()->text(),'body' => wfMessage( 'createwiki-email-body' )->inContentLanguage()->parse()], [ 'Universal Omega' ] ); [21:25:27] [url] Test3 | test3.miraheze.org [21:25:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:28:14] [02CreateWiki] 07Universal-Omega edited pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [21:28:16] [02CreateWiki] 07Universal-Omega edited pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [21:28:17] [02CreateWiki] 07Universal-Omega edited pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [21:33:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JPbH8 [21:33:08] [02miraheze/mw-config] 07Universal-Omega 03df80c78 - set wgAllowHTMLEmail to true [21:33:10] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [21:33:11] [02mw-config] 07Universal-Omega opened pull request 03#4202: set wgAllowHTMLEmail to true - 13https://git.io/JPbHB [21:34:19] miraheze/mw-config - Universal-Omega the build passed. [21:35:07] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.43, 3.06, 3.40 [21:37:54] CosmicAlpha: hold [21:38:14] PROBLEM - cp13 Current Load on cp13 is WARNING: WARNING - load average: 1.89, 1.29, 0.80 [21:40:13] RECOVERY - cp13 Current Load on cp13 is OK: OK - load average: 0.98, 1.10, 0.79 [21:41:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.24, 3.63, 3.56 [21:47:47] [02mw-config] 07Universal-Omega closed pull request 03#4202: set wgAllowHTMLEmail to true - 13https://git.io/JPbHB [21:47:48] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [21:47:50] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:47:51] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JPbQF [21:47:53] [02miraheze/mw-config] 07Universal-Omega 03180ddf8 - set wgAllowHTMLEmail to true (#4202) [21:48:51] miraheze/mw-config - Universal-Omega the build passed. [21:48:58] !log [@test3] starting deploy of {'config': True} to skip [21:48:59] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [21:49:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:50:25] [02CreateWiki] 07Universal-Omega synchronize pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [21:50:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.08, 4.08, 3.70 [21:50:58] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.99, 6.44, 5.32 [21:51:57] [02CreateWiki] 07Universal-Omega edited pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [21:52:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.33, 3.85, 3.66 [21:52:59] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.06, 5.81, 5.22 [21:59:21] miraheze/CreateWiki - Universal-Omega the build passed. [22:01:15] [02CreateWiki] 07Universal-Omega edited pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [22:04:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.15, 2.83, 3.27 [22:12:03] !log [@mw11] starting deploy of {'config': True} to all [22:12:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:12:18] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 14s [22:12:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:16:37] did puppet run times change? [22:18:11] [02CreateWiki] 07Universal-Omega edited pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [22:18:13] [02CreateWiki] 07Universal-Omega edited pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [22:18:14] [02CreateWiki] 07Universal-Omega edited pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [22:18:15] [02CreateWiki] 07Universal-Omega edited pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [22:22:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.17, 23.03, 18.52 [22:23:26] [02CreateWiki] 07Universal-Omega edited pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [22:24:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.47, 21.06, 18.35 [22:26:05] [02mw-config] 07CloakSelf commented on pull request 03#4200: Add FameData and WikiData to import - 13https://git.io/JPbFQ [22:26:56] [02CreateWiki] 07Universal-Omega synchronize pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [22:28:58] [02mw-config] 07Universal-Omega commented on pull request 03#4200: Add FameData and WikiData to import - 13https://git.io/JPbbJ [22:30:45] [02CreateWiki] 07Universal-Omega synchronize pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [22:30:46] [02CreateWiki] 07Universal-Omega synchronize pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [22:30:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.64, 23.40, 20.22 [22:31:03] CosmicAlpha: yes [22:31:27] They are now randomised on mw* [22:31:47] [02CreateWiki] 07Universal-Omega synchronize pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [22:32:54] CosmicAlpha: which line? [22:33:39] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.46, 5.59, 4.49 [22:34:31] Bongo-Cat: ? [22:34:49] [mw-config] Universal-Omega commented on pull request #4200: Add FameData and WikiData to import - https://git.io/JPbbJ [22:34:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.35, 22.92, 20.89 [22:35:39] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.99, 4.97, 4.37 [22:35:57] I literally cant see which line it is >.> [22:39:16] [02CreateWiki] 07Universal-Omega edited pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [22:39:18] [02CreateWiki] 07Universal-Omega edited pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [22:39:39] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.79, 5.28, 4.62 [22:41:26] Bongo-Cat: read the CI failure [22:41:39] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.88, 4.63, 4.45 [22:41:56] You need a , on the line you haven't for a start [22:43:06] miraheze/CreateWiki - Universal-Omega the build passed. [22:43:11] miraheze/CreateWiki - Universal-Omega the build passed. [22:43:35] miraheze/CreateWiki - Universal-Omega the build passed. [22:43:52] .op [22:43:53] Attempting to OP... [22:44:14] .deop [22:44:14] Attempting to OP... [22:47:52] miraheze/CreateWiki - Universal-Omega the build passed. [22:49:40] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+3/-0/±11] 13https://git.io/JPbAC [22:49:42] [02miraheze/CreateWiki] 07Universal-Omega 03366e1f2 - Fix notifications and add NotificationsManager service (#256) [22:49:43] [02CreateWiki] 07Universal-Omega closed pull request 03#256: Fix notifications and add NotificationsManager service - 13https://git.io/JP1fg [22:52:10] miraheze/CreateWiki - Universal-Omega the build has errored. [22:53:22] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JPbAP [22:53:24] [02miraheze/CreateWiki] 07Universal-Omega 0371a5132 - Remove whitespace [22:56:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.10, 17.75, 19.89 [23:00:10] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JPbxs [23:00:12] [02miraheze/mediawiki] 07Universal-Omega 032a14586 - Update CreateWiki [23:01:49] miraheze/CreateWiki - Universal-Omega the build passed. [23:02:13] !log [universalomega@mw11] starting deploy of {'world': True} to all [23:02:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:03:13] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JPbxw [23:03:15] [02miraheze/mediawiki] 07Universal-Omega 038a1c16f - Update CreateWiki [23:04:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 41.25, 26.63, 22.15 [23:04:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.85, 3.30, 3.10 [23:05:25] PROBLEM - cp14 Stunnel Http for mw11 on cp14 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.008 second response time [23:05:44] PROBLEM - cp15 Stunnel Http for mw11 on cp15 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.246 second response time [23:05:50] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.50, 8.19, 6.30 [23:05:56] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 17.07, 10.28, 6.99 [23:06:00] !log [universalomega@mw11] DEPLOY ABORTED: Canary check failed for localhost [23:06:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:06:06] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.006 second response time [23:06:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:06:11] PROBLEM - cp12 Stunnel Http for mw11 on cp12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.240 second response time [23:06:15] PROBLEM - cp15 Varnish Backends on cp15 is CRITICAL: 1 backends are down. mw11 [23:06:30] PROBLEM - cp13 Stunnel Http for mw11 on cp13 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.010 second response time [23:06:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.18, 2.82, 2.94 [23:07:15] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw11 [23:07:25] It did its job I guess [23:07:28] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 2 backends are down. mw9 mw11 [23:07:30] PROBLEM - cp14 Varnish Backends on cp14 is CRITICAL: 2 backends are down. mw9 mw11 [23:07:34] CosmicAlpha: ^ [23:07:44] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.63, 7.75, 6.38 [23:07:53] RhinosF1: I see, not sure why, deploy had not even finished pulling to staging yet. [23:08:16] God [23:08:18] I cancelled it once alerts started. [23:08:31] CosmicAlpha: it deployed [23:08:41] The log stated checks failed for localhost [23:08:54] Can you not do test3 first? [23:08:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.89, 22.76, 21.63 [23:08:55] RhinosF1: Yeah, once I cancelled it. [23:09:07] and I did have the changes pulled to test3 already. [23:09:20] Why is prod failing then [23:09:32] What happens if you try run a maint script [23:09:37] Or force mw11 [23:09:39] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.18, 8.64, 6.86 [23:10:16] RhinosF1: Did not deploy. checked mw11 contents, the changes are not there. [23:10:22] CosmicAlpha: right [23:10:29] Do you want me to look [23:11:06] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:11:31] RhinosF1: If you want. Also maint scripts work. [23:11:37] Okay [23:11:40] Give me a sec [23:11:57] mw9 is down also? [23:12:24] Could be php workers [23:12:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.47, 7.34, 6.20 [23:14:50] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.76, 3.48, 3.10 [23:15:55] !log rhinos@mw11:~$ sudo service php7.3-fpm restart [23:16:00] CosmicAlpha: no php workers [23:16:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:16:29] RECOVERY - cp13 Stunnel Http for mw11 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.047 second response time [23:16:31] it shouldn't check pre deploy [23:16:45] but it's stopped because php was faulty [23:16:48] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.92, 2.91, 2.94 [23:16:51] not code issue [23:16:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.44, 18.81, 20.30 [23:17:07] !log mw9: sudo service php7.3-fpm restart [23:17:15] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 9 backends are healthy [23:17:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:17:17] RECOVERY - cp14 Stunnel Http for mw11 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.038 second response time [23:17:20] ah. thanks. Am I good to start again then? [23:17:28] CosmicAlpha: give me 2 [23:17:28] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 9 backends are healthy [23:17:30] RECOVERY - cp14 Varnish Backends on cp14 is OK: All 9 backends are healthy [23:17:41] alright [23:17:44] RECOVERY - cp15 Stunnel Http for mw11 on cp15 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.329 second response time [23:18:00] just want to make sure it doesn't spike again [23:18:07] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20011 bytes in 0.265 second response time [23:18:11] RECOVERY - cp12 Stunnel Http for mw11 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.337 second response time [23:18:15] RECOVERY - cp15 Varnish Backends on cp15 is OK: All 9 backends are healthy [23:18:26] PROBLEM - cp15 Current Load on cp15 is CRITICAL: CRITICAL - load average: 2.36, 2.54, 1.50 [23:18:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.73, 6.47, 6.25 [23:18:43] CosmicAlpha: should be good [23:18:51] !log [universalomega@mw11] starting deploy of {'world': True} to all [23:18:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:19:13] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.04, 7.84, 7.74 [23:19:37] thanks RhinosF1 [23:19:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.84, 7.36, 7.62 [23:20:23] PROBLEM - cp15 Current Load on cp15 is WARNING: WARNING - load average: 1.10, 1.98, 1.41 [23:20:44] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.74, 3.30 [23:22:20] RECOVERY - cp15 Current Load on cp15 is OK: OK - load average: 0.42, 1.44, 1.28 [23:23:40] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.07, 5.54, 6.80 [23:24:29] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.99, 9.82, 7.18 [23:24:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 3.97, 3.52 [23:24:47] !log [universalomega@mw11] finished deploy of {'world': True} to all - SUCCESS in 355s [23:24:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:25:01] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.21, 5.21, 6.59 [23:26:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.03, 3.34, 3.34 [23:28:36] CosmicAlpha: just. had a look at logs and l10n update started as you deployed [23:28:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.08, 20.61, 20.07 [23:29:26] CosmicAlpha: ping when you are past mw11 [23:30:21] RhinosF1: Deploy finished all servers already. [23:30:29] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.08, 6.44, 6.56 [23:30:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.00, 19.64, 19.75 [23:31:01] oh ye [23:32:30] And finally all CreateWiki notifications should be functional. [23:33:17] Every single one had issues before. [23:33:54] !log manually run l10n update on mw11 [23:34:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:36:25]