[00:02:23] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.22, 22.27, 20.36 [00:02:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.11, 3.59, 3.53 [00:04:21] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 3 minutes ago with 13 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [00:05:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.97, 7.15, 7.93 [00:06:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.10, 22.08, 20.83 [00:06:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.09, 3.72, 3.62 [00:06:40] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:06:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.53, 6.64, 7.64 [00:07:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.74, 6.79, 7.96 [00:07:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.22, 6.88, 7.98 [00:08:19] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.99, 23.93, 21.63 [00:09:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.08, 7.30, 8.00 [00:09:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.32, 8.12, 8.29 [00:09:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.19, 7.87, 8.03 [00:10:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.60, 21.89, 21.15 [00:10:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.96, 7.85, 7.86 [00:11:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:11:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.35, 7.36, 7.83 [00:12:19] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.62, 23.99, 22.00 [00:12:33] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:12:34] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:12:44] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:28] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:13:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:13:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.16, 6.68, 7.71 [00:13:53] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:14:07] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:14:07] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:14:08] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:14:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.76, 2.89, 3.36 [00:14:39] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:15:16] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.76, 7.13, 7.86 [00:15:29] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 1.809 second response time [00:16:04] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.410 second response time [00:16:04] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.340 second response time [00:16:45] [02mw-config] 07Kelvs599 opened pull request 03#4367: Add HasSomeColours settings to ManageWikiSettings and add narrow and wide options for wgMedikContentWidth - 13https://git.io/J9Gqo [00:17:46] miraheze/mw-config - Kelvs599 the build passed. [00:18:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.62, 5.34, 4.33 [00:19:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.67, 7.17, 7.64 [00:20:52] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:21:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.85, 7.83, 7.69 [00:22:22] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.30, 3.33 [00:23:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.80, 7.29, 7.65 [00:23:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.55, 7.55, 7.63 [00:24:22] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.802 second response time [00:24:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.42, 3.36 [00:24:58] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.145 second response time [00:25:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:25:49] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[postgresql-contrib-13] [00:26:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.42, 3.37 [00:26:44] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.67, 4.99, 4.73 [00:27:07] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 6.996 second response time [00:27:08] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 6.131 second response time [00:27:21] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 4.797 second response time [00:27:26] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 5.824 second response time [00:27:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.27, 5.67, 6.64 [00:28:31] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.019 second response time [00:28:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.86, 3.55 [00:29:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.27, 7.12, 7.36 [00:29:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:29:41] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.86, 7.49, 7.46 [00:30:19] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [00:30:39] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.57, 5.15, 4.83 [00:31:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:31:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.15, 7.33, 7.04 [00:31:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.56, 7.45, 7.45 [00:32:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.98, 3.66 [00:32:37] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.02, 4.74, 4.71 [00:32:51] [02MirahezeMagic] 07dmehus opened pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [00:33:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.94, 7.36, 7.42 [00:33:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:33:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.47, 7.46, 7.14 [00:33:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.23, 7.56, 7.47 [00:34:08] Agent: ^ [00:34:14] I have to do `qqq.json` still [00:34:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.13, 22.44, 23.80 [00:34:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 3.88, 3.65 [00:34:37] CosmicAlpha, I'm going to work on `qqq.json` then you can merge [00:35:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.86, 7.88, 7.60 [00:35:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.17, 7.96, 7.37 [00:36:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.85, 3.67 [00:37:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb [00:37:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.82, 7.99, 7.46 [00:37:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.83, 7.45, 7.48 [00:38:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.50, 4.16, 3.81 [00:38:46] dmehus: I am kinda against adding defaults that point to meta. For some of the default user groups, like bureaucrats and sysops, I don't think we should force local defaults to point to central miraheze by default. [00:39:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:39:33] I won't completely block merging that either though. [00:39:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.60, 8.28, 7.64 [00:39:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.35, 7.83, 7.62 [00:40:03] miraheze/MirahezeMagic - dmehus the build passed. [00:40:19] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.26, 22.11, 23.13 [00:40:32] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [00:42:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.16, 21.70, 22.83 [00:42:24] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.59, 20.72, 19.22 [00:43:09] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:43:12] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:43:18] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:43:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [00:43:39] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:44:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:44:25] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [00:44:49] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:45:02] CosmicAlpha: done [00:45:09] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.325 second response time [00:45:14] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:45:21] dmehus: thank you! [00:45:45] Do you think you can Special:MyLanguage the links though? [00:45:48] CosmicAlpha, no the reason is so it explains to users what the default groups are and how to locally override them with local MediaWiki interface messages [00:46:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:46:18] Agent, no problem, but is that necessary? We don't do that for the global groups, no? [00:46:48] dmehus: It'd allow users to read a description of the user group in their language [00:46:54] miraheze/MirahezeMagic - dmehus the build passed. [00:47:00] Would be useful for non-English wikis [00:47:04] but yeah, CosmicAlpha, this is better because they point to [[User groups]], rather than Meta specific group pages [00:47:04] https://meta.miraheze.org/wiki/User_groups [00:47:17] Agent, oh, ok, so we should do that for all the groups then? [00:47:24] dmehus: yes please [00:47:25] do I need too? :P [00:47:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:47:34] lol no :P [00:47:39] heh ok [00:48:15] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.00, 20.29, 19.54 [00:48:32] dmehus: that won't override them if default exists, I'm not sure if there's defaults for this. [00:49:49] And overriden messages (the way we do overrides) aren't easy to figure out how to modify either, as you have to modify the `miraheze-` prefixed messages. [00:50:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [00:50:16] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.01, 5.11, 4.75 [00:50:17] CosmicAlpha, what do you mean? [00:50:19] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.55, 22.21, 22.13 [00:50:40] yes, if a user created MediaWiki:Group-autopatrolled-member locally, it will override the global interface message :) [00:50:52] TestWiki does this for many messages :) [00:51:01] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.736 second response time [00:51:12] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:51:26] miraheze/MirahezeMagic - dmehus the build passed. [00:51:26] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:51:30] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.424 second response time [00:51:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:51:33] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:51:35] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 6.229 second response time [00:51:35] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.965 second response time [00:51:46] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:51:52] dmehus: if those interface messages have any defaults at all this won't work. The default groups has default messages, so they can't be overriden by MirahezeMagic, unless they go in /overrides, and are added to https://github.com/miraheze/MirahezeMagic/blob/d96e751c67d9fdc100d6d57e55cb8729d7aedea2/includes/MirahezeMagicHooks.php#L167, at which case can't be overriden locally, the same way anyways. [00:51:52] [url] MirahezeMagic/MirahezeMagicHooks.php at d96e751c67d9fdc100d6d57e55cb8729d7aedea2 · miraheze/MirahezeMagic · GitHub | github.com [00:51:53] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.361 second response time [00:52:13] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:14] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.69, 4.46, 4.55 [00:52:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.07, 22.22, 22.13 [00:52:29] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:50] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:50] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:51] CosmicAlpha, I don't think that's the case here, as we've done it before for all the default user groups :) [00:53:00] except we pointed them to Meta group pages [00:53:10] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.659 second response time [00:53:15] dmehus: we haven't done any there, for knes that have default messages before. [00:53:20] *ones [00:53:21] yeah we did [00:53:25] like back in 2020 [00:53:34] Then it doesn't work. [00:53:42] no, we removed them' [00:53:48] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 2.195 second response time [00:53:50] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [00:53:53] because Redmin didn't like that they pointed to Meta: namespace pages [00:54:14] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 27451 MB (6% inode=97%); [00:54:15] just trust me this will work, we can merge and if there's any that don't work, take them out [00:54:24] dmehus I'm going to just let that case go. [00:54:25] the only one possibly we might have to take out is `user` [00:54:31] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.531 second response time [00:55:10] dmehus: I will test it, but if they do indeed have defaults, I'm 90% sure it won't work. [00:55:51] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [00:56:25] CosmicAlpha, all I know is that we definitely did do this before from whenever we had MirahezeMagic until we removed them [00:57:20] No need to test it, though. We can just deploy on production, and save a step :) [00:57:25] We've always done that before :P [00:57:34] as it's just a minor interface message [00:59:06] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 9.286 second response time [00:59:52] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 9.632 second response time [01:00:40] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [01:01:09] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 9.093 second response time [01:01:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.80, 7.66, 7.97 [01:02:05] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 2.793 second response time [01:02:26] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.008 second response time [01:03:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:03:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.67, 8.19, 8.13 [01:03:54] miraheze/MirahezeMagic - dmehus the build passed. [01:04:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.69, 18.43, 20.15 [01:06:48] dmehus Hey, in case if you haven't read my message, I did say a few minutes ago that I would let that certain case go, as you were right. [01:07:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:08:06] miraheze/MirahezeMagic - dmehus the build passed. [01:13:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:13:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:14:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.62, 20.58, 20.31 [01:14:25] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:14:27] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:15:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.05, 7.33, 7.97 [01:16:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.58, 19.91, 20.10 [01:16:31] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 9.908 second response time [01:16:33] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 9.853 second response time [01:19:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.64, 7.47, 7.83 [01:20:51] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:20:57] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:21:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.28, 7.18, 7.67 [01:21:36] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:21:42] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:21:47] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:21:52] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:03] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:07] dmehus: if you mean https://github.com/miraheze/MirahezeMagic/commit/e4c3f861272fcd291d253a53bc819c9fe3befb47#diff-0312a202c48479900070c5137107b7a768fe7ce38d14e4599d32bc447e19fdd4 none of those have defaults. [01:22:08] [url] Remove Meta-specific messages (#240) · miraheze/MirahezeMagic@e4c3f86 · GitHub | github.com [01:22:15] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:22:37] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:58] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 9.324 second response time [01:23:03] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 8.191 second response time [01:23:22] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:23:47] only user, autoconfirmed, bot, sysop, interface-admin, bureaucrat, and suppress have default messages. [01:24:26] CosmicAlpha, oh you know what we have `checkuser` as a MM interface message [01:24:37] but it still shows up as Check users and check user [01:24:42] which, I assume, are the defaults [01:25:11] can you tell me what I need to do to add those to the /overrides? [01:25:21] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 2.444 second response time [01:25:24] but the grouppage should still be fine, I think [01:25:33] just the group- and group- -member need fixing [01:25:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.94, 7.71, 7.73 [01:25:38] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.091 second response time [01:25:41] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.317 second response time [01:25:57] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 8.439 second response time [01:26:04] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 6.908 second response time [01:26:06] Yeah checkuser probably does have a default, given by extension. And not for the groups I mentioned, grouppage won't work either dmehus. But I'll review the PR with steps of what needs done. [01:26:07] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.343 second response time [01:26:23] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 0.388 second response time [01:26:48] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 9.436 second response time [01:27:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.75, 7.79, 7.77 [01:27:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.10, 7.16, 8.00 [01:28:58] CosmicAlpha, so can we do the same for checkuser then too...that's probably why it's not been showing up correctly [01:29:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:31:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:31:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.11, 7.92, 7.81 [01:31:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.38, 7.49, 7.92 [01:32:08] dmehus: https://github.com/miraheze/MirahezeMagic/pull/311#pullrequestreview-847207055 [01:32:09] [url] Re-add global interface messages for default user groups by dmehus · Pull Request #311 · miraheze/MirahezeMagic · GitHub | github.com [01:32:34] But doing this at all prevents local overrides in the typical way. [01:34:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.69, 3.92 [01:37:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:37:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [01:38:03] I mean, for CheckUser and Oversight it's not a big deal, but for the others, I'd say merge it as is and see if it works [01:38:05] it should [01:38:30] as we definitely have had grouppages for bureaucrat, sysop, interface-admin, autopatrolled, and rollbacker [01:38:48] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:39:18] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:39:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:40:11] [02MirahezeMagic] 07dmehus commented on pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9Gm9 [01:40:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.72, 4.18, 4.05 [01:40:52] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 6.478 second response time [01:40:55] CosmicAlpha, can we try merging it like this and see if it works? I'm quite sure it will as we've definitely done this for the grouppages for the default user groups [01:41:22] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.351 second response time [01:43:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:44:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.35, 3.91, 3.99 [01:45:39] [02MirahezeMagic] 07Universal-Omega commented on pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9Gm5 [01:46:07] [02MirahezeMagic] 07Universal-Omega edited a comment on pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9Gm5 [01:46:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.49, 4.02, 4.01 [01:48:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.52, 3.63, 3.88 [01:48:38] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:48:44] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:48:46] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:49:07] dmehus: we have never done this for groups with defaults I went back in history 2 years and still nothing has ever been done (that works) for groups with defaults. [01:49:09] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:31] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:50:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.97, 3.96, 3.95 [01:50:39] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.277 second response time [01:50:45] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.749 second response time [01:50:53] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.896 second response time [01:51:13] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 6.794 second response time [01:51:20] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:21] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:25] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:29] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.043 second response time [01:51:34] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:41] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:42] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:52] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:55] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:52:02] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:52:21] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:52:29] CosmicAlpha, oh, hrm [01:52:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.92, 3.96, 3.96 [01:52:55] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:53:05] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:53:31] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.939 second response time [01:53:53] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.950 second response time [01:53:56] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.829 second response time [01:54:04] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 2.737 second response time [01:54:05] Agent, what should we do here, should we either (a) drop the idea of interface message overrides for the default groups (described above) or (b) override them using the method CosmicAlpha defined above and then update [[User groups]] to instruct users on how to create a local MediaWiki:Miraheze-group-bureaucrat override? [01:54:05] https://meta.miraheze.org/wiki/User_groups [01:54:22] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.893 second response time [01:54:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.80, 4.28, 4.07 [01:54:58] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 7.144 second response time [01:55:10] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 6.536 second response time [01:55:27] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 7.014 second response time [01:55:28] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 6.948 second response time [01:55:31] dmehus: I thought the goal all along was to do (b) [01:55:36] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.202 second response time [01:55:44] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.331 second response time [01:55:45] I'm in favor of adding a section instructing users how to override it [01:55:52] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 3.035 second response time [01:56:57] Agent, okay, yeah, I think we should repeat how to override the messages within each section though [01:57:11] hmm [01:57:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:57:31] Reason being: users won't want to scroll all the way to the bottom of the page, I think [01:57:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:57:43] but if you think one section is better, that's fine too [02:05:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.71, 7.14, 7.91 [02:07:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.52, 8.41, 8.28 [02:10:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.98, 20.94, 19.76 [02:13:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:13:47] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [02:14:01] dmehus: there's a table of contents. [02:14:19] So you don't have to scroll, just click the section. [02:15:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.61, 7.50, 7.93 [02:18:13] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26708 MB (5% inode=97%); [02:18:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.19, 20.38, 20.04 [02:23:06] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:24:17] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:24:23] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:24:25] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:24:50] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:25:00] CosmicAlpha, yeah I guess that's fine [02:25:16] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:25:37] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.27, 8.00, 7.80 [02:25:49] dmehus: if you update PR I'll get to it tomorrow. I'm going for a little bit now.. May be back later tonight though. [02:27:18] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 5.421 second response time [02:28:28] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.063 second response time [02:28:32] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.949 second response time [02:28:33] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.285 second response time [02:29:07] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 8.801 second response time [02:29:20] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.127 second response time [02:29:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:29:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:29:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.63, 7.46, 7.65 [02:30:55] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/J9GYg [02:30:56] [02miraheze/MirahezeMagic] 07Universal-Omega 031de7fda - Update en.json [02:30:58] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#308: customise import message - 13https://git.io/JDoza [02:32:03] [02MirahezeMagic] 07Universal-Omega closed pull request 03#308: customise import message - 13https://git.io/JDoza [02:32:04] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/J9GYa [02:32:06] [02miraheze/MirahezeMagic] 07Reception123 0300f9ff2 - customise import message (#308) [02:32:07] [02MirahezeMagic] 07Universal-Omega deleted branch 03Reception123-patch-1 - 13https://git.io/fQRGX [02:32:09] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Reception123-patch-1 [02:35:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.45, 7.99, 7.81 [02:38:46] miraheze/MirahezeMagic - Universal-Omega the build passed. [02:41:26] miraheze/MirahezeMagic - Universal-Omega the build passed. [02:41:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [02:42:17] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:42:37] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:42:54] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:43:14] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:43:15] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:44:04] [02puppet] 07Universal-Omega synchronize pull request 03#2259: MediaWiki: configure shellbox - 13https://git.io/JSDjg [02:44:35] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.723 second response time [02:44:54] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 2.423 second response time [02:45:13] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.197 second response time [02:45:14] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.347 second response time [02:46:05] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:48:19] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:49:02] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:49:24] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:49:52] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:49:54] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:50:03] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:50:52] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:51:05] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:51:29] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.182 second response time [02:51:31] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:51:51] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:51:56] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23541 bytes in 8.006 second response time [02:52:05] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.067 second response time [02:52:19] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:52:23] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.014 second response time [02:52:57] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 8.544 second response time [02:53:07] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 6.672 second response time [02:53:12] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.209 second response time [02:53:15] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:53:33] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 4.165 second response time [02:53:49] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 3.088 second response time [02:54:15] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 1.885 second response time [02:54:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.26, 20.26, 19.21 [02:55:11] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.058 second response time [02:56:13] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 27411 MB (6% inode=97%); [02:56:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.53, 20.40, 19.43 [03:00:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.56, 21.08, 20.02 [03:01:18] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:03:15] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.321 second response time [03:03:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:04:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.07, 19.31, 19.57 [03:07:31] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:11:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [03:13:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [03:13:44] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:16:15] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:16:19] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:16:20] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:16:28] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:21:49] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:21:49] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:22:08] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:22:13] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:22:16] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:22:21] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:22:21] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:22:41] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:22:51] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:23:45] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.804 second response time [03:24:12] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.213 second response time [03:24:12] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 0.659 second response time [03:24:16] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.025 second response time [03:25:56] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.654 second response time [03:26:22] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.425 second response time [03:26:33] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.262 second response time [03:26:45] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.076 second response time [03:27:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.24, 7.05, 7.86 [03:27:06] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.21, 20.97, 20.16 [03:27:06] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 9.796 second response time [03:27:40] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [03:28:10] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.018 second response time [03:28:24] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.041 second response time [03:28:27] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.677 second response time [03:28:41] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [03:28:55] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 1.397 second response time [03:29:03] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.75, 20.24, 20.01 [03:29:10] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.862 second response time [03:29:35] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [03:29:49] CosmicAlpha: does that look right now? [03:31:01] miraheze/MirahezeMagic - dmehus the build has errored. [03:31:06] miraheze/MirahezeMagic - dmehus the build has errored. [03:31:24] Ummm [03:31:27] why did it error? [03:31:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:31:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:31:48] oh [03:31:49] comma [03:32:35] miraheze/MirahezeMagic - dmehus the build has errored. [03:32:45] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [03:33:15] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [03:35:24] miraheze/MirahezeMagic - dmehus the build has errored. [03:35:53] miraheze/MirahezeMagic - dmehus the build has errored. [03:35:59] umm why? [03:49:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.56, 7.32, 7.33 [03:52:17] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [03:52:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.99, 19.78, 19.02 [03:52:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [03:54:05] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9GOY [03:54:07] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9GOO [03:54:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.49, 18.94, 18.81 [03:54:32] dmehus: ^ the last entry can't have a "," [03:55:57] That's a json syntax error to have a "," at the last entry. [03:56:05] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:56:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:59:52] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [04:00:42] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:00:55] CosmicAlpha, ack [04:00:57] looking [04:01:02] right [04:01:07] I forgot this is json, not php [04:01:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.59, 6.46, 7.74 [04:02:05] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [04:02:38] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [04:02:45] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 5.514 second response time [04:03:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.63, 7.27, 7.88 [04:04:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb [04:04:30] dmehus: remove all of the new groups which are now in overrides from the other i18n (both en and qqq) [04:05:50] CosmicAlpha, oh, so they only need to be in overrides then? [04:05:59] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw10 [04:06:03] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.05, 19.88, 19.17 [04:06:10] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:06:23] Yes. And then also need to be listed in https://github.com/miraheze/MirahezeMagic/blob/d96e751c67d9fdc100d6d57e55cb8729d7aedea2/includes/MirahezeMagicHooks.php#L167-L183 [04:06:24] [url] MirahezeMagic/MirahezeMagicHooks.php at d96e751c67d9fdc100d6d57e55cb8729d7aedea2 · miraheze/MirahezeMagic · GitHub | github.com [04:06:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.61, 3.92 [04:06:51] dmehus: ^ [04:07:14] CosmicAlpha, ack [04:07:17] will do that after [04:07:26] Alright. [04:07:27] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:07:47] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [04:07:49] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:08:22] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:08:29] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:08:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.93, 4.01 [04:08:45] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [04:09:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.71, 6.69, 7.75 [04:09:27] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.335 second response time [04:09:47] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 3.572 second response time [04:09:50] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9GOE [04:09:57] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.60, 19.79, 19.29 [04:10:08] it says I must be on the branch to propose changes to the magichooks file? [04:10:21] oh [04:10:27] miraheze/MirahezeMagic - dmehus the build passed. [04:10:28] do I have to make the change from my branch? [04:10:34] dmehus: missed one I think. (grouppage-user) [04:10:37] And yes. [04:10:40] You do. [04:10:43] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:10:46] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:11:07] CosmicAlpha, oh right [04:11:10] thanks :) [04:11:11] https://github.com/dmehus/MirahezeMagic/blob/patch-1/includes/MirahezeMagicHooks.php [04:11:12] [url] MirahezeMagic/MirahezeMagicHooks.php at patch-1 · dmehus/MirahezeMagic · GitHub | github.com [04:11:27] Is the file you edit. [04:11:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:11:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:11:59] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 18 backends are healthy [04:13:02] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [04:13:03] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:13:16] miraheze/MirahezeMagic - dmehus the build passed. [04:13:48] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.84, 20.70, 19.80 [04:13:51] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:13:53] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:14:06] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:14:32] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.014 second response time [04:14:39] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.341 second response time [04:15:03] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.429 second response time [04:15:17] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:15:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [04:16:33] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 8.543 second response time [04:16:36] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:16:41] miraheze/MirahezeMagic - dmehus the build passed. [04:16:49] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:16:51] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:17:01] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:17:26] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [04:17:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [04:18:35] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:18:55] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [04:18:58] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:19:03] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:19:27] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:19:40] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.74, 20.17, 19.88 [04:19:43] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:20:33] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.025 second response time [04:20:34] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9GOD [04:20:36] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.316 second response time [04:20:50] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.358 second response time [04:20:58] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [04:21:00] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.308 second response time [04:21:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.90, 7.94, 7.62 [04:21:08] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 0.564 second response time [04:21:08] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.198 second response time [04:21:17] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.520 second response time [04:21:24] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.206 second response time [04:21:41] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.927 second response time [04:22:18] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9GOS [04:22:36] miraheze/MirahezeMagic - dmehus the build passed. [04:22:43] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [04:23:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.36, 6.93, 7.29 [04:23:32] [02MirahezeMagic] 07dmehus reviewed pull request 03#311 commit - 13https://git.io/J9GO9 [04:25:11] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:25:31] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:25:40] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:25:44] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:25:57] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [04:25:58] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:26:09] miraheze/MirahezeMagic - dmehus the build passed. [04:26:12] miraheze/MirahezeMagic - dmehus the build has errored. [04:26:13] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26705 MB (5% inode=97%); [04:26:19] miraheze/MirahezeMagic - dmehus the build passed. [04:26:24] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 2.946 second response time [04:26:33] miraheze/MirahezeMagic - dmehus the build has errored. [04:26:34] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.472 second response time [04:27:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.10, 6.25, 7.66 [04:27:18] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.471 second response time [04:27:22] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.984 second response time [04:27:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:27:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:27:51] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.978 second response time [04:27:52] CosmicAlpha, not sure why the one commit has errored [04:28:27] miraheze/MirahezeMagic - dmehus the build has errored. [04:28:30] miraheze/MirahezeMagic - dmehus the build passed. [04:29:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.36, 6.07, 6.75 [04:31:05] paladox, I'm wondering if MirahezeBot even needs the `+b` flag in here? I would think most ops would use ChanServ to kick/ban users from this channel, and it keeps banning miraheze-github, which is most annoying. Should we remove `-b` from the bot? [04:31:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.85, 7.43, 7.83 [04:31:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [04:31:31] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 8.524 second response time [04:31:44] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.329 second response time [04:32:10] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.669 second response time [04:32:25] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9GOd [04:32:27] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9GOF [04:32:28] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9GOb [04:32:30] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9GON [04:32:31] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9GOA [04:32:32] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9GOx [04:32:34] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9GOp [04:32:36] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9GOh [04:32:37] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9GOj [04:32:39] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9G3e [04:32:40] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9G3v [04:32:41] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9G3f [04:32:43] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9G3J [04:32:44] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9G3U [04:32:46] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9G3T [04:32:47] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9G3k [04:32:49] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9G3I [04:32:50] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9G3L [04:32:52] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9G3t [04:32:54] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9G3q [04:32:55] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9G3m [04:33:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.49, 7.58, 7.85 [04:33:09] dmehus: ^ [04:33:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:33:53] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.416 second response time [04:34:03] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.321 second response time [04:35:36] .op [04:35:36] Attempting to OP... [04:35:40] CosmicAlpha, oh wow, this feels like the never-ending PR :P [04:35:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.86, 6.87, 6.85 [04:35:57] .deop [04:35:57] Attempting to OP... [04:36:01] CosmicAlpha, if you have `+f`, maybe remove -b from the bot? [04:36:27] that was supposed to be fixed and the bot wasn't supposed to ban miraheze-github [04:36:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.10, 7.11, 7.94 [04:37:25] dmehus: it only bans if it is miraheze-github_ as in to many simultaneous actions. [04:37:35] And I don't have +f [04:37:48] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.62, 6.76, 6.80 [04:38:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.74, 7.66, 8.04 [04:39:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.22, 7.89, 7.84 [04:39:32] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [04:40:45] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [04:41:01] CosmicAlpha, oh [04:41:11] I should probably exempt miraheze-github_ then? [04:41:18] Let me see if I can find the repo [04:41:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.49, 7.89, 7.27 [04:42:04] miraheze/MirahezeMagic - dmehus the build has errored. [04:43:29] miraheze/MirahezeMagic - dmehus the build has errored. [04:43:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.73, 7.64, 7.24 [04:43:43] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9G38 [04:45:29] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.70, 8.19, 7.47 [04:46:37] it still errored [04:47:11] dmehus: I know, and I knew it would see the most recent comment (https://git.io/J9G38) [04:47:24] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.55, 7.62, 7.36 [04:47:25] I looked, but only saw the Special:MyLanguage comment [04:47:33] Click the link. [04:47:41] It'll take you straight to it. [04:47:46] oh! [04:47:52] I never noticed the single quotes [04:47:53] lol [04:48:10] And also, remove the json type values. [04:48:32] That is the cause of the error, the single quotes are just a formatting thing. [04:49:19] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.76, 8.14, 7.58 [04:49:22] dmehus: ^ [04:52:55] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [04:53:13] CosmicAlpha, seen, I think I got it now [04:56:17] dmehus: yep, besides the remaining spaces->tabs. [04:58:45] CosmicAlpha, oh, which ones did I miss? [04:59:02] also, I pushed https://github.com/MirahezeBots/AntiSpamSopel/commit/08ee08d2546d77f3feed6c37fb2956a96c6aba63 to the master to whitelist miraheze-github_ [04:59:02] [url] Page not found · GitHub · GitHub | github.com [04:59:12] but I'm not sure if I need to restart the bot or not [04:59:27] dmehus: I marked the ones you got as resolved so all remaining reviews that aren't resolved are missed. [04:59:57] CosmicAlpha, ok looking thanks :) [05:00:37] miraheze/MirahezeMagic - dmehus the build passed. [05:01:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.16, 6.73, 7.82 [05:02:20] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [05:03:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.35, 7.28, 7.89 [05:03:47] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [05:04:43] dmehus: https://github.com/miraheze/MirahezeMagic/pull/311/files/5f4b9fb649815b4b4f410778219782bfa66196ac#diff-61543f52c10c951ed1e2d3ba31cb7f8e104446c90419c149df73fe21a3ecf747 should be all now I think. [05:04:44] [url] Re-add global interface messages for default user groups by dmehus · Pull Request #311 · miraheze/MirahezeMagic · GitHub | github.com [05:05:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.95, 7.62, 7.97 [05:05:23] CosmicAlpha, cool [05:05:26] Oh and https://github.com/miraheze/MirahezeMagic/pull/311#discussion_r780733747 if you're doing anything with that. [05:05:27] [url] Re-add global interface messages for default user groups by dmehus · Pull Request #311 · miraheze/MirahezeMagic · GitHub | github.com [05:06:00] I don't know what you mean by the last one? [05:06:08] Agent requested Special:MyLanguage be added [05:06:20] so users are redirected to their language page on Meta [05:06:23] dmehus: doesn't need to be in qqq.json though. [05:06:30] oh [05:06:33] ok [05:07:19] for the i18n and the overrides qqq.json? [05:07:38] dmehus: also, Special:MyLanguage won't exactly work for the user groups page with the section anchors, as the section anchors differ between localised languages. [05:07:49] oh, right [05:07:57] maybe it's best to remove Special:MyLanguage then [05:08:06] Agent: see CosmicAlpha's explanation [05:08:40] dmehus: if Special:MyLanguage is in both qqq.json then yeah both is what I meant, but maybe it will just be removed altogether I guess? [05:08:52] yeah [05:09:11] good point about the section anchors being localised on the translation pages [05:09:24] miraheze/MirahezeMagic - dmehus the build passed. [05:09:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.72, 6.82, 7.59 [05:10:46] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [05:11:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.14, 7.16, 7.59 [05:11:50] dmehus: yeah, another concern I have is that when this is sent to translatewiki, if the translation there do not *exactly* match up with the section it would mess the anchors up. So maybe... we could utilise usages of {{int:group-message}} on the User Groups for the localised headers to use the localised group messages? And maybe even using int in i18n for section headers would work also, though I don't know. [05:11:50] https://meta.miraheze.org/wiki/int:Template:group-message [05:12:13] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [05:12:55] CosmicAlpha, oh, yeah, we could do that potentially with [[User groups]] [05:12:55] https://meta.miraheze.org/wiki/User_groups [05:13:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.29, 7.19, 7.55 [05:13:01] that should be an easy fix [05:13:23] but I'll remove Special:MyLanguage for now [05:14:16] dmehus: yeah if we use {{int}} on there it would, and then we also need to help {{int}} in section anchors on i18n I think, though not certain. But still it wouldn't be good with Special:MyLanguage. [05:14:16] https://meta.miraheze.org/wiki/Template:int [05:14:30] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [05:14:47] I would want that done before merging also so that it can be before it's sent to translatewiki. [05:14:59] Otherwise we'd have a mess. [05:15:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.80, 7.54, 7.94 [05:15:45] miraheze/MirahezeMagic - dmehus the build passed. [05:16:06] CosmicAlpha, do you mean that I need to add #{{Group-groupname}} to the json files also? [05:16:06] https://meta.miraheze.org/wiki/Template:Group-groupname [05:16:07] If its sent to translatewiki like this chances are anchors would end up slightly differing breaking it on other languages. [05:16:33] err, #{{int-group-groupname}} [05:16:33] https://meta.miraheze.org/wiki/Template:int-group-groupname [05:16:46] Dmehus: if we use the {{int:}} in section headers I think so but it would be #{{int:Group-groupname}} [05:16:46] https://meta.miraheze.org/wiki/int:Template: [05:17:33] CosmicAlpha, okay, so name in both places then [05:17:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.44, 8.67, 8.31 [05:17:56] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [05:18:37] miraheze/MirahezeMagic - dmehus the build passed. [05:19:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.24, 7.52, 7.52 [05:19:33] dmehus: first on-wiki so I can verify it'll work in i18n before you switch it all and incase it doesn't work [05:19:52] okay [05:20:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.76, 4.00 [05:20:49] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [05:21:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.93, 7.72, 7.63 [05:21:15] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:21:30] miraheze/MirahezeMagic - dmehus the build passed. [05:21:36] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [05:22:00] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:22:13] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:22:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:23:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.87, 7.97, 7.72 [05:23:11] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 1.667 second response time [05:23:16] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9G3X [05:23:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.15, 7.27, 7.82 [05:23:56] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.947 second response time [05:24:15] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 3.942 second response time [05:24:24] dmehus: ^ should be the last spaces -> tabs [05:24:27] miraheze/MirahezeMagic - dmehus the build passed. [05:25:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.86, 7.99, 7.76 [05:25:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.14, 7.74, 7.93 [05:27:07] miraheze/MirahezeMagic - dmehus the build passed. [05:27:10] dmehus: I'll finish reviewing it tomorrow. I'm going for now. (Not for the night, just for now) [05:27:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.30, 7.55, 7.28 [05:27:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.27, 7.71, 7.91 [05:28:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.70, 4.03, 4.02 [05:29:03] CosmicAlpha, https://meta.miraheze.org/w/index.php?title=User_groups&diff=225489&oldid=225473&diffmode=source [05:29:06] [url] Difference between revisions of "User groups" - Miraheze Meta | meta.miraheze.org [05:29:08] okay, sounds good [05:29:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.33, 8.22, 8.07 [05:30:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.78, 3.93 [05:30:49] miraheze/MirahezeMagic - dmehus the build passed. [05:31:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.86, 6.99, 7.82 [05:31:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.89, 7.38, 7.29 [05:32:38] [02MirahezeMagic] 07dmehus reviewed pull request 03#311 commit - 13https://git.io/J9G3y [05:33:33] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9G3S [05:34:24] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [05:34:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.44, 3.75, 3.85 [05:34:45] [02MirahezeMagic] 07dmehus reviewed pull request 03#311 commit - 13https://git.io/J9G39 [05:35:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.61, 7.66, 7.42 [05:36:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.65, 3.81 [05:37:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.36, 7.12, 7.26 [05:37:05] dmehus: can you delete https://meta.miraheze.org/w/index.php?title=MediaWiki:Grouppage-bureaucrat? And also yes using int in anchors works, so you can do that. [05:39:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.56, 7.29, 7.28 [05:40:19] CosmicAlpha, sure [05:40:52] dmehus: yes I was just testing it. [05:41:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.15, 7.40, 7.47 [05:41:18] miraheze/MirahezeMagic - dmehus the build passed. [05:42:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.98, 19.56, 18.52 [05:42:21] CosmicALpha, ah, okay, remind me what changes I have left to make [05:42:40] CosmicAlpha, also miraheze-github_ should be whitelisted now :) [05:42:45] dmehus: using {{Int:}} in anchors. [05:42:45] https://meta.miraheze.org/wiki/Int:Template: [05:42:47] ThanksA [05:42:48] that's why I restarted the bots [05:42:51] Thanks! [05:42:57] CosmicAlpha, oh right and no problem [05:43:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.71, 7.89, 7.42 [05:44:13] dmehus: I do any more reviews and hopefully merge tomorrow. [05:45:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.09, 7.46, 7.52 [05:45:33] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [05:46:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.01, 18.79, 18.50 [05:47:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.06, 7.54, 7.43 [05:48:00] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [05:49:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.66, 7.79, 7.52 [05:50:05] wait where's my overrides for the default groups? [05:50:08] I'm lost [05:50:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.49, 3.77, 3.73 [05:50:41] https://github.com/miraheze/MirahezeMagic/pull/311/files#diff-ea1011f7464b56fa9bfc3d33f28f4c4a4984c21a7324f5bd0e72ab10b7551869 [05:50:42] [url] Re-add global interface messages for default user groups by dmehus · Pull Request #311 · miraheze/MirahezeMagic · GitHub | github.com [05:50:48] nvm was in wrong repo [05:50:49] lol [05:50:53] Lol [05:50:59] That'd be why.... [05:51:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.66, 7.50, 7.46 [05:52:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.25, 3.54, 3.65 [05:52:56] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [05:53:02] miraheze/MirahezeMagic - dmehus the build passed. [05:53:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.88, 7.08, 8.00 [05:54:10] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [05:54:22] CosmicAlpha, lol yeah [05:56:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.76, 4.07, 3.80 [05:57:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.75, 6.75, 7.60 [05:57:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.07, 7.50, 7.41 [05:58:00] miraheze/MirahezeMagic - dmehus the build passed. [05:58:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.19, 3.66, 3.68 [05:59:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.48, 7.50, 7.42 [06:00:09] miraheze/MirahezeMagic - dmehus the build passed. [06:03:30] miraheze/MirahezeMagic - dmehus the build passed. [06:03:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.20, 5.97, 6.67 [06:03:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.04, 7.40, 7.99 [06:04:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.84, 6.70, 7.89 [06:05:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.49, 5.47, 6.67 [06:05:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.25, 7.51, 7.94 [06:05:50] RECOVERY - phab2 APT on phab2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:06:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:07:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.11, 8.29, 7.57 [06:07:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.38, 7.43, 7.08 [06:08:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.69, 3.85, 3.69 [06:09:03] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.79, 5.20, 6.49 [06:10:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:12:24] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.04, 20.35, 19.09 [06:12:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.18, 7.10, 7.48 [06:13:18] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:13:39] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:13:41] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb [06:13:56] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:13:58] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:14:20] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:15:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.16, 6.68, 6.70 [06:15:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.16, 7.11, 6.73 [06:15:07] RECOVERY - mw10 APT on mw10 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:15:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.99, 7.61, 7.52 [06:15:41] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.385 second response time [06:15:55] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.557 second response time [06:15:57] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.935 second response time [06:16:20] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 2.835 second response time [06:16:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.62, 7.68, 7.64 [06:17:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.31, 6.24, 6.54 [06:17:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.83, 6.47, 6.54 [06:17:16] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.015 second response time [06:17:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.14, 7.96, 7.67 [06:20:12] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 20.10, 20.20, 19.56 [06:20:17] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb [06:20:55] RECOVERY - jobchron1 APT on jobchron1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:21:16] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:21:50] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:22:39] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:22:49] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:22:59] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:23:08] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:23:21] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.88, 1.50, 1.00 [06:23:22] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 7.624 second response time [06:23:44] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:24:01] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:24:31] RECOVERY - mw13 APT on mw13 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:24:53] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.183 second response time [06:25:02] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.948 second response time [06:25:21] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.24, 1.37, 1.01 [06:25:47] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.818 second response time [06:25:54] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 3.805 second response time [06:26:00] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.82, 21.01, 20.08 [06:26:47] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 3.124 second response time [06:26:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.10, 6.90, 7.12 [06:27:17] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.336 second response time [06:27:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:27:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.68, 7.53, 7.78 [06:27:52] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:28:03] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.823 second response time [06:28:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.14, 6.94, 7.11 [06:29:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.29, 7.92, 7.90 [06:30:29] RECOVERY - mw11 APT on mw11 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:31:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.73, 6.59, 6.43 [06:31:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.33, 7.60, 7.80 [06:31:51] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.06, 19.54, 19.75 [06:32:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.20, 7.22, 7.16 [06:33:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.56, 6.25, 6.33 [06:33:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.26, 8.09, 7.97 [06:35:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.17, 7.16, 7.66 [06:36:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.30, 7.14, 7.17 [06:37:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.35, 7.02, 6.62 [06:39:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.17, 6.74, 6.57 [06:39:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.34, 7.38, 7.59 [06:40:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.13, 7.13, 7.11 [06:41:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.55, 7.84, 6.70 [06:43:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.55, 8.07, 7.12 [06:43:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.16, 6.94, 7.42 [06:44:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.74, 3.63, 3.90 [06:46:32] RECOVERY - mw9 APT on mw9 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:46:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.72, 7.85, 7.58 [06:47:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.56, 7.55, 7.16 [06:47:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.95, 7.58, 7.06 [06:47:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.62, 5.68, 6.80 [06:47:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.57, 6.88, 7.87 [06:48:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.67, 8.29, 7.76 [06:49:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.11, 7.99, 7.36 [06:49:06] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.07, 8.05, 7.29 [06:49:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.78, 6.66, 7.81 [06:50:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.58, 3.77 [06:51:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.05, 7.61, 7.22 [06:51:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.14, 6.64, 7.50 [06:54:46] RECOVERY - mail2 APT on mail2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:55:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.63, 7.73, 7.33 [06:55:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.42, 7.31, 7.66 [06:55:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:57:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.64, 7.37, 7.26 [06:57:21] PROBLEM - ping6 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 100% [06:57:29] PROBLEM - bacula2 Current Load on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:57:54] PROBLEM - bacula2 SSH on bacula2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:57:59] RECOVERY - mon2 APT on mon2 is OK: APT OK: 7 packages available for upgrade (0 critical updates). [06:58:14] RECOVERY - mw12 APT on mw12 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:58:20] PROBLEM - bacula2 Disk Space on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:58:37] PROBLEM - bacula2 conntrack_table_size on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:58:40] PROBLEM - bacula2 NTP time on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:58:41] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:58:51] PROBLEM - Host bacula2 is DOWN: PING CRITICAL - Packet loss = 100% [06:59:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.13, 7.04, 7.50 [07:01:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.96, 8.26, 7.89 [07:01:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.10, 7.59, 7.30 [07:01:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.25, 7.32, 7.65 [07:01:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.12, 7.44, 7.82 [07:02:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:04:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.69, 3.86 [07:07:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.76, 7.80, 7.93 [07:07:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.79, 7.28, 7.30 [07:09:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.75, 5.22, 6.54 [07:10:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.68, 3.75 [07:10:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.80, 7.33, 7.94 [07:12:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.58, 3.69 [07:12:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:12:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.16, 8.81, 8.43 [07:13:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.13, 7.87, 7.80 [07:13:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.19, 6.95, 7.14 [07:15:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.32, 7.48, 7.68 [07:15:06] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.54, 7.79, 7.40 [07:15:41] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.64, 6.90, 7.10 [07:16:09] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.86, 7.49, 7.08 [07:16:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:17:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.55, 7.95, 7.83 [07:20:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.32, 3.54, 3.56 [07:21:25] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.77, 7.15, 6.62 [07:21:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.59, 8.05, 7.47 [07:21:59] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.63, 7.61, 7.27 [07:22:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.05, 3.33, 3.48 [07:23:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.82, 7.40, 7.30 [07:25:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.05, 7.82, 7.82 [07:25:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.07, 7.66, 7.41 [07:25:52] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.10, 7.74, 7.31 [07:26:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.85, 3.66 [07:27:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.48, 7.02, 7.22 [07:27:49] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.69, 7.01, 7.10 [07:28:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.85, 3.68 [07:29:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.02, 7.23, 7.88 [07:29:18] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.09, 6.59, 6.66 [07:30:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.40, 6.99, 7.98 [07:31:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:33:39] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.38, 6.42, 6.79 [07:33:41] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.89, 5.87, 6.67 [07:34:13] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 27291 MB (6% inode=97%); [07:34:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.23, 3.00, 3.38 [07:36:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:37:34] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.08, 7.04, 6.98 [07:39:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.06, 7.60, 7.47 [07:39:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.43, 6.22, 6.80 [07:39:30] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.66, 6.38, 6.74 [07:40:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.43, 3.43 [07:41:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.50, 6.83, 7.21 [07:43:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.56, 7.75, 7.50 [07:43:06] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.91, 7.45, 6.59 [07:43:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.09, 6.99, 6.95 [07:43:25] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.56, 7.55, 7.13 [07:44:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.64, 8.02, 7.81 [07:45:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.43, 7.38, 7.38 [07:45:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.43, 6.05, 6.61 [07:47:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.04, 8.11, 7.66 [07:47:03] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.97, 6.78, 6.53 [07:47:18] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.28, 7.03, 7.07 [07:49:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.14, 7.51, 7.51 [07:50:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 3.69, 3.52 [07:50:58] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.42, 7.57, 6.90 [07:51:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.51, 8.26, 7.80 [07:51:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.64, 7.72, 7.31 [07:51:56] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.50, 7.46, 7.07 [07:52:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.03, 3.36, 3.42 [07:52:57] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.18, 7.02, 6.78 [07:53:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.16, 7.85, 7.02 [07:54:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.62, 3.50 [07:54:55] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.82, 8.16, 7.23 [07:57:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.40, 7.71, 7.82 [07:57:40] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 8.00, 7.75, 7.40 [08:00:51] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.06, 7.48, 7.32 [08:02:49] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.26, 7.71, 7.42 [08:02:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.25, 6.94, 7.71 [08:04:48] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.95, 7.31, 7.30 [08:05:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 8.00, 7.98, 7.83 [08:05:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.43, 7.65, 7.61 [08:06:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.46, 7.81, 7.88 [08:07:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.71, 7.10, 7.27 [08:08:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.17, 3.78, 3.96 [08:08:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.23, 7.39, 7.73 [08:09:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.87, 7.02, 7.22 [08:09:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.94, 7.42, 7.56 [08:09:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.62, 8.14, 7.74 [08:10:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 4.07, 4.05 [08:10:45] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.73, 7.21, 7.14 [08:11:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.61, 6.14, 6.68 [08:11:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.43, 7.91, 7.71 [08:12:45] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.85, 7.50, 7.26 [08:12:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.93, 7.81, 7.76 [08:13:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.61, 8.72, 8.03 [08:14:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.09, 3.73, 3.93 [08:15:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.62, 7.80, 7.37 [08:15:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.50, 7.72, 7.81 [08:15:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.78, 7.86, 7.80 [08:16:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:17:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.04, 7.11, 7.17 [08:18:41] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.03, 5.86, 6.68 [08:19:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.50, 7.78, 7.41 [08:19:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.31, 6.81, 6.85 [08:19:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.43, 7.40, 7.61 [08:21:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.36, 7.91, 7.26 [08:21:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.95, 7.47, 7.63 [08:21:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [08:23:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:23:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:25:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.12, 7.62, 7.61 [08:26:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.18, 4.13, 3.88 [08:26:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.05, 6.93, 6.81 [08:27:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.05, 7.22, 7.26 [08:27:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.62, 7.46, 7.55 [08:28:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.94, 3.83 [08:28:33] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.00, 6.04, 6.51 [08:28:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:29:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.11, 7.57, 7.76 [08:30:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.29, 4.22, 3.93 [08:30:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 3.73, 6.89, 7.93 [08:31:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.95, 5.68, 6.59 [08:32:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.89, 3.72, 3.79 [08:33:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.23, 5.74, 6.66 [08:35:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.24, 5.91, 6.77 [08:39:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.64, 5.68, 6.59 [08:39:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.68, 6.62, 6.88 [08:39:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:40:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.99, 6.31, 6.15 [08:40:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 3.57, 3.63 [08:40:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.23, 5.72, 6.70 [08:41:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.20, 6.43, 6.78 [08:42:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.26, 3.16, 3.48 [08:44:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.53, 7.22, 7.15 [08:45:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.82, 6.77, 6.88 [08:46:17] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.76, 6.60, 6.36 [08:46:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.17, 7.38, 7.21 [08:48:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.88, 7.21, 7.14 [08:50:13] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.37, 7.04, 6.65 [08:50:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.79, 3.63 [08:50:33] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.44, 7.65, 6.93 [08:50:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.81, 8.11, 7.48 [08:51:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.91, 6.73, 6.54 [08:52:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.81, 7.05, 6.80 [08:52:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.80, 3.56, 3.58 [08:54:09] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.49, 6.66, 6.58 [08:54:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:55:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.45, 7.52, 6.90 [08:56:16] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.18, 7.70, 7.12 [08:56:22] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.43, 7.43, 6.77 [08:58:05] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.58, 7.21, 6.79 [08:58:11] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.07, 7.25, 7.03 [08:58:13] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26700 MB (5% inode=97%); [08:58:20] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.58, 7.61, 6.93 [08:59:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.56, 7.65, 7.11 [08:59:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:02:01] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.04, 7.39, 6.98 [09:02:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.83, 3.62 [09:02:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.27, 7.95, 7.78 [09:03:54] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.48, 6.38, 6.74 [09:04:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.84, 3.56, 3.56 [09:04:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.40, 8.01, 7.81 [09:05:58] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.50, 6.39, 6.67 [09:06:11] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.76, 7.87, 7.29 [09:06:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.31, 7.13, 7.49 [09:07:15] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.42, 6.30, 6.74 [09:08:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.41, 7.75, 7.32 [09:09:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.22, 6.07, 6.63 [09:09:12] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.80, 1.33, 0.92 [09:09:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:10:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.08, 5.44, 6.77 [09:11:12] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.81, 1.13, 0.90 [09:12:04] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.30, 5.76, 6.58 [09:18:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.48, 2.96, 3.35 [09:18:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.95, 7.23, 6.93 [09:19:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.95, 7.25, 6.64 [09:20:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.94, 6.77, 6.81 [09:21:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.16, 7.87, 6.72 [09:21:44] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.87, 6.88, 6.57 [09:21:46] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 12.98, 8.83, 7.12 [09:21:51] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.72, 7.90, 7.17 [09:22:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 3.49, 3.50 [09:22:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.66, 6.07, 6.55 [09:23:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb [09:23:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [09:23:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:23:40] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.76, 5.78, 6.21 [09:23:49] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:19] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:25:09] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:25:14] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:25:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:25:44] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 0.349 second response time [09:26:19] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.345 second response time [09:27:13] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.634 second response time [09:27:19] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:27:21] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.676 second response time [09:27:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.42, 6.50, 6.38 [09:27:39] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:27:57] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:28:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.57, 4.37, 3.85 [09:29:22] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.783 second response time [09:29:26] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.70, 5.92, 6.19 [09:29:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [09:29:39] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.423 second response time [09:29:52] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.307 second response time [09:31:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:31:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [09:33:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:34:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.46, 3.99, 3.94 [09:35:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.93, 7.35, 7.60 [09:35:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.13, 7.39, 7.70 [09:39:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.92, 5.19, 6.63 [09:39:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.77, 5.96, 7.67 [09:39:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 2.99, 5.16, 6.73 [09:40:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.60, 3.73 [09:40:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.40, 7.37, 6.76 [09:42:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.62, 3.72 [09:42:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.06, 7.30, 6.82 [09:46:57] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.50, 6.61, 6.68 [09:47:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.36, 5.27, 6.73 [09:48:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 3.58, 3.65 [09:50:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.54, 3.23, 3.52 [09:50:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.69, 7.18, 6.88 [09:51:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.45, 6.82, 6.42 [09:52:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.04, 6.69, 6.73 [09:53:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.43, 6.84, 6.47 [09:54:29] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.59, 7.74, 6.98 [09:54:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:56:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.75, 6.82, 6.73 [09:57:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.73, 6.12, 6.29 [09:58:25] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.31, 5.68, 6.33 [09:59:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:00:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.35, 2.56, 3.17 [10:06:08] CosmicAlpha, dmehus: I have added anchor tags to each section and have named them according to their technical name (i.e. sysop, interface-admin, etc) so linking to Special:MyLanguage/User groups#sysop for example should redirect users to the "Administrators" section of the User groups page regardless of the language [10:06:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:11:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:21:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.41, 7.33, 6.14 [10:23:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.18, 6.60, 6.02 [10:23:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.86, 7.41, 6.66 [10:27:37] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.82, 7.46, 6.81 [10:29:34] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.47, 6.77, 6.64 [10:30:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:35:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:41:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:43:16] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.80, 6.45, 5.95 [10:43:59] [02mw-config] 07AgentIsai opened pull request 03#4368: Sitenotice for feedback on Category Browser removal - 13https://git.io/J9Gay [10:44:13] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 27255 MB (6% inode=97%); [10:44:58] miraheze/mw-config - AgentIsai the build passed. [10:45:11] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.71, 7.35, 6.32 [10:45:17] RhinosF1: can you merge? ^ [10:46:08] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.19, 7.74, 6.63 [10:46:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:46:45] [02mw-config] 07RhinosF1 closed pull request 03#4368: Sitenotice for feedback on Category Browser removal - 13https://git.io/J9Gay [10:46:46] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9GVa [10:46:48] [02miraheze/mw-config] 07AgentIsai 035303dfe - Sitenotice for feedback on Category Browser removal (#4368) [10:46:53] Thank you! [10:47:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.68, 7.28, 6.45 [10:47:07] Agent: it deploys within 30 minutes etc etc [10:47:08] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.47, 6.97, 5.85 [10:47:12] you know by now [10:47:14] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9GVK [10:47:15] [02miraheze/mw-config] 07github-actions 032006d77 - CI: lint code to MediaWiki standards [10:47:22] ack [10:47:29] !log [@test101] starting deploy of {'config': True} to skip [10:47:30] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [10:47:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:47:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:47:55] miraheze/mw-config - RhinosF1 the build passed. [10:49:31] Agent: it looks like an alternative will exist [10:49:53] so hopefully we can do a replacement to enable it as an extension [10:51:06] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.76, 8.02, 6.94 [10:51:41] !log [@mwtask111] starting deploy of {'config': True} to scsvg [10:51:47] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 6s [10:51:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:52:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:52:17] oh good [10:52:35] so they're just going to end up moving it to an extension [10:52:40] likely [10:52:53] obviously it'll need security review [10:53:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.30, 7.20, 6.46 [10:53:02] so we need it soon [10:53:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.29, 7.50, 6.89 [10:53:11] the branch point isn't far off [10:53:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 8.00, 6.35, 5.57 [10:53:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.68, 7.72, 7.17 [10:54:10] !log [@test3] starting deploy of {'config': True} to skip [10:54:11] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [10:54:19] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.88, 6.93, 6.04 [10:54:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:54:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:55:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.12, 5.99, 5.55 [10:56:17] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.37, 7.72, 6.43 [10:56:49] Agent: we'd be aiming for a 21st April release [10:57:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.97, 7.69, 6.79 [10:57:04] 24th Feb is branch cut [10:57:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.04, 6.72, 6.70 [10:57:32] so 7 weeks to cut [10:57:37] 15 to release [10:57:53] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.89, 9.59, 7.98 [10:58:14] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.77, 7.21, 6.43 [10:59:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.54, 7.89, 6.99 [11:00:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:03:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.10, 6.75, 6.05 [11:04:07] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.70, 6.36, 6.29 [11:05:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.01, 6.91, 6.81 [11:05:15] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.95, 5.98, 5.85 [11:05:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.38, 7.83, 7.76 [11:07:00] [02dns] 07RhinosF1 synchronize pull request 03#233: mediawiki: failover deployment & mwtask - 13https://git.io/JDyu6 [11:07:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.89, 6.37, 6.70 [11:07:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.68, 6.32, 6.60 [11:10:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:11:16] !log [@mw11] starting deploy of {'config': True} to ovlon [11:11:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:11:35] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.61, 7.12, 7.36 [11:13:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:14:17] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.09, 1.94, 1.53 [11:15:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.38, 6.72, 6.58 [11:15:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.91, 6.67, 6.53 [11:15:30] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.74, 7.72, 7.58 [11:16:11] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.86, 2.03, 1.62 [11:17:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.24, 6.70, 6.60 [11:17:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.90, 6.25, 6.38 [11:18:05] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.98, 1.65, 1.53 [11:18:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:19:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.72, 8.00, 7.68 [11:21:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 7.87, 8.05, 7.22 [11:21:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.58, 7.00, 6.70 [11:22:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:23:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.51, 7.58, 7.13 [11:23:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.11, 5.90, 6.33 [11:23:20] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.68, 7.85, 7.79 [11:24:20] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.14, 6.54, 5.89 [11:25:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.97, 3.79, 3.37 [11:25:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.05, 7.20, 6.40 [11:25:51] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.07, 7.14, 6.32 [11:26:18] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.73, 6.76, 6.06 [11:27:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.57, 3.63, 3.36 [11:27:14] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.81, 9.02, 8.19 [11:27:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.77, 6.30, 6.17 [11:27:48] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.19, 6.51, 6.19 [11:29:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.59, 2.86, 3.11 [11:30:13] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26684 MB (5% inode=97%); [11:31:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.10, 7.66, 7.34 [11:31:45] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.21, 6.66, 6.29 [11:32:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:33:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.54, 7.08, 7.18 [11:33:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.72, 6.89, 6.43 [11:34:10] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.60, 6.91, 6.28 [11:35:04] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.71, 7.80, 8.00 [11:35:37] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.20, 7.55, 6.74 [11:35:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.36, 7.31, 6.63 [11:36:08] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.47, 6.52, 6.23 [11:37:33] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.50, 7.24, 6.72 [11:37:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:39:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.78, 7.65, 6.97 [11:40:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.38, 3.30 [11:41:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.31, 7.78, 7.09 [11:42:03] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.95, 7.33, 6.72 [11:42:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.56, 8.44, 8.13 [11:43:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.60, 7.68, 7.13 [11:44:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.49, 3.19, 3.26 [11:46:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.88, 7.71, 6.78 [11:47:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 7.91, 8.18, 7.61 [11:47:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.64, 7.47, 7.05 [11:47:41] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.27, 8.11, 7.38 [11:48:13] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.92, 7.53, 6.82 [11:49:56] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.98, 6.62, 6.62 [11:50:08] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.18, 7.85, 7.02 [11:51:07] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:51:32] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:51:49] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [11:52:02] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.82, 7.77, 7.08 [11:52:19] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:52:43] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:52:49] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:53:12] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23539 bytes in 5.983 second response time [11:53:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.13, 7.52, 7.35 [11:53:30] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.876 second response time [11:53:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.01, 7.27, 7.31 [11:53:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:54:16] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.522 second response time [11:54:45] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.367 second response time [11:54:48] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 2.710 second response time [11:55:51] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.65, 8.30, 7.43 [11:59:40] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.35, 7.74, 7.44 [12:01:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.28, 7.48, 7.29 [12:02:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:03:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.23, 7.31, 7.81 [12:03:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.41, 6.84, 7.09 [12:03:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.72, 6.26, 6.72 [12:05:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.29, 7.26, 7.20 [12:07:40] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.88, 7.52, 6.67 [12:07:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.58, 7.63, 7.17 [12:07:59] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.52, 3.20 [12:09:38] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.34, 6.85, 6.55 [12:09:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.72, 7.29, 7.10 [12:09:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.32, 3.40, 3.18 [12:10:00] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 9.41, 7.88, 4.69 [12:11:37] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.49, 7.49, 6.81 [12:11:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.49, 8.00, 7.37 [12:13:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.30, 7.51, 7.50 [12:13:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.85, 6.29, 6.72 [12:14:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [12:14:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.58, 6.94, 7.76 [12:15:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.18, 6.81, 7.24 [12:15:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:15:57] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 2.13, 5.59, 4.79 [12:16:06] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:17:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.21, 7.15, 7.30 [12:17:57] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 1.35, 4.24, 4.39 [12:18:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.63, 7.91, 7.92 [12:19:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.04, 6.87, 6.75 [12:20:11] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 149.56.141.75/cpweb [12:21:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.51, 7.49, 7.02 [12:21:42] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:21:49] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:21:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [12:22:58] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:23:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.87, 7.59, 7.59 [12:23:12] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:25:01] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.402 second response time [12:25:18] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.219 second response time [12:25:49] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.145 second response time [12:25:52] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 3.874 second response time [12:25:52] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 6.15, 5.93, 4.82 [12:26:14] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:26:30] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:26:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.87, 7.24, 7.75 [12:27:06] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:27:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.22, 6.53, 6.77 [12:27:10] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:27:18] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:27:23] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:47] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:27:49] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:27:52] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 1.92, 4.36, 4.38 [12:28:36] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 8.418 second response time [12:29:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.46, 5.36, 6.62 [12:29:30] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 9.467 second response time [12:29:42] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:29:48] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.509 second response time [12:30:20] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.006 second response time [12:30:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.87, 7.90, 7.90 [12:31:06] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.342 second response time [12:31:16] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.014 second response time [12:31:16] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.333 second response time [12:33:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.60, 6.94, 6.98 [12:35:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.11, 8.04, 7.38 [12:37:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.75, 6.74, 7.70 [12:40:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.70, 7.50, 7.89 [12:41:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.81, 7.58, 7.44 [12:42:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.37, 8.27, 8.12 [12:43:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.64, 8.18, 7.69 [12:44:30] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.19, 6.83, 6.57 [12:46:13] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 27899 MB (6% inode=97%); [12:46:25] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.35, 6.73, 6.57 [12:46:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.39, 7.90, 8.00 [12:47:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.39, 7.84, 7.67 [12:47:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.02, 7.89, 7.78 [12:48:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.61, 8.67, 8.28 [12:49:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.10, 8.06, 7.78 [12:49:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.74, 7.79, 7.75 [12:50:14] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.62, 7.43, 6.86 [12:50:41] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 11.69, 6.18, 4.21 [12:51:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [12:52:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.06, 21.03, 19.21 [12:53:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.14, 7.68, 7.69 [12:54:03] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.74, 7.46, 6.99 [12:54:40] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 1.88, 4.63, 4.10 [12:55:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.25, 7.77, 7.16 [12:57:12] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.09, 1.54, 1.18 [12:57:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.64, 7.90, 7.84 [12:59:12] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.15, 1.38, 1.17 [12:59:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.79, 4.13, 3.57 [12:59:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:01:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.65, 3.45 [13:01:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.37, 8.05, 7.89 [13:01:41] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.87, 7.29, 7.21 [13:02:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.82, 18.99, 19.33 [13:03:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.51, 7.53, 7.97 [13:03:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.29, 7.87, 7.85 [13:03:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.77, 7.32, 7.96 [13:05:04] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 3.25, 3.35 [13:05:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.79, 7.63, 7.95 [13:08:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.62, 3.48 [13:09:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.57, 7.39, 7.70 [13:09:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.63, 7.61, 7.84 [13:10:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.74, 4.09, 3.68 [13:11:13] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.08, 8.14, 7.46 [13:11:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.14, 7.98, 7.91 [13:11:31] dmehus: yeh. I think we should remove MirahezeBot ability to ban in here. It keeps banning the wrong people. [13:11:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.78, 8.44, 7.90 [13:13:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.52, 7.61, 7.72 [13:13:04] paladox: the fix is easy [13:14:46] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.99, 3.66, 3.61 [13:15:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.38, 8.40, 8.01 [13:15:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.58, 7.73, 7.87 [13:15:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.34, 7.45, 7.62 [13:15:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.31, 7.59, 7.88 [13:15:41] RhinosF1: it’s been doing it for over a year [13:16:35] paladox: it's up to someone to decide to move it to its own irc account so it has more connections or whitelist the logged out nicks [13:17:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.07, 7.71, 7.88 [13:17:56] Nobody has bothered because it's azure IPs for miraheze-github and it uses a new one each time [13:18:38] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.58, 3.56 [13:18:53] RhinosF1: not really anyones problem though. That’s the bots fault. [13:19:02] Not being MirahezeBots [13:19:29] I mean it is MirahezeBots sorry [13:19:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.69, 6.99, 7.60 [13:19:46] paladox: an account failing to login because we haven't set it up right so be banned for being a proxy rightly based on the abuse we used to get [13:19:57] Is 100% Miraheze's fault [13:20:10] We set the github actions alerts up [13:20:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.06, 3.32, 3.46 [13:24:18] RhinosF1: paladox it won’t be able to login right now as we have many bots running - you can’t have more then 5 simultaneous logins. [13:24:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.65, 3.55 [13:25:10] MacFan4000: which is why it needs its own account [13:25:21] As it can have 2-3 at once sometimes [13:26:16] And that’s also why it still sometimes can’t login because when people do many commits in rapid, one instance might join before another fully quits and because we sre already at the limit, a login isn’t possible [13:26:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.07, 3.08, 3.36 [13:26:41] Yes [13:26:48] We know the issue [13:26:56] It's just no one wants to fix it [13:27:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.93, 7.79, 7.89 [13:27:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.52, 7.47, 7.60 [13:33:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.59, 7.40, 7.56 [13:37:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.25, 7.55, 7.54 [13:37:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.70, 6.43, 6.80 [13:39:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.64, 7.69, 7.60 [13:41:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.98, 8.01, 7.72 [13:43:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.67, 7.36, 7.93 [13:43:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.12, 7.45, 7.54 [13:43:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.16, 7.40, 7.11 [13:45:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.76, 7.97, 8.09 [13:45:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.05, 7.73, 7.61 [13:45:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.45, 6.63, 6.86 [13:47:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.41, 7.97, 7.57 [13:47:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.59, 7.50, 7.35 [13:49:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.90, 7.52, 7.17 [13:50:44] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:50:57] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:51:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.36, 7.93, 7.75 [13:51:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [13:51:32] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:51:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.35, 7.28, 7.12 [13:51:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [13:52:43] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 2.374 second response time [13:52:53] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.052 second response time [13:53:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.56, 7.37, 7.53 [13:53:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.32, 8.51, 7.99 [13:53:27] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.338 second response time [13:53:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:53:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.43, 8.12, 7.45 [13:54:12] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:54:24] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:37] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:55:03] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:55:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.71, 7.53, 7.32 [13:56:10] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.031 second response time [13:56:22] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 0.295 second response time [13:56:32] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.016 second response time [13:57:04] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.341 second response time [13:57:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:59:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.79, 7.77, 7.92 [14:01:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.42, 7.01, 7.52 [14:07:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.71, 6.88, 7.83 [14:07:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.27, 7.19, 7.43 [14:08:58] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.51, 6.18, 7.76 [14:09:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.32, 7.79, 7.40 [14:09:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.20, 7.55, 7.48 [14:10:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.38, 3.75, 3.51 [14:10:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.28, 7.29, 7.96 [14:11:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.23, 7.29, 7.75 [14:11:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.35, 7.38, 7.22 [14:11:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.45, 7.22, 7.37 [14:12:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.91, 3.60 [14:13:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.06, 7.96, 7.63 [14:14:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.95, 3.66 [14:15:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.38, 7.34, 7.68 [14:17:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.86, 8.21, 7.95 [14:17:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.36, 7.41, 7.52 [14:17:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.87, 7.75, 7.69 [14:19:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.24, 8.11, 7.73 [14:19:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.93, 8.75, 8.04 [14:20:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.35, 4.50, 3.97 [14:23:15] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:23:20] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:23:31] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:23:37] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:24:07] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:24:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.43, 3.71, 3.79 [14:25:12] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:25:31] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:27:04] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.04, 22.90, 20.30 [14:27:32] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:28:58] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.32, 22.05, 20.29 [14:29:28] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.261 second response time [14:29:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:30:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.75, 3.74 [14:30:33] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 7.998 second response time [14:31:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:31:26] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.276 second response time [14:31:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.83, 7.49, 7.94 [14:31:42] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.323 second response time [14:31:52] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.310 second response time [14:31:53] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.308 second response time [14:32:48] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.04, 20.35, 19.97 [14:33:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.04, 7.75, 7.99 [14:35:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.83, 7.38, 7.82 [14:36:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.89, 3.75, 3.80 [14:38:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.49, 4.15, 3.94 [14:41:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.05, 7.92, 7.86 [14:42:43] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [14:43:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:44:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.36, 20.48, 20.05 [14:44:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.36, 3.92, 3.95 [14:45:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:46:13] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26697 MB (5% inode=97%); [14:46:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.87, 19.52, 19.76 [14:46:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.85, 4.01, 3.98 [14:46:36] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:48:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.90, 3.99, 3.98 [14:50:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:51:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [14:51:38] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:51:39] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:51:39] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:52:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.52, 20.35, 20.02 [14:52:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.54, 4.19, 4.05 [14:52:37] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:52:42] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:53:46] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.421 second response time [14:54:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.82, 19.94, 19.89 [14:54:44] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.467 second response time [14:54:50] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 9.239 second response time [14:55:43] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.331 second response time [14:55:48] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.602 second response time [14:56:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:57:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.38, 7.60, 8.00 [14:58:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.11, 3.82, 3.96 [14:59:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.01, 7.20, 7.87 [15:03:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.47, 7.23, 7.91 [15:03:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.24, 7.49, 7.79 [15:04:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.66, 3.84, 3.84 [15:05:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.95, 6.97, 7.57 [15:05:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.74, 6.64, 7.69 [15:06:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.88, 3.86 [15:07:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.08, 7.18, 8.00 [15:07:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.59, 6.75, 7.23 [15:07:41] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.34, 7.58, 7.90 [15:08:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9GSc [15:08:23] [02miraheze/puppet] 07paladox 03d553906 - Update nutcracker.pp [15:09:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.94, 7.73, 7.92 [15:10:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.21, 4.03, 3.91 [15:11:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.82, 7.45, 7.68 [15:11:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.81, 7.79, 7.68 [15:11:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.06, 7.89, 7.96 [15:12:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.32, 3.84, 3.86 [15:13:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.59, 7.70, 7.94 [15:13:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.64, 7.45, 7.55 [15:15:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.51, 7.66, 7.60 [15:16:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.83, 4.08, 3.94 [15:17:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.40, 7.88, 7.99 [15:17:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.23, 7.84, 7.87 [15:17:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.90, 7.94, 7.69 [15:17:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.44, 7.88, 7.97 [15:18:32] Paladox: I thought we got rid of nutcracker? [15:18:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.90, 3.57, 3.77 [15:19:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.52, 8.40, 8.16 [15:19:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.61, 8.33, 8.03 [15:19:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.73, 7.70, 7.63 [15:19:19] We did because I couldn’t get it to work with memcached. I figured out why it didn’t work and got it working with the new cluster. [15:19:27] JohnLewis: ^ [15:20:01] Ah, I thought we got rid of it as it was not maintained anymore? [15:20:19] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.18, 20.80, 19.46 [15:20:29] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:20:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.17, 3.65, 3.76 [15:21:01] Essentially the MemcachedClient class doesn’t work with it but the pecl class does [15:21:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.31, 8.52, 7.95 [15:21:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [15:21:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:21:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.03, 7.38, 7.34 [15:21:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.79, 8.21, 8.03 [15:22:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.01, 20.59, 19.56 [15:22:25] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.018 second response time [15:22:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.68, 3.77 [15:23:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.81, 7.32, 7.71 [15:23:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:23:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:23:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 3.82, 6.22, 6.94 [15:25:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.89, 7.79, 7.83 [15:26:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.29, 20.23, 19.70 [15:27:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.38, 7.46, 7.28 [15:27:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.71, 7.91, 7.98 [15:29:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.63, 8.59, 8.21 [15:31:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.32, 7.36, 7.95 [15:31:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.65, 7.99, 7.93 [15:32:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.48, 4.21, 3.91 [15:36:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.69, 3.77 [15:37:06] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.73, 7.90, 7.81 [15:37:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.37, 7.47, 7.57 [15:39:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.00, 7.84, 7.82 [15:40:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.38, 20.37, 19.54 [15:40:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.69, 3.73 [15:41:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [15:43:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:44:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.96, 3.85 [15:45:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.21, 7.54, 7.91 [15:46:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 4.10, 3.91 [15:47:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.40, 8.09, 8.08 [15:49:32] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:49:46] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:50:26] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:50:54] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:51:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.25, 7.27, 7.79 [15:52:15] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.49, 19.61, 19.79 [15:52:30] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.560 second response time [15:52:52] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.186 second response time [15:53:34] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.051 second response time [15:53:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.69, 7.47, 7.45 [15:53:53] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.248 second response time [15:54:20] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 6.081 second response time [15:55:14] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.06, 7.62, 7.78 [15:55:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.99, 7.81, 7.60 [15:56:18] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb [15:56:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.97, 3.93 [15:57:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.12, 7.86, 7.64 [16:00:11] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:02:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.28, 4.55, 4.13 [16:06:57] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.37, 20.77, 20.36 [16:07:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.54, 7.03, 7.93 [16:07:58] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [16:09:54] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:13:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.36, 8.01, 8.00 [16:14:34] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.94, 20.04, 20.19 [16:15:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:17:52] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [16:19:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [16:20:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.16, 21.57, 20.74 [16:21:15] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:21:17] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:21:38] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:22:16] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:22:57] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:23:03] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:19] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.738 second response time [16:23:24] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.799 second response time [16:23:34] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:24:12] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.327 second response time [16:24:55] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.397 second response time [16:25:04] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 2.230 second response time [16:25:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [16:27:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.07, 7.50, 7.97 [16:27:10] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:27:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:28:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.06, 19.38, 20.10 [16:29:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.42, 7.90, 8.06 [16:29:08] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 0.798 second response time [16:32:54] dmehus: do you think before typing [16:33:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.44, 6.83, 7.66 [16:35:12] dmehus: look at the bottom of https://wiki.tnonline.net/w/Blog/NFS_on_Btrfs?useskin=vector [16:35:13] [url] Blog/NFS on Btrfs - Forza's ramblings | wiki.tnonline.net [16:35:18] [02mw-config] 07Universal-Omega commented on pull request 03#4368: Sitenotice for feedback on Category Browser removal - 13https://git.io/J9GQ3 [16:35:39] Below the category list they are there again [16:36:15] I can't find a decent example on that site of something in a subcategory [16:36:47] RhinosF1, Agent: see my review on PR. [16:37:06] Are you sure that isn't causing fatals for wikis using category browser? [16:37:07] CosmicAlpha: hmm, think or know [16:37:16] Beyond no one complaining no [16:37:51] I'm 70% sure. Category browser was only added to ManageWiki recently so many might not use it. I know it happened when testing 2 sitenotices once before. [16:38:10] If they use the same function name. [16:38:17] it does cause a fatal [16:38:28] Hmm [16:38:30] CosmicAlpha: please revert [16:38:33] https://cpw.miraheze.org/wiki/Main_Page [16:38:34] I will fix. [16:38:38] Not revert. [16:38:52] thanks [16:39:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9GQW [16:39:45] [02miraheze/mw-config] 07Universal-Omega 032efe399 - Fix SiteNotice [16:40:56] miraheze/mw-config - Universal-Omega the build passed. [16:41:03] Agent: ^ [16:41:22] CosmicAlpha: thank you, you are a lifesaver :) [16:41:50] !log [@mw11] starting deploy of {'config': True} to ovlon [16:41:54] You're welcome! [16:42:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:42:11] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 21s [16:42:40] Agent: works now [16:42:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:43:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.97, 7.51, 7.53 [16:45:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [16:45:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [16:47:01] !log [@test101] starting deploy of {'config': True} to skip [16:47:02] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [16:47:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.72, 7.38, 7.50 [16:47:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:48:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:49:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.03, 8.69, 7.96 [16:50:35] [02mw-config] 07AgentIsai opened pull request 03#4369: Fix sitenotice; escape $ - 13https://git.io/J9GQo [16:50:42] CosmicAlpha: can you merge? ^ [16:50:59] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.93, 20.60, 19.88 [16:51:02] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:03] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:14] !log [@mwtask111] starting deploy of {'config': True} to scsvg [16:51:15] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:20] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 6s [16:51:32] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:39] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:40] miraheze/mw-config - AgentIsai the build passed. [16:51:55] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:52:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:52:52] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:52:53] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.15, 20.22, 19.81 [16:53:06] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.345 second response time [16:53:15] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.484 second response time [16:53:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:54:17] !log [@test3] starting deploy of {'config': True} to skip [16:54:18] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 1s [16:55:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:56:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:56:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.17, 3.40, 3.96 [16:59:06] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.425 second response time [16:59:22] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.774 second response time [16:59:47] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.646 second response time [17:00:05] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.045 second response time [17:00:28] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 6.968 second response time [17:01:26] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:01:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:01:42] PROBLEM - ns2 GDNSD Datacenters on ns2 is UNKNOWN: NRPE: Unable to read output [17:02:23] PROBLEM - ns2 Auth DNS on ns2 is CRITICAL: CRITICAL - Plugin timed out while executing system call [17:02:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.81, 21.74, 20.58 [17:02:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.54, 3.77, 3.92 [17:03:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:04:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.57, 3.68, 3.87 [17:08:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.42, 19.90, 20.15 [17:08:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.96, 4.12, 3.99 [17:09:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [17:09:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb [17:11:00] [02mw-config] 07Universal-Omega closed pull request 03#4369: Fix sitenotice; escape $ - 13https://git.io/J9GQo [17:11:01] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9G7O [17:11:03] [02miraheze/mw-config] 07AgentIsai 03139bd69 - $ -> $ (#4369) [17:11:30] !log [@mw11] starting deploy of {'config': True} to ovlon [17:11:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:12:02] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 31s [17:12:07] miraheze/mw-config - Universal-Omega the build passed. [17:12:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:15:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:15:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:17:20] !log [@test101] starting deploy of {'config': True} to skip [17:17:21] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [17:17:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:17:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:18:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.07, 3.67, 3.93 [17:21:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:21:32] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:21:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb [17:21:51] !log [@mwtask111] starting deploy of {'config': True} to scsvg [17:22:02] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 10s [17:22:21] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:23:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:23:36] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.842 second response time [17:24:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.61, 3.83, 3.90 [17:24:59] !log [@test3] starting deploy of {'config': True} to skip [17:25:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:25:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:26:34] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 4.181 second response time [17:28:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.78, 3.86 [17:29:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:34:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.21, 4.03, 3.92 [17:35:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [17:36:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.75, 3.83 [17:37:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:37:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.05, 6.75, 7.82 [17:39:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.38, 7.21, 7.85 [17:40:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.65, 3.95, 3.87 [17:44:18] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 149.56.141.75/cpweb [17:45:16] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:46:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [17:46:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.95, 3.92 [17:46:53] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:46:56] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:47:07] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:47:10] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:48:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.67, 4.01, 3.96 [17:50:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.88, 3.91 [17:50:56] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.99, 19.95, 19.60 [17:52:06] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:52:08] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:52:12] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:52:50] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.80, 20.07, 19.70 [17:54:07] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.288 second response time [17:54:09] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:54:10] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.578 second response time [17:54:15] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.150 second response time [17:55:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.62, 7.24, 7.94 [17:55:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.32, 6.43, 7.78 [17:55:57] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 5.975 second response time [17:56:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.75, 3.83 [17:57:12] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.331 second response time [17:57:18] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.191 second response time [17:57:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.77, 7.20, 7.89 [17:57:35] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 6.698 second response time [17:57:40] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.941 second response time [17:58:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.15, 3.54, 3.75 [17:59:15] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.38, 8.22, 8.09 [17:59:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:00:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.94, 3.88 [18:02:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.87, 3.86 [18:03:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.45, 7.64, 7.93 [18:04:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.85, 4.08, 3.92 [18:05:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.88, 7.96, 8.00 [18:06:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.70, 20.43, 20.06 [18:07:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.85, 7.93, 7.99 [18:08:13] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:08:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.89, 3.90 [18:08:52] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:09:13] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:09:15] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:09:33] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.41, 20.17, 18.86 [18:09:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.44, 8.83, 8.31 [18:09:43] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:10:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.32, 18.98, 19.61 [18:10:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 4.04, 3.95 [18:10:43] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [18:11:02] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [18:11:37] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.014 second response time [18:12:20] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 1.867 second response time [18:12:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.50, 3.49, 3.76 [18:12:35] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:12:52] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.711 second response time [18:13:02] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:17] PROBLEM - mem2 SSH on mem2 is CRITICAL: Server answer [18:13:18] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:21] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.546 second response time [18:13:22] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.022 second response time [18:13:27] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:33] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.49, 20.13, 19.18 [18:13:41] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:46] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:51] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:14:03] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:14:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 3.87, 3.87 [18:15:01] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:15:06] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:15:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.68, 7.41, 7.99 [18:15:17] RECOVERY - mem2 SSH on mem2 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [18:15:48] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.302 second response time [18:15:53] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 7.240 second response time [18:16:04] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.442 second response time [18:17:04] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.118 second response time [18:17:10] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.928 second response time [18:18:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.97, 3.93 [18:19:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.21, 7.60, 7.91 [18:19:29] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.222 second response time [18:19:43] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 4.603 second response time [18:19:56] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.391 second response time [18:20:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.17, 3.96, 3.92 [18:20:46] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.821 second response time [18:21:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.81, 7.47, 7.84 [18:21:23] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:38] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:40] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:42] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:50] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:22:01] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:22:02] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:22:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.75, 3.73, 3.86 [18:23:03] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:23:10] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:15] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:23:22] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 0.479 second response time [18:23:35] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.470 second response time [18:23:35] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.354 second response time [18:23:39] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.324 second response time [18:23:46] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:23:55] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.487 second response time [18:24:05] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:24:18] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:24:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.62, 4.30, 4.06 [18:25:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.64, 8.18, 8.01 [18:25:17] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 9.500 second response time [18:25:18] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.703 second response time [18:25:22] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.300 second response time [18:25:42] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.960 second response time [18:26:09] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 6.631 second response time [18:26:13] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.681 second response time [18:26:14] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.207 second response time [18:26:19] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.581 second response time [18:27:50] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:27:56] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:28:00] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:28:04] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:28:18] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:28:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.86, 3.83, 3.95 [18:29:17] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.881 second response time [18:29:44] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:29:58] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.285 second response time [18:29:58] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 8.832 second response time [18:30:04] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.276 second response time [18:30:10] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.360 second response time [18:30:32] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:31:40] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.361 second response time [18:32:32] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 3.306 second response time [18:34:18] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:34:26] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:34:30] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:34:35] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:35:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.36, 6.62, 7.91 [18:36:22] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.988 second response time [18:36:35] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 9.717 second response time [18:36:35] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:35] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.558 second response time [18:36:40] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.376 second response time [18:36:44] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:36:49] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.898 second response time [18:36:57] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:37:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.38, 7.37, 8.00 [18:37:40] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.27, 1.94, 1.28 [18:37:52] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:38:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.46, 3.80, 3.80 [18:38:37] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 2.185 second response time [18:38:45] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.256 second response time [18:38:58] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.600 second response time [18:39:40] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.51, 1.44, 1.18 [18:40:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.05, 3.58, 3.72 [18:43:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [18:44:45] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:45:06] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:45:07] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:45:09] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:45:12] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:45:12] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:45:14] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:45:20] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:45:24] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:45:53] RhinosF1, yes, I think before I am typing. and can you confirm if the link you referred to is referring to the bottom or top portion of the categories, or both? [18:45:59] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:46:20] lol, what does nutcracker do? [18:46:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.77, 3.74 [18:46:41] dmehus I'm active on IRC now, in case you haven't read the message. :) [18:47:30] paladox, yeah, I'm fine if you guys want to remove the MirahezeBot's ability to use the +b function in -sre. I did whitelist miraheze-github_ last night, though, so should be resolved, but yeah, I agree its utility is limited [18:47:52] dmehus: the bottom [18:48:00] The list that's one per line not across [18:48:29] And nutcracker is a proxy for memcache so you don't have to reconnect as often [18:48:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.72, 3.38, 3.61 [18:49:07] It should be a list but I can't find any useful example of how it works [18:49:30] RhinosF1, ack, that's fine then [18:49:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.35, 7.34, 7.95 [18:50:34] dmehus: but no I don't have a good example of it [18:50:54] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.023 second response time [18:50:58] RhinosF1, ack re: nutcracker. I don't really understand the specifics, but I'll abstain from that discussion among SRE to decide its utility [18:51:22] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.210 second response time [18:51:23] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.211 second response time [18:51:35] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.737 second response time [18:51:37] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.620 second response time [18:51:42] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.807 second response time [18:51:42] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 6.755 second response time [18:51:42] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.213 second response time [18:51:44] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.86, 8.16, 8.17 [18:52:08] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.015 second response time [18:52:14] dmehus: it looks like it'll be migrated to an extension [18:52:29] So we can easily swap it out [18:53:27] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:53:38] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 3.566 second response time [18:53:40] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:54:08] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:54:17] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:54:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.81, 3.71 [18:54:34] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:55:23] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.300 second response time [18:55:38] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.010 second response time [18:56:07] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.329 second response time [18:56:13] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.036 second response time [18:56:31] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23541 bytes in 2.801 second response time [18:56:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.10, 3.74, 3.72 [18:58:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.04, 4.11, 3.85 [19:00:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.82, 3.77 [19:01:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:01:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:03:12] [02ssl] 07Reception123 closed pull request 03#467: Remove wiki.raghuveer.net - 13https://git.io/J9GL0 [19:03:13] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/J9GFR [19:03:15] [02miraheze/ssl] 07Universal-Omega 03fe62ca9 - Remove wiki.raghuveer.net (#467) [19:03:21] [02ssl] 07Reception123 closed pull request 03#465: Remove wiki.helioss.co - 13https://git.io/J9GIh [19:03:23] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/J9GFE [19:03:24] [02miraheze/ssl] 07Universal-Omega 03e8ae5a5 - Remove wiki.helioss.co (#465) [19:03:26] [02ssl] 07Reception123 closed pull request 03#466: Remove thesimswiki.com redirect - 13https://git.io/J9GLZ [19:03:27] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9GFu [19:03:29] [02miraheze/ssl] 07Universal-Omega 037c088e2 - Remove thesimswiki.com redirect (#466) [19:03:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.71, 7.52, 7.97 [19:05:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.33, 7.31, 7.94 [19:07:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.78, 8.50, 8.31 [19:07:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.24, 8.02, 8.04 [19:08:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [19:08:46] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±17] 13https://git.io/J9GFo [19:08:47] [02miraheze/puppet] 07JohnFLewis 03c4f5324 - users: move universalomega to mediawiki-roots [19:09:06] Reception123: ^ cc [19:09:15] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:09:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.14, 7.44, 7.81 [19:09:41] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.23, 6.82, 7.73 [19:09:57] JohnLewis: thanks! [19:10:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:11:14] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.428 second response time [19:12:59] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:13:08] PROBLEM - mwtask1 Puppet on mwtask1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:13:25] PROBLEM - mw102 Puppet on mw102 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:13:26] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:13:29] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:13:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.32, 7.93, 7.92 [19:13:44] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.89, 8.18, 8.05 [19:15:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:16:10] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:16:11] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:16:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:17:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.38, 7.38, 7.96 [19:18:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.47, 3.78, 3.58 [19:19:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.09, 7.99, 8.11 [19:19:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.56, 7.08, 7.97 [19:19:28] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:19:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.75, 7.25, 7.70 [19:19:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.81, 7.66, 7.96 [19:19:50] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:19:55] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:20:20] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:20:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.01, 3.56, 3.53 [19:21:26] PROBLEM - mw112 Puppet on mw112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:21:31] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.911 second response time [19:21:31] PROBLEM - mw13 Puppet on mw13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:21:50] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:11] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:22:20] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:22:25] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:22:29] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:22:40] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:23:18] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:23:55] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 8.947 second response time [19:23:59] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:24:01] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:24:06] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.916 second response time [19:24:15] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.106 second response time [19:24:27] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.552 second response time [19:24:31] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.502 second response time [19:24:37] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 9.906 second response time [19:24:47] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.856 second response time [19:24:50] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.71, 3.10, 3.35 [19:25:34] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.36, 7.76, 7.87 [19:25:53] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:26:35] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.76, 7.30, 7.61 [19:26:46] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:27:51] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb [19:27:54] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.300 second response time [19:28:03] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.252 second response time [19:28:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.46, 20.73, 20.10 [19:29:02] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.619 second response time [19:29:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:29:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.49, 8.06, 7.69 [19:30:02] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:30:50] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.92, 3.58, 3.47 [19:31:53] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:32:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.22, 4.05, 3.64 [19:34:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.15, 3.72, 3.58 [19:35:24] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:36:16] PROBLEM - mw111 Puppet on mw111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:40:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb [19:41:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:42:07] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:44:07] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.49, 19.79, 20.27 [19:44:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.57, 3.18, 3.38 [19:45:32] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:45:54] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:45:59] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb [19:46:31] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:47:33] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.443 second response time [19:47:53] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.078 second response time [19:48:12] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb [19:48:36] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 9.377 second response time [19:50:50] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.72, 4.17, 3.69 [19:51:02] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.50, 19.97, 18.82 [19:51:26] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.28, 4.95, 4.01 [19:51:55] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.19, 21.75, 20.79 [19:52:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.05, 3.79, 3.63 [19:52:56] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 20.35, 20.02, 18.98 [19:53:01] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 0.83, 2.13, 1.33 [19:53:26] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.83, 4.60, 3.98 [19:53:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.82, 21.75, 20.91 [19:55:00] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.68, 1.70, 1.28 [19:56:33] RhinosF1, ah, yeah that's not so bad then [19:57:10] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:57:21] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:30] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:57:35] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:58:31] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:58:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.80, 3.67 [20:00:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.75, 3.68 [20:02:37] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.390 second response time [20:03:25] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.212 second response time [20:03:33] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 1.056 second response time [20:03:36] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.321 second response time [20:03:43] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.009 second response time [20:04:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 4.13, 3.84 [20:05:38] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.30, 19.69, 20.22 [20:07:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:09:22] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:13:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:13:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [20:15:40] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:15:50] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:15:51] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:55] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:15:58] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:16:59] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:17:30] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:17:36] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:17:57] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 9.921 second response time [20:17:58] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.080 second response time [20:21:14] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.49, 21.48, 20.57 [20:21:52] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.00, 1.78, 1.17 [20:22:20] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:22:21] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:22] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:22:24] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:22:58] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:23:51] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.41, 1.57, 1.17 [20:24:24] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.478 second response time [20:24:26] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 9.319 second response time [20:24:28] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.032 second response time [20:24:30] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.010 second response time [20:24:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.84, 3.57, 3.91 [20:24:58] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.619 second response time [20:25:09] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.51, 22.37, 21.07 [20:25:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.53, 6.94, 7.81 [20:27:06] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.40, 22.12, 21.11 [20:27:17] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.806 second response time [20:27:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.85, 7.98, 8.08 [20:27:56] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.335 second response time [20:28:11] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23541 bytes in 2.327 second response time [20:28:33] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.466 second response time [20:28:35] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.011 second response time [20:28:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.44, 3.61, 3.81 [20:29:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:29:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:29:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.14, 7.68, 7.96 [20:30:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.90, 7.19, 7.82 [20:31:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.43, 7.50, 7.96 [20:32:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.22, 3.64, 3.79 [20:33:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.22, 7.78, 8.00 [20:34:56] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.31, 19.69, 20.31 [20:35:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.77, 6.85, 7.81 [20:35:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.12, 7.12, 7.73 [20:37:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.15, 7.71, 8.00 [20:37:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.80, 8.19, 8.05 [20:40:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.31, 7.84, 7.69 [20:41:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.33, 7.52, 7.54 [20:42:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.03, 7.48, 7.57 [20:42:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.61, 4.20, 3.92 [20:43:33] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.03, 7.09, 7.37 [20:44:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.82, 3.62, 3.74 [20:45:32] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.17, 8.47, 7.86 [20:47:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:47:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:47:55] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:48:11] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:49:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.54, 7.83, 7.73 [20:49:36] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.97, 20.46, 20.07 [20:49:56] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.374 second response time [20:50:11] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.260 second response time [20:50:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.86, 8.43, 7.93 [20:50:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.21, 4.02, 3.88 [20:51:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.12, 7.84, 7.74 [20:51:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.55, 20.07, 19.98 [20:52:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.92, 3.86 [20:54:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/J9GAp [20:54:35] [02miraheze/puppet] 07paladox 0340858ee - matomo/icinga: adjust php config [20:54:36] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [20:54:38] [02puppet] 07paladox opened pull request 03#2288: matomo/icinga: adjust php config - 13https://git.io/J9GAh [20:54:40] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:54:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.90, 3.85 [20:55:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:55:26] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:56:03] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:56:23] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:56:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/J9Gxv [20:56:36] [02miraheze/puppet] 07paladox 03f2f65e8 - Update mon111.yaml [20:56:38] [02puppet] 07paladox synchronize pull request 03#2288: matomo/icinga: adjust php config - 13https://git.io/J9GAh [20:56:45] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 9.697 second response time [20:56:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.70, 3.77 [20:57:28] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.40, 21.64, 20.64 [20:59:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:59:26] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:00:14] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:15] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:00:24] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:00:39] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:00:40] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:00:52] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:00:54] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:01:02] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:01:04] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:01:06] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:01:07] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:01:10] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:01:14] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:01:16] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:01:28] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.12, 20.04, 20.26 [21:01:29] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:01:29] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:01:47] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:01:49] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:02:01] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:02:01] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:02:18] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:02:25] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.807 second response time [21:03:04] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.006 second response time [21:03:14] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw11 mw13 [21:03:25] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.006 second response time [21:03:27] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.006 second response time [21:03:43] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.030 second response time [21:03:48] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.335 second response time [21:04:00] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.191 second response time [21:04:05] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.424 second response time [21:04:25] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.452 second response time [21:04:35] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.335 second response time [21:04:36] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.010 second response time [21:04:41] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.322 second response time [21:05:03] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 1.436 second response time [21:05:07] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 1.753 second response time [21:05:09] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.824 second response time [21:05:09] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.847 second response time [21:05:12] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.004 second response time [21:05:14] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.218 second response time [21:05:46] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw10 [21:06:14] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw11 mw13 [21:06:21] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 0.250 second response time [21:07:05] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.871 second response time [21:07:14] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 18 backends are healthy [21:07:45] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 18 backends are healthy [21:08:13] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [21:08:57] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.01, 2.17, 1.48 [21:09:20] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.177 second response time [21:09:28] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:09:30] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:09:32] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:10:52] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.82, 1.68, 1.38 [21:11:15] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:11:25] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:11:27] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.879 second response time [21:11:29] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:11:31] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 4.775 second response time [21:11:33] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.451 second response time [21:11:39] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:11:47] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:12:48] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 5.581 second response time [21:12:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.34, 2.99, 3.36 [21:12:52] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:12:52] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:13:10] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.505 second response time [21:13:20] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.317 second response time [21:13:28] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.47, 19.94, 19.73 [21:13:30] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.027 second response time [21:13:42] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:13:45] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:14:02] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:14:09] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:14:43] CosmicAlpha: 'Infrastructure Engineer' doesn't exist as a role [21:15:08] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.329 second response time [21:15:15] JohnLewis: Oh, I will revert that change. [21:15:16] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:15:21] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:15:36] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.006 second response time [21:15:37] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.007 second response time [21:15:54] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:15:57] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:15:57] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.568 second response time [21:16:06] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.335 second response time [21:17:28] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.76, 19.55, 19.63 [21:18:59] JohnLewis: Fixed, sorry. [21:19:22] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.351 second response time [21:19:22] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.349 second response time [21:19:41] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.008 second response time [21:19:44] thank you :) [21:19:58] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 2.241 second response time [21:20:01] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.025 second response time [21:20:24] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.576 second response time [21:21:05] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.030 second response time [21:21:18] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.741 second response time [21:21:28] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.11, 20.47, 20.00 [21:21:36] JohnLewis: You're welcome. Also, it seems puppet is failing on all MediaWiki servers following the change to add me to mediawiki-roots. "Evaluation Error: Error while evaluating a Function Call, has_key(): expects the first argument to be a hash, got "" which is of type String (file: /etc/puppetlabs/puppet/environments/production/modules/users/manifests/hashuser.pp, line: 8, column: 8)" - do you happen to have any ideas why? [21:21:39] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.78, 3.82, 3.57 [21:21:54] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:06] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:13] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.625 second response time [21:22:17] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 5.344 second response time [21:22:17] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:28] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:22:35] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:37] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:23:35] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:23:39] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:23:47] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:23:47] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:23:51] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.519 second response time [21:24:01] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:24:12] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.442 second response time [21:24:12] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.020 second response time [21:24:15] CosmicAlpha: typo :) [21:24:21] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:24:25] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:24:31] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 5.802 second response time [21:24:32] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.387 second response time [21:24:37] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.072 second response time [21:24:45] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9GxH [21:24:46] [02miraheze/puppet] 07JohnFLewis 035041ee7 - s in UO [21:25:12] Oh, thanks! [21:25:30] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.007 second response time [21:25:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.55, 7.39, 7.91 [21:25:34] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.754 second response time [21:27:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:29:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:31:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:31:28] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.14, 22.62, 21.13 [21:31:31] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:32:02] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.245 second response time [21:32:10] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 2.808 second response time [21:32:10] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 2.766 second response time [21:32:34] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 0.359 second response time [21:32:38] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.013 second response time [21:32:50] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:33:07] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.18, 2.99, 3.33 [21:33:28] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.44, 21.63, 20.95 [21:33:31] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.82, 5.41, 6.70 [21:33:33] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:33:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.23, 6.25, 7.83 [21:34:57] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:35:05] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:35:09] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:35:13] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:35:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.17, 7.07, 7.92 [21:35:52] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:35:58] RECOVERY - mw111 Puppet on mw111 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [21:36:56] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.314 second response time [21:36:57] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.56, 3.77, 3.57 [21:37:04] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 0.837 second response time [21:37:06] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.628 second response time [21:37:11] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.192 second response time [21:37:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:37:32] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.453 second response time [21:38:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.72, 3.58 [21:39:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.10, 6.85, 6.86 [21:41:16] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:41:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:42:00] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [21:42:18] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [21:42:33] RECOVERY - mw102 Puppet on mw102 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:42:49] RECOVERY - mwtask1 Puppet on mwtask1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:43:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 149.56.141.75/cpweb [21:43:45] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:45:22] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:45:28] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.15, 19.91, 20.31 [21:45:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.77, 7.84, 7.45 [21:45:48] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.632 second response time [21:47:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:47:21] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.325 second response time [21:47:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:47:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.81, 7.43, 7.91 [21:48:21] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [21:48:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.56, 3.53 [21:50:01] RECOVERY - mw112 Puppet on mw112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:50:45] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [21:50:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.65, 3.57 [21:51:08] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [21:51:20] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:51:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [21:51:40] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:51:43] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:46] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:53] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [21:51:55] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:56] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:51:57] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:59] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:52:07] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:52:16] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:53:14] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.056 second response time [21:53:26] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 6.543 second response time [21:53:39] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:53:41] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.028 second response time [21:53:46] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.924 second response time [21:53:51] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.330 second response time [21:53:57] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.580 second response time [21:53:58] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 3.131 second response time [21:54:02] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.749 second response time [21:54:05] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.805 second response time [21:54:13] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.517 second response time [21:54:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.03, 4.29, 3.83 [21:55:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:55:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:56:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.36, 3.74, 3.67 [21:57:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.08, 7.46, 7.74 [21:58:42] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [21:58:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.76, 3.68 [21:59:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.78, 7.25, 7.96 [21:59:28] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.79, 20.11, 19.98 [22:00:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.14, 7.34, 7.89 [22:01:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.57, 7.70, 8.04 [22:01:28] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.50, 19.90, 19.92 [22:02:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.22, 3.45, 3.57 [22:03:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.92, 7.06, 7.76 [22:04:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.46, 3.77, 3.67 [22:05:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.16, 7.47, 7.73 [22:06:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.37, 3.75, 3.68 [22:07:31] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.39, 6.05, 6.59 [22:09:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.40, 7.24, 7.56 [22:09:45] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.59, 7.55, 7.64 [22:10:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.91, 7.68, 7.64 [22:11:57] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:12:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.72, 4.26, 3.85 [22:13:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.22, 7.73, 7.77 [22:13:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.37, 7.36, 7.03 [22:13:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.62, 7.82, 7.76 [22:13:52] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:14:17] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.96, 3.79 [22:14:56] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:16:18] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.503 second response time [22:16:54] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.717 second response time [22:17:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.10, 7.46, 7.11 [22:19:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.00, 6.88, 6.94 [22:25:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.49, 7.60, 7.43 [22:25:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.85, 8.08, 7.61 [22:27:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.57, 7.19, 7.27 [22:28:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.66, 3.60 [22:29:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.47, 7.89, 7.51 [22:29:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.32, 7.60, 7.27 [22:30:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.24, 3.13, 3.41 [22:31:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.64, 7.10, 7.12 [22:33:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.11, 7.70, 7.52 [22:34:01] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [22:35:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [22:35:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.75, 8.06, 7.68 [22:35:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.97, 7.78, 7.37 [22:37:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:37:51] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:39:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.09, 7.76, 7.48 [22:40:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.09, 3.13, 3.35 [22:41:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.32, 7.97, 7.59 [22:43:23] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:43:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:43:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [22:43:47] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:44:07] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:44:10] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:44:20] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:45:22] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 3.191 second response time [22:45:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.27, 7.59, 7.81 [22:45:43] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 1.427 second response time [22:46:02] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 0.899 second response time [22:46:10] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 0.521 second response time [22:46:17] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.320 second response time [22:47:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:49:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.63, 7.57, 7.66 [22:50:35] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:50:58] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:51:03] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:19] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:26] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:51:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.48, 7.92, 7.83 [22:51:29] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.05, 7.21, 7.92 [22:51:46] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:52:05] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:52:16] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:52:32] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:52:33] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:53:02] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 5.494 second response time [22:53:02] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.000 second response time [22:53:22] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.925 second response time [22:53:33] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.873 second response time [22:53:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.36, 7.48, 7.92 [22:53:45] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.654 second response time [22:53:49] RhinosF1, I just tried to revert your rollback due to an edit conflict, but invariably my attempted revert created another edit conflict [22:53:57] so it looks like you self-reverted [22:54:01] Yes [22:54:08] ok [22:54:08] I'm not sure how I managed that [22:54:19] Meta is quite slow [22:54:21] so I can see it [22:54:38] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.221 second response time [22:54:39] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.060 second response time [22:54:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.55, 3.43 [22:55:12] In any case, I'll reply to YellowFrogger [22:55:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.14, 7.21, 7.76 [22:56:09] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 2.712 second response time [22:56:19] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.567 second response time [22:56:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.28, 3.16, 3.30 [22:57:20] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:57:28] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.67, 7.79, 7.70 [22:57:45] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:58:00] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:59:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.13, 7.57, 7.76 [23:00:01] !log [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg [23:00:02] !log [@mw11] starting deploy of {'l10nupdate': True} to ovlon [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:03] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.79, 3.54 [23:01:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:12] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.749 second response time [23:01:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:01:23] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.498 second response time [23:01:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:01:37] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 2.015 second response time [23:01:51] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.102 second response time [23:02:05] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.319 second response time [23:02:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.90, 3.56, 3.49 [23:03:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.96, 6.85, 7.27 [23:04:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.04, 3.26, 3.37 [23:05:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.73, 7.43, 7.42 [23:06:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.80, 7.62, 7.98 [23:07:46] !log [@test101] finished deploy of {'l10nupdate': True} to skip - FAIL: [35072, 0, 0] in 464s [23:07:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:08:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.43, 7.83, 8.01 [23:08:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.53, 3.50 [23:11:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:13:00] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:13:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [23:13:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.57, 19.81, 19.36 [23:14:01] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:14] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:14:58] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.784 second response time [23:15:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:15:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:15:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.73, 7.68, 7.78 [23:15:57] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.672 second response time [23:16:11] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 0.921 second response time [23:17:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.66, 8.09, 7.92 [23:17:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.27, 20.12, 19.61 [23:18:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.83, 3.60 [23:19:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.04, 7.64, 7.77 [23:20:19] !log [@mwtask111] finished deploy of {'l10nupdate': True} to scsvg - SUCCESS in 1218s [23:20:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.35, 3.57, 3.52 [23:20:55] !log [universalomega@test101] starting deploy of {'l10nupdate': True} to skip [23:21:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:21:28] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:21:28] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.40, 21.31, 20.20 [23:21:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.44, 7.96, 7.87 [23:22:11] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:22:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [23:22:19] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:22:21] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:22:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.22, 4.26, 3.79 [23:23:27] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.572 second response time [23:23:28] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.48, 20.32, 19.98 [23:24:07] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.020 second response time [23:24:15] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.047 second response time [23:24:18] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 0.387 second response time [23:26:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:27:23] dmehus: https://github.com/miraheze/MirahezeMagic/pull/311#pullrequestreview-847319019 [23:27:24] [url] Re-add global interface messages for default user groups by dmehus · Pull Request #311 · miraheze/MirahezeMagic · GitHub | github.com [23:27:30] !log [universalomega@test101] finished deploy of {'l10nupdate': True} to skip - FAIL: [35072, 0, 0] in 395s [23:27:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:28:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.93, 7.75, 7.98 [23:28:51] How come that gets killed on test101 but not mwtask111? [23:30:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.58, 3.69, 3.74 [23:31:50] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:32:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 7.82, 8.05, 8.07 [23:32:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.77, 4.07, 3.86 [23:33:26] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [23:35:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:35:41] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:36:20] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.62, 21.10, 20.43 [23:36:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.62, 7.81, 7.98 [23:38:17] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.09, 19.66, 20.00 [23:39:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.16, 7.03, 7.86 [23:40:00] CosmicAlpha, looking [23:40:53] [02MirahezeMagic] 07dmehus commented on pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9Zed [23:40:58] CosmicAlpha, see PR [23:41:33] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.91, 7.82, 8.05 [23:42:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.21, 2.40, 1.06 [23:42:09] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.61, 22.07, 20.87 [23:42:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.70, 8.59, 8.21 [23:43:20] [02MirahezeMagic] 07Universal-Omega commented on pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9Zej [23:43:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.32, 7.39, 7.87 [23:44:01] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 0.47, 1.63, 0.93 [23:44:31] dmehus: seen [23:44:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.54, 3.52, 3.78 [23:45:15] [02MirahezeMagic] 07dmehus commented on pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9Zvf [23:45:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.07, 8.59, 8.27 [23:46:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb [23:46:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.140.43/cpweb [23:46:45] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:48:04] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.76, 21.99, 21.29 [23:48:49] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.752 second response time [23:51:08] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:27] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:28] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:39] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:45] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:51] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:52] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:51:54] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:52:00] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:52:01] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:53:07] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.814 second response time [23:53:27] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.801 second response time [23:53:28] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.776 second response time [23:53:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.76, 7.07, 7.82 [23:53:37] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.413 second response time [23:53:46] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.822 second response time [23:53:52] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.061 second response time [23:53:52] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:53:53] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 5.331 second response time [23:53:57] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.379 second response time [23:53:57] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.44, 22.59, 21.75 [23:53:59] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.718 second response time [23:54:00] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 1.936 second response time [23:54:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:54:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.43, 3.88, 3.79 [23:55:55] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.59, 22.79, 21.94 [23:56:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.17, 7.61, 7.98 [23:56:39] dmehus: see https://github.com/dmehus/MirahezeMagic/pull/1 but now I'm confused on whether we're using anchors or {{int:}} [23:56:39] https://meta.miraheze.org/wiki/int:Template: [23:56:39] [url] +Special:MyLanguage by AgentIsai · Pull Request #1 · dmehus/MirahezeMagic · GitHub | github.com [23:56:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.32, 3.66, 3.72 [23:57:16] CosmicAlpha: what do you suggest? Using {{anchor}} or {{int:}} for this? [23:57:16] https://meta.miraheze.org/wiki/Template:anchor [23:57:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.32, 7.57, 7.84 [23:58:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.36, 7.83, 8.01 [23:58:24] Agent: {{int}} I guess [23:58:24] https://meta.miraheze.org/wiki/Template:int [23:58:36] lovely [23:58:39] thank you [23:58:46] no problem