[00:00:14] !log [paladox@mw11] starting deploy of {'world': True} to all [00:00:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:00:30] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.58, 3.14 [00:00:49] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.91, 7.49, 5.69 [00:01:45] miraheze/CreateWiki - paladox the build passed. [00:02:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.42, 3.89, 3.30 [00:04:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.58, 3.26 [00:04:51] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.31, 8.45, 6.41 [00:05:13] miraheze/CreateWiki - paladox the build passed. [00:06:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.09, 4.06, 3.46 [00:06:49] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.11, 7.89, 6.49 [00:08:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.84, 3.96, 3.50 [00:08:50] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.26, 6.64, 6.20 [00:10:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.62, 4.27, 3.68 [00:12:48] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.44, 23.22, 19.26 [00:15:00] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.83, 6.33, 4.42 [00:16:58] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.31, 5.44, 4.34 [00:18:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.01, 3.73, 3.75 [00:20:05] !log [paladox@mw11] finished deploy of {'world': True} to all - SUCCESS in 1190s [00:20:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:16] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 27.68, 22.23, 19.16 [00:20:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 7.31, 4.61, 4.04 [00:20:54] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.21, 4.25, 4.15 [00:22:12] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.03, 20.04, 18.72 [00:22:48] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 14.52, 21.21, 21.54 [00:26:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.86, 3.91 [00:26:48] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 10.58, 16.78, 19.72 [00:30:53] !log remove southparkfan from board alias - T8365 [00:30:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:34:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.71, 3.82 [00:38:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.06, 3.66, 3.81 [00:50:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.76, 4.03, 3.80 [00:54:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.80, 3.68, 3.75 [01:02:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 2.97, 3.38 [01:08:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.43, 3.43 [01:10:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.84, 3.25, 3.37 [01:22:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.91, 3.64 [01:36:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.71, 3.11, 3.38 [01:52:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.92, 4.63, 3.78 [01:53:33] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.28, 1.62, 1.10 [01:55:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.71, 1.56, 1.13 [01:57:27] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.74, 1.22, 1.05 [01:58:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.87, 3.74 [02:06:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.86, 3.74 [02:08:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.66, 3.69 [02:14:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.59, 3.58 [02:16:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.40, 3.12, 3.40 [02:51:50] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.92, 3.40, 3.22 [02:53:48] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.80, 3.22, 3.18 [02:59:39] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.43, 3.27 [03:03:35] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.02, 3.27, 3.24 [03:22:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.36, 3.22 [03:24:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.57, 3.13, 3.16 [03:34:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.92, 3.37, 3.21 [03:38:50] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.34, 3.98, 3.46 [03:40:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.95, 3.52, 3.35 [03:42:46] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.66, 3.85, 3.49 [03:44:44] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.37, 3.74, 3.49 [03:50:37] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.34, 3.32, 3.38 [03:54:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.68, 3.55 [04:00:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.94, 4.03, 3.69 [04:02:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.88, 3.43, 3.50 [04:06:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.99, 3.16, 3.37 [05:22:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.30, 2.93 [05:26:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.47, 3.20, 3.01 [06:05:10] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.41, 5.72, 3.49 [06:05:14] RECOVERY - mw9 APT on mw9 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:06:38] RECOVERY - db12 APT on db12 is OK: APT OK: 8 packages available for upgrade (0 critical updates). [06:13:10] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.01, 6.58, 5.00 [06:15:16] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49073 MB (11% inode=98%); [06:19:16] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 46761 MB (10% inode=98%); [06:22:10] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 25.07, 17.89, 10.10 [06:23:00] RECOVERY - cloud5 APT on cloud5 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:28:04] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 3.06, 7.38, 7.75 [06:29:45] RECOVERY - mw13 APT on mw13 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:31:59] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 1.97, 4.40, 6.44 [06:36:19] RECOVERY - mw11 APT on mw11 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:40:43] RECOVERY - mw12 APT on mw12 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:54:45] RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:55:46] RECOVERY - mail2 APT on mail2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:57:39] RECOVERY - mw10 APT on mw10 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:58:14] RECOVERY - cloud4 APT on cloud4 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [07:00:38] RECOVERY - puppet3 APT on puppet3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [07:00:49] RECOVERY - graylog2 APT on graylog2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [07:51:16] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49269 MB (11% inode=98%); [07:54:47] [02miraheze/ssl] 07Reception123 pushed 034 commits to 03master [+11/-3/±17] 13https://git.io/JM6or [07:54:48] [02miraheze/ssl] 07Reception123 03318c548 - remove matttest.mtwiki.cf (wiki deleted) [07:54:50] [02miraheze/ssl] 07Reception123 03cd13974 - renew SSL certs [07:54:51] [02miraheze/ssl] 07Reception123 03f2eabb0 - remove wiki.digitalcandela.com cert (wiki deleted) [07:54:53] [02miraheze/ssl] 07Reception123 033a4c771 - remove wiki.thefactoryhka.com.pa cert (wiki deleted) [07:55:16] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48534 MB (10% inode=98%); [07:55:49] !log sudo apt-get upgrade on mwtask1 / test3 [07:55:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:56:04] RECOVERY - test3 APT on test3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [07:57:06] RECOVERY - mwtask1 APT on mwtask1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [07:59:13] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [08:00:19] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.20, 5.49, 4.08 [08:00:46] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [08:01:06] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [08:02:19] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.74, 5.35, 4.22 [08:02:45] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:03:05] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:13:37] RECOVERY - wiki.konjitownmc.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.konjitownmc.cf' will expire on Wed 02 Mar 2022 06:43:41 GMT +0000. [08:16:47] RECOVERY - wiki.digitalcandela.com - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:16:52] RECOVERY - commons.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'commons.polandballwiki.com' will expire on Wed 02 Mar 2022 06:44:16 GMT +0000. [08:17:32] RECOVERY - stardewmodding.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'stardewmodding.wiki' will expire on Wed 02 Mar 2022 06:46:09 GMT +0000. [08:17:39] RECOVERY - wiki.newnormal.fans - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.newnormal.fans' will expire on Wed 02 Mar 2022 06:52:15 GMT +0000. [08:18:13] RECOVERY - matttest.mtwiki.cf - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:19:43] RECOVERY - ambient.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ambient.wiki' will expire on Wed 02 Mar 2022 06:49:39 GMT +0000. [08:20:15] RECOVERY - wiki.thefactoryhka.com.pa - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:22:23] RECOVERY - equestripedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'equestripedia.org' will expire on Wed 02 Mar 2022 06:50:14 GMT +0000. [08:23:14] RECOVERY - wiki.globasa.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.globasa.net' will expire on Wed 02 Mar 2022 06:51:39 GMT +0000. [08:23:21] PROBLEM - wiki.konjitownmc.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.konjitownmc.cf' expires in 21 hours (Fri 03 Dec 2021 05:41:46 GMT +0000) [08:23:43] RECOVERY - wiki.slimeapedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.slimeapedia.org' will expire on Wed 02 Mar 2022 06:48:53 GMT +0000. [08:24:39] RECOVERY - wiki.wikimedia.cat - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wikimedia.cat' will expire on Wed 02 Mar 2022 06:46:53 GMT +0000. [08:25:56] RECOVERY - avas.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'avas.wiki' will expire on Wed 02 Mar 2022 06:50:57 GMT +0000. [08:26:07] RECOVERY - www.stardewmodding.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'stardewmodding.wiki' will expire on Wed 02 Mar 2022 06:46:09 GMT +0000. [08:26:48] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:30:19] RECOVERY - wiki.konjitownmc.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.konjitownmc.cf' will expire on Wed 02 Mar 2022 06:43:41 GMT +0000. [08:31:11] RECOVERY - wiki.worldbuilding.network - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worldbuilding.network' will expire on Wed 02 Mar 2022 06:52:54 GMT +0000. [08:43:22] !log rm -rf /mnt/mediawiki-static/metawiki/thumb (per T8346) [08:43:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:43:26] https://www.irccloud.com/pastebin/P0EFZ1V7/ [08:43:31] ^ paladox do we still need this? [09:02:48] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+15/-0/±14] 13https://git.io/JM65P [09:02:49] [02miraheze/ssl] 07Reception123 032a3cee3 - renew SSL certs [09:06:56] PROBLEM - mw13 Puppet on mw13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [09:07:04] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [09:07:15] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [09:08:26] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [09:09:08] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [09:09:13] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [09:13:04] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [09:19:29] [02miraheze/ssl] 07Reception123 pushed 033 commits to 03master [+0/-3/±6] 13https://git.io/JM6NX [09:19:31] [02miraheze/ssl] 07Reception123 033494795 - remove www.chinatech.wiki cert (not pointing) [09:19:32] [02miraheze/ssl] 07Reception123 032214ef3 - remove studentwiki.ddns.net cert (not pointing) [09:19:34] [02miraheze/ssl] 07Reception123 0334785f2 - remove privacy-wiki.0x.no cert (not pointing) [09:20:56] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:21:04] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [09:22:26] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [09:25:06] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/JM6AN [09:25:07] [02miraheze/ssl] 07Reception123 03c53a43c - add wiki.pacebi.org cert [09:25:14] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [09:25:19] RECOVERY - commons.ballmedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'commons.ballmedia.org' will expire on Wed 02 Mar 2022 07:54:48 GMT +0000. [09:26:21] RECOVERY - wiki.phgalaxy.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.phgalaxy.eu.org' will expire on Wed 02 Mar 2022 07:58:32 GMT +0000. [09:27:09] RECOVERY - ballmedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ballmedia.org' will expire on Wed 02 Mar 2022 07:57:32 GMT +0000. [09:27:18] RECOVERY - repository.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'repository.archiopedia.org' will expire on Wed 02 Mar 2022 08:01:27 GMT +0000. [09:27:36] RECOVERY - jp.popira.info - LetsEncrypt on sslhost is OK: OK - Certificate 'jp.popira.info' will expire on Wed 02 Mar 2022 07:54:05 GMT +0000. [09:27:55] RECOVERY - small.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'small.polandballwiki.com' will expire on Wed 02 Mar 2022 08:00:52 GMT +0000. [09:28:08] RECOVERY - zenith.nsmbu.net - LetsEncrypt on sslhost is OK: OK - Certificate 'zenith.nsmbu.net' will expire on Wed 02 Mar 2022 07:49:12 GMT +0000. [09:28:13] RECOVERY - wiki.potabi.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.potabi.com' will expire on Wed 02 Mar 2022 07:57:57 GMT +0000. [09:28:27] RECOVERY - www.ballmedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ballmedia.org' will expire on Wed 02 Mar 2022 07:57:32 GMT +0000. [09:28:33] RECOVERY - pwiki.drydraytonvillagehall.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.drydraytonvillagehall.org.uk' will expire on Wed 02 Mar 2022 08:02:05 GMT +0000. [09:29:02] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:29:12] RECOVERY - wiki.apadotech.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.apadotech.ga' will expire on Wed 02 Mar 2022 07:59:24 GMT +0000. [09:29:58] RECOVERY - usaha.click - LetsEncrypt on sslhost is OK: OK - Certificate 'usaha.click' will expire on Wed 02 Mar 2022 08:00:07 GMT +0000. [09:30:16] RECOVERY - famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.famepedia.org' will expire on Wed 02 Mar 2022 07:51:52 GMT +0000. [09:30:38] RECOVERY - trollpasta.com - LetsEncrypt on sslhost is OK: OK - Certificate 'trollpasta.com' will expire on Wed 02 Mar 2022 07:56:46 GMT +0000. [09:31:38] RECOVERY - central.ballmedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'central.ballmedia.org' will expire on Wed 02 Mar 2022 07:55:22 GMT +0000. [09:32:20] RECOVERY - www.trollpasta.com - LetsEncrypt on sslhost is OK: OK - Certificate 'trollpasta.com' will expire on Wed 02 Mar 2022 07:56:46 GMT +0000. [09:32:46] RECOVERY - en.famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.famepedia.org' will expire on Wed 02 Mar 2022 07:51:52 GMT +0000. [09:42:57] PROBLEM - mwtask1 Puppet on mwtask1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.pacebi.org_private] [09:44:10] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.pacebi.org_private] [09:44:59] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.pacebi.org_private] [10:11:41] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [10:12:23] RECOVERY - mwtask1 Puppet on mwtask1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:13:13] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:13:25] RECOVERY - bluearchive.wiki - reverse DNS on sslhost is OK: SSL OK - bluearchive.wiki reverse DNS resolves to cp21.miraheze.org - CNAME OK [11:01:28] RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 14 Jan 2022 03:39:45 GMT +0000. [11:13:21] I got a database error on The Kirby Encyclopedia Wiki whenever I went into the Recent Changes. [11:13:21] https://cdn.discordapp.com/attachments/808001911868489748/915923587427360768/bandicam_2021-12-02_06-12-31-128.jpg [11:13:49] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.46, 6.45, 4.50 [11:15:48] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.79, 5.42, 4.36 [11:19:50] Please file task [11:20:32] Okay then. [11:21:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.79, 6.60, 4.56 [11:23:46] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.11, 5.31, 4.33 [11:24:04] https://phabricator.miraheze.org/T8374 I just filed the said task @RhinosF1. [11:24:05] [url] ⚓ T8374 Database error on kirby.miraheze.org regarding the Special:RecentChanges page | phabricator.miraheze.org [11:24:22] Ok [11:24:52] I even included the exact same screenshot I uploaded here. [11:38:07] !log running delbackups ( bash ./delbackups.sh /home/reception/delwikis.txt /srv/mediawiki/w/maintenance/dumpBackup.php) [11:38:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:39:19] Hello, is there a way to access Google Spreadsheets data on a wiki? Does the GoogleDocs4MW (https://m.mediawiki.org/wiki/Extension:GoogleDocs4MW) have this functionality or is it only for embedding? [11:39:20] [url] Extension:GoogleDocs4MW - MediaWiki | m.mediawiki.org [11:40:36] For example, accessing a value of a specific cell in row x [11:43:33] !log reception@mw11:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=devilmanwiki [11:43:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:56:46] PROBLEM - wiki.etn.link - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.etn.link could not be found [11:57:43] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.12, 5.86, 4.34 [11:59:44] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.60, 5.32, 4.33 [12:08:34] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JMiEc [12:08:36] [02miraheze/CreateWiki] 07translatewiki 0356013fb - Localisation updates from https://translatewiki.net. [12:08:37] [url] Main page - translatewiki.net | translatewiki.net [12:08:37] [02miraheze/GlobalNewFiles] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JMiEC [12:08:39] [02miraheze/GlobalNewFiles] 07translatewiki 03b940b1c - Localisation updates from https://translatewiki.net. [12:08:40] [url] Main page - translatewiki.net | translatewiki.net [12:08:40] [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JMiEW [12:08:42] [02miraheze/DataDump] 07translatewiki 03b5bb055 - Localisation updates from https://translatewiki.net. [12:08:43] [url] Main page - translatewiki.net | translatewiki.net [12:08:43] [02miraheze/MatomoAnalytics] 07translatewiki pushed 031 commit to 03master [+0/-0/±15] 13https://git.io/JMiEl [12:08:45] [02miraheze/MatomoAnalytics] 07translatewiki 03a38aab2 - Localisation updates from https://translatewiki.net. [12:08:46] [url] Main page - translatewiki.net | translatewiki.net [12:08:46] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JMiE8 [12:08:48] [02miraheze/MirahezeMagic] 07translatewiki 036298756 - Localisation updates from https://translatewiki.net. [12:08:49] [url] Main page - translatewiki.net | translatewiki.net [12:09:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.26, 2.76 [12:11:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.56, 2.99, 2.72 [12:13:17] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.10, 7.89, 5.58 [12:17:17] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.18, 6.67, 5.71 [12:25:57] Reception123: the exports? No, wouldn’t think so [12:26:22] miraheze/DataDump - translatewiki the build passed. [12:28:14] miraheze/GlobalNewFiles - translatewiki the build passed. [12:28:17] miraheze/CreateWiki - translatewiki the build passed. [12:29:01] miraheze/MatomoAnalytics - translatewiki the build passed. [12:30:27] paladox: yeah, it seems that they're like 7GB so I was thinking of getting rid of them [12:32:47] miraheze/MirahezeMagic - translatewiki the build passed. [12:45:21] hello [12:45:38] good night(Taiwan time) [12:46:43] excuseme, already Exceed time.( i waitting 2 days), not notice now. [12:47:01] .tell JohnLewis Hi. Not sure if maybe I asked you before but the MW team is out of ideas re why upgrading to bullseye makes servers OOM and what we can do to fix it. Do you think you could find some time to look into it yourself? (In case you didn't know Paladox upgraded mw8 back to bullseye and OOMs are back) [12:47:01] Reception123: I'll pass that on when JohnLewis is around. [12:47:47] interwiki global rights admain. [12:48:20] dmehus have agree give me this. [12:48:27] thanks. [12:48:55] dmehus said he'd assess your local election for local interwiki administrator, not global interwiki administrator [12:49:02] also, this is the wrong channel to ask in [12:49:26] ok [12:49:43] so i how go to where channel talk? [12:49:49] #miraheze-relay [12:49:52] ok [12:49:55] thanks. [13:14:13] !log DELETED and DROPPED https://phabricator.miraheze.org/P449 [13:14:14] [url] ✎ P449 Deleted wikis 02/12/2021 | phabricator.miraheze.org [13:14:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:34:47] !log !log MariaDB [phabricator_herald]> UPDATE herald_rule SET isDisabled = '1' WHERE id = '5'; (not in use) [13:34:49] ergh, whatever [13:34:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:41:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.34, 3.48, 2.98 [13:43:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.82, 3.39, 3.01 [13:49:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.54, 3.19 [13:51:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.55, 3.23, 3.13 [14:06:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:11:38] ok : [RESOLVED] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:15:08] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:20:08] ok : [RESOLVED] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:21:08] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:22:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.75, 4.03, 3.32 [14:26:08] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:36:08] ok : [RESOLVED] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:39:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.75, 3.57, 3.99 [14:49:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.22, 4.03, 3.89 [14:53:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.32, 3.98, 3.93 [15:09:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.10, 2.91, 3.33 [15:13:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.99, 6.43, 4.71 [15:15:17] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.02, 5.70, 4.65 [15:18:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.80, 3.43, 3.39 [15:23:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.87, 3.58 [15:25:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.24, 3.71, 3.56 [15:33:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.04, 2.83, 3.19 [15:34:57] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.009 second response time [15:35:11] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.239 second response time [15:35:12] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.012 second response time [15:35:44] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.013 second response time [15:35:47] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw11 [15:35:55] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.244 second response time [15:35:56] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw11 [15:36:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:36:23] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw11 [15:36:43] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.15, 6.70, 5.30 [15:36:46] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw11 [15:38:38] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.08, 6.58, 5.41 [15:40:32] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.79, 6.52, 5.52 [15:41:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:41:51] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.13, 6.51, 5.40 [15:43:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.23, 6.72, 5.23 [15:43:50] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.61, 6.95, 5.68 [15:44:16] PROBLEM - mw11 APT on mw11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:44:21] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.30, 6.94, 5.90 [15:45:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.35, 3.13 [15:45:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.61, 6.42, 5.32 [15:46:15] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.06, 6.71, 5.94 [15:46:16] RECOVERY - mw11 APT on mw11 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:47:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.16, 3.13, 3.07 [15:47:48] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.64, 7.36, 6.16 [15:49:48] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 14.48, 9.50, 7.05 [15:50:06] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.56, 8.21, 6.66 [15:51:33] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.34, 9.45, 6.87 [15:52:52] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.97, 19.24, 16.63 [15:54:52] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.82, 18.88, 16.83 [15:55:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.70, 7.67, 6.71 [15:56:08] Reception123: mw11 is down [15:56:27] Please check php service status [15:57:24] paladox: why is mw11 missing from grafana [15:57:40] It's not showing as a host on the mediawiki board [15:57:48] it's not? [15:58:05] !log reboot mw11 [15:58:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:59:55] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.459 second response time [16:00:45] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 9 backends are healthy [16:00:46] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.009 second response time [16:01:03] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.128 second response time [16:01:20] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.312 second response time [16:01:44] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 0.269 second response time [16:01:48] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 9 backends are healthy [16:01:56] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 9 backends are healthy [16:02:22] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [16:03:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.22, 6.16, 6.47 [16:03:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 2.28, 6.57, 7.85 [16:03:47] paladox: that was not there a few minute ago [16:04:46] * Reception123 is back [16:06:06] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 3.45, 6.55, 7.74 [16:09:40] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.38, 4.71, 6.56 [16:12:06] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.08, 5.17, 6.72 [16:20:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.36, 3.02 [16:22:14] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.08, 2.99, 2.93 [16:28:51] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.91, 7.63, 6.68 [16:30:45] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.37, 6.27, 6.30 [16:51:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.50, 3.52, 3.15 [16:55:47] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.51, 3.35, 3.19 [17:02:26] PROBLEM - wiki.pacebi.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['dns1.registrar-servers.com.', 'dns2.registrar-servers.com.'], 'CNAME': 'pacebi.miraheze.org.'} [17:16:04] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.83, 3.62, 3.19 [17:17:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.57, 3.23 [17:21:47] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.79, 3.39 [17:23:42] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.25, 3.26, 3.25 [17:37:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.73, 3.52, 3.41 [17:41:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.61, 3.18, 3.31 [18:11:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.10, 6.39, 5.40 [18:12:06] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.65, 6.23, 5.16 [18:13:17] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.56, 7.07, 5.39 [18:13:46] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.26, 5.95, 5.35 [18:14:06] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.59, 6.07, 5.24 [18:15:17] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.25, 6.25, 5.29 [18:18:24] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.46, 3.79, 3.34 [18:22:06] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.41, 6.81, 5.87 [18:23:36] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.66, 9.08, 6.81 [18:24:06] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.43, 5.91, 5.66 [18:26:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.35, 3.81, 3.59 [18:27:36] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.77, 7.40, 6.69 [18:29:51] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.49, 4.03, 3.73 [18:31:36] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.32, 6.43, 6.47 [18:32:11] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.97, 5.96, 5.13 [18:33:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.71, 3.61, 3.64 [18:35:45] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 16.04, 9.67, 7.21 [18:36:06] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.83, 9.09, 7.27 [18:37:58] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.23, 6.07, 5.53 [18:38:06] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.54, 7.31, 6.84 [18:40:07] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.10, 7.67, 7.00 [18:40:09] Hey, got this message on my wiki [18:40:09] https://cdn.discordapp.com/attachments/808001911868489748/916036028580634634/unknown.png [18:40:18] Just wanted to let you all know that I've chosen to proxy my requests through Cloudflare which is why this is occuring [18:40:31] I'm not willing to miss out on all the nice Cloudflare stuff by skipping it [18:40:34] and it works fine [18:40:40] is that okay, or am I required to change it? [18:40:56] Who wrote that though? [18:41:17] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.76, 6.50, 5.62 [18:41:21] Oh, silly question. It says System Administrator right there. [18:41:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.48, 3.89, 3.72 [18:41:43] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.90, 6.33, 5.62 [18:41:48] Yeah they left it on my talk page [18:41:50] "Reception123" [18:42:56] Oh, @Reception123 wrote that? But why did he write System administrator, although he is one? [18:43:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.70, 6.87, 5.87 [18:43:17] No idea [18:43:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.59, 3.48, 3.59 [18:43:43] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.82, 6.28, 5.70 [18:44:40] @jmk Hi, I'm sorry that this is an inconvenience for you but the reason we don't allow Cloudflare proxying is because it makes our Content Security Policy not work therefore allowing external scripts to potentially be used which is a risk to our users. [18:44:49] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 14.81, 10.36, 7.46 [18:45:09] Oh I'm actually not allowed to use it? [18:45:17] That's really frustrating 😔 [18:45:17] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.07, 6.36, 5.79 [18:45:27] Is there a way I can fix that rather than disabling it? [18:45:40] @jmk Was there a particular reason for why you wanted to use it? Perhaps we can find a compromise [18:47:10] The overview provided by Cloudflare, page rules, etc [18:47:21] The Cloudflare features are important to me [18:47:36] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.32, 6.76, 6.41 [18:47:41] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.80, 7.46, 7.54 [18:47:43] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.71, 7.20, 6.21 [18:49:18] Which one is your wiki again? [18:49:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 3.90, 3.71 [18:49:36] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.71, 7.15, 6.57 [18:49:42] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.83, 8.98, 8.06 [18:49:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.99, 7.76, 6.50 [18:51:43] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.70, 7.20, 6.47 [18:53:34] PROBLEM - mw10 APT on mw10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:53:43] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.10, 7.50, 6.66 [18:54:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:55:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.66, 3.57, 3.68 [18:55:31] RECOVERY - mw10 APT on mw10 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:55:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.85, 7.64, 6.82 [18:57:51] Reception123: roapiwiki [18:59:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:01:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.33, 3.54, 3.58 [19:01:43] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.03, 7.40, 6.96 [19:02:13] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.95, 7.55, 7.81 [19:03:43] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.42, 7.22, 6.94 [19:05:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.83, 3.70 [19:07:36] Reception123 when you have time, please check DM [19:07:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.59, 7.79, 7.22 [19:09:43] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.25, 7.12, 7.03 [19:11:02] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.11, 7.20, 6.67 [19:11:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.39, 2.88, 3.31 [19:12:06] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.02, 6.74, 7.84 [19:13:01] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.03, 6.46, 6.46 [19:13:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.97, 6.37, 7.74 [19:13:43] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.81, 6.37, 6.75 [19:13:49] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.40, 5.68, 6.63 [19:15:36] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 3.75, 6.24, 7.58 [19:18:06] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.29, 7.21, 7.58 [19:19:34] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.76, 8.36, 8.01 [19:21:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.96, 7.98, 7.93 [19:21:36] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.16, 7.28, 7.43 [19:22:59] [02mw-config] 07magogre opened pull request 03#4244: Add a new canned response - 13https://git.io/JMXBG [19:23:36] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.12, 7.07, 7.34 [19:25:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.26, 6.94, 6.83 [19:27:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.68, 6.42, 6.65 [19:31:35] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.004 second response time [19:31:59] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.020 second response time [19:32:04] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.012 second response time [19:32:06] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 1.55, 6.63, 7.92 [19:32:22] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw12 [19:32:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:32:33] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.259 second response time [19:32:35] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.232 second response time [19:32:46] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw12 [19:33:27] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.51, 5.40, 6.59 [19:33:36] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.99, 7.55, 7.25 [19:33:43] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.91, 7.27, 6.37 [19:33:46] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.65, 7.76, 6.61 [19:33:48] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw12 [19:33:56] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw12 [19:36:06] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 0.36, 3.20, 6.21 [19:36:44] [02mw-config] 07DarkMatterMan4500 commented on pull request 03#4244: Add a new canned response - 13https://git.io/JMX08 [19:39:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.73, 3.35, 3.21 [19:41:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.30, 3.23, 3.18 [19:42:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:45:18] miraheze/mw-config - Universal-Omega the build passed. [20:03:36] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.29, 3.52, 3.16 [20:07:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.53, 3.23 [20:11:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.67, 3.77, 3.38 [20:13:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.48, 3.32, 3.26 [20:17:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 3.84, 3.49 [20:19:10] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.57, 6.76, 5.77 [20:19:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.10, 3.43, 3.38 [20:21:09] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.01, 6.09, 5.68 [20:22:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:22:16] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.003 second response time [20:23:00] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.229 second response time [20:23:09] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.004 second response time [20:23:12] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.005 second response time [20:23:29] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.230 second response time [20:23:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.99, 4.62, 3.83 [20:23:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [20:25:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [20:26:12] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.49, 20.52, 19.06 [20:28:09] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.30, 20.09, 19.08 [20:29:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:31:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:33:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [20:37:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:39:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.75, 3.87 [20:41:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.40, 3.95, 3.92 [20:45:26] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JMXon [20:45:27] [02miraheze/mediawiki] 07Reception123 03151685f - Update PageForms [20:47:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [20:51:32] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:51:35] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:51:51] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:52] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:51:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [20:53:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.36, 3.66, 3.97 [20:53:35] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 8.144 second response time [20:53:39] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 8.262 second response time [20:53:40] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.18, 21.15, 20.27 [20:53:56] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 6.470 second response time [20:53:57] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 7.738 second response time [20:55:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:57:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.48, 3.95, 4.01 [20:57:33] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.25, 20.32, 20.16 [20:59:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:03:20] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.33, 20.91, 20.42 [21:04:43] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:05:04] !log [universalomega@test3] starting deploy of {'world': True} to skip [21:05:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:05:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:07:54] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49491 MB (11% inode=98%); [21:08:36] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [21:09:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [21:10:54] !log [universalomega@test3] finished deploy of {'world': True} to skip - SUCCESS in 349s [21:11:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:15:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.40, 3.37, 3.82 [21:15:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:16:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:21:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.46, 3.84, 3.85 [21:22:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb [21:23:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.51, 3.72 [21:23:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb [21:28:29] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:30:26] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.792 second response time [21:33:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.30, 3.91, 3.70 [21:35:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.85, 3.71 [21:37:03] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.55, 19.47, 20.32 [21:38:06] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:38:10] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:40:12] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15869 bytes in 8.686 second response time [21:40:14] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 9.653 second response time [21:42:37] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:43:08] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:43:26] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:44:36] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:44:38] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:45:03] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.49, 20.15, 20.27 [21:45:20] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:47:03] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.88, 20.33, 20.34 [21:47:24] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15869 bytes in 9.145 second response time [21:47:44] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 7.403 second response time [21:49:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.58, 3.52 [21:50:45] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 8.152 second response time [21:50:48] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 8.550 second response time [21:50:51] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 9.664 second response time [21:51:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.74, 3.39, 3.47 [21:53:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 3.08, 3.35 [21:54:32] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:54:50] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:55:29] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:58:33] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:01:51] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15869 bytes in 9.659 second response time [22:02:26] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 5.926 second response time [22:02:28] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:02:41] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 6.169 second response time [22:02:42] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:43] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:02:48] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 6.842 second response time [22:03:10] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 7.830 second response time [22:06:35] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 4.879 second response time [22:06:53] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 8.668 second response time [22:06:54] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 8.142 second response time [22:08:57] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:11:05] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 9.177 second response time [22:13:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.24, 3.77, 3.37 [22:15:31] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:17:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.63, 3.42 [22:17:36] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:19:14] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:19:55] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:21:03] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.63, 20.84, 20.22 [22:21:20] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 9.684 second response time [22:21:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.83, 3.54 [22:21:43] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 7.874 second response time [22:21:54] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 6.781 second response time [22:21:57] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 5.681 second response time [22:25:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.57, 3.51 [22:25:43] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:26:16] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:27:41] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 1.420 second response time [22:28:13] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 1.421 second response time [22:28:53] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:29:07] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:30:50] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 1.688 second response time [22:31:03] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.16, 20.00, 20.15 [22:31:04] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.012 second response time [22:31:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:33:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.66, 3.13, 3.33 [22:35:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [22:35:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:37:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.50, 3.42 [22:37:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:41:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.37, 3.41, 3.41 [22:45:38] paladox: what is up with ^? continuing to go down again and again. [22:46:03] looks like it recovered, i'm not sure though. [22:51:07] CosmicAlpha: I imagine mw8 ooming is pushing the load elsewhere [22:51:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 3.84, 3.59 [22:53:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.69, 3.36, 3.45 [22:55:03] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.66, 20.59, 20.07 [22:55:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.54, 3.99, 3.68 [22:59:03] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.12, 20.20, 20.06 [22:59:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 1.89, 3.22, 3.46 [23:00:02] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:10] !log [@mw11] starting deploy of {'l10nupdate': True} to all [23:00:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.14, 3.16, 3.40 [23:03:04] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.60, 21.96, 20.80 [23:03:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2607:5300:201:3100::929a/cpweb [23:05:03] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.70, 24.51, 21.96 [23:05:33] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:07:54] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb [23:09:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:09:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:11:23] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:13:03] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.44, 23.59, 22.67 [23:13:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [23:15:11] PROBLEM - mw10 APT on mw10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:15:47] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JMXN5 [23:15:49] [02miraheze/mediawiki] 07Universal-Omega 03c7d9ec3 - Update TimeMachine [23:16:14] !log [universalomega@test3] starting deploy of {'world': True} to skip [23:16:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:19:17] RECOVERY - mw10 APT on mw10 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:20:14] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:20:28] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:20:34] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:20:42] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:21:10] !log [universalomega@test3] finished deploy of {'world': True} to skip - SUCCESS in 296s [23:21:19] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:21:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:07] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:22:11] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.316 second response time [23:22:22] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.044 second response time [23:22:31] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.353 second response time [23:22:38] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15869 bytes in 0.383 second response time [23:23:16] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 1.039 second response time [23:24:05] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:25:11] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:25:54] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48959 MB (10% inode=98%); [23:27:14] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 3.297 second response time [23:27:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:29:53] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.61, 3.86, 3.43 [23:30:52] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:31:43] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 1901s [23:31:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:32:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:33:40] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:35:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.76, 3.67, 3.54 [23:35:42] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 5.147 second response time [23:36:45] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:37:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 4.02, 3.69 [23:37:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:40:38] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [23:41:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.79, 3.68 [23:41:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb [23:44:08] !log [@mw11] DEPLOY ABORTED: Canary check failed for mw9 [23:45:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:45:40] !log reboot mw10 [23:46:34] !log [universalomega@test3] starting deploy of {'landing': True} to skip [23:46:35] !log [universalomega@test3] finished deploy of {'landing': True} to skip - SUCCESS in 0s [23:47:01] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:47:03] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.07, 21.46, 21.39 [23:47:16] !log reboot mw10 [23:47:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.33, 2.93, 3.35 [23:47:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:48:28] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:48:58] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.353 second response time [23:49:03] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.94, 20.84, 21.18 [23:49:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.35, 7.21, 5.87 [23:49:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:51:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.47, 3.69, 3.59 [23:51:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.35, 6.99, 5.96 [23:52:25] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JMXjX [23:52:26] [02miraheze/landing] 07Universal-Omega 0318d8118 - Update getTranslations.php [23:52:28] [02landing] 07Universal-Omega synchronize pull request 03#69: Cleanup and support qqx - 13https://git.io/JMENy [23:52:46] !log [universalomega@test3] starting deploy of {'landing': True} to skip [23:52:47] !log [universalomega@test3] finished deploy of {'landing': True} to skip - SUCCESS in 0s [23:52:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:52:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:53:20] miraheze/landing - Universal-Omega the build passed. [23:53:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.24, 7.56, 6.30 [23:58:15] [02landing] 07Universal-Omega closed pull request 03#69: Cleanup and support qqx - 13https://git.io/JMENy [23:58:17] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JM1ew [23:58:18] [02miraheze/landing] 07Universal-Omega 03ee598bb - Cleanup and support qqx (#69) [23:58:20] [02landing] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fxEQA [23:58:21] [02miraheze/landing] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [23:59:12] miraheze/landing - Universal-Omega the build passed. [23:59:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.37, 7.77, 6.93