[00:00:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.04, 11.43, 10.67 [00:01:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.31, 11.61, 10.70 [00:03:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.14, 4.16, 3.55 [00:03:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.32, 11.38, 10.75 [00:04:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.82, 11.62, 10.95 [00:05:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.19, 3.66, 3.44 [00:09:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.87, 3.21, 3.31 [00:12:55] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.76, 8.68, 9.83 [00:15:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.16, 8.65, 9.79 [00:26:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.28, 7.22, 6.52 [00:30:36] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.72, 7.93, 6.93 [00:30:37] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.21, 6.24, 4.39 [00:31:49] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.46, 3.20, 2.73 [00:32:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.61, 7.43, 6.86 [00:33:49] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.20, 2.82, 2.65 [00:34:28] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.10, 6.80, 6.69 [00:36:29] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.19, 6.50, 5.27 [00:38:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.65, 7.00, 6.80 [00:40:19] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.26, 6.60, 6.67 [00:41:15] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:42:10] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.49, 10.27, 9.50 [00:43:13] RECOVERY - cp33 APT on cp33 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [00:43:30] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.07, 4.20, 3.28 [00:44:05] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.18, 9.28, 9.24 [00:44:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.75, 7.07, 6.85 [00:45:27] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.96, 3.68, 3.18 [00:47:25] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.86, 2.92, 2.96 [00:48:06] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.73, 6.59, 6.73 [00:52:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.90, 7.31, 7.04 [00:54:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.76, 7.81, 7.25 [00:56:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.84, 7.13, 7.08 [00:58:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.96, 8.07, 7.43 [00:58:07] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.99, 3.63, 3.23 [01:00:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.05, 6.88, 7.07 [01:00:04] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.72, 3.40, 3.20 [01:00:15] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.64, 6.18, 4.48 [01:02:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.38, 5.21, 4.34 [01:04:02] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.24, 5.99, 6.66 [01:15:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.79, 4.12, 3.41 [01:16:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.75, 7.48, 7.00 [01:18:02] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.82, 6.54, 6.71 [01:31:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.80, 3.81, 3.91 [01:35:03] [02MirahezeMagic] 07AgentIsai opened pull request 03#420: Cleanup and organize group names - 13https://github.com/miraheze/MirahezeMagic/pull/420 [01:37:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.00, 2.83, 3.40 [01:39:04] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 8251 MB (6% inode=99%); [01:41:56] [02dns] 07Universal-Omega opened pull request 03#407: Remove mwcosmos.com and dcmultiversewiki.com zones - 13https://github.com/miraheze/dns/pull/407 [01:43:34] miraheze/MirahezeMagic - AgentIsai the build passed. [02:27:04] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7969 MB (5% inode=99%); [02:30:15] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.21, 6.22, 3.78 [02:32:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.31, 5.37, 3.80 [02:41:19] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:41:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.27, 3.76, 2.84 [02:43:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.11, 3.38, 2.81 [02:43:56] RECOVERY - cp33 APT on cp33 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [02:47:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.13, 3.51, 3.00 [02:49:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.47, 3.18, 2.95 [03:02:20] PROBLEM - nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nonbinary.wiki' expires in 15 day(s) (Wed 10 May 2023 02:59:00 GMT +0000). [03:02:20] PROBLEM - www.allthetropes.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'allthetropes.org' expires in 15 day(s) (Wed 10 May 2023 02:57:45 GMT +0000). [03:02:51] PROBLEM - allthetropes.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'allthetropes.org' expires in 15 day(s) (Wed 10 May 2023 02:57:45 GMT +0000). [03:04:26] PROBLEM - sdiy.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sdiy.info' expires in 15 day(s) (Wed 10 May 2023 02:59:38 GMT +0000). [03:04:30] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.53, 3.72, 3.30 [03:05:24] PROBLEM - wiki.worlduniversityandschool.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.worlduniversityandschool.org' expires in 15 day(s) (Wed 10 May 2023 02:58:52 GMT +0000). [03:06:01] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0be08d67469d...8021e9d9929e [03:06:04] [02miraheze/ssl] 07MirahezeSSLBot 038021e9d - Bot: Update SSL cert for nonbinary.wiki [03:06:38] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8021e9d9929e...1f76105c3d47 [03:06:38] [02miraheze/ssl] 07MirahezeSSLBot 031f76105 - Bot: Update SSL cert for allthetropes.org [03:08:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1f76105c3d47...6098ccb1195f [03:08:13] [02miraheze/ssl] 07MirahezeSSLBot 036098ccb - Bot: Update SSL cert for sdiy.info [03:09:21] PROBLEM - www.sdiy.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sdiy.info' expires in 15 day(s) (Wed 10 May 2023 02:59:38 GMT +0000). [03:09:43] PROBLEM - adadevelopersacademy.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'adadevelopersacademy.wiki' expires in 15 day(s) (Wed 10 May 2023 02:59:16 GMT +0000). [03:09:59] PROBLEM - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'permanentfuturelab.wiki' expires in 15 day(s) (Wed 10 May 2023 02:56:55 GMT +0000). [03:10:19] PROBLEM - publictestwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'publictestwiki.com' expires in 15 day(s) (Wed 10 May 2023 02:57:07 GMT +0000). [03:10:22] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.02, 3.89, 3.48 [03:10:31] PROBLEM - wiki.autocountsoft.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.autocountsoft.com' expires in 15 day(s) (Wed 10 May 2023 02:59:09 GMT +0000). [03:12:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.56, 2.99, 3.19 [03:13:04] PROBLEM - speleo.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'speleo.wiki' expires in 15 day(s) (Wed 10 May 2023 02:58:16 GMT +0000). [03:13:21] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'antiguabarbudacalypso.com' expires in 15 day(s) (Wed 10 May 2023 02:56:44 GMT +0000). [03:13:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6098ccb1195f...048ad850f1a8 [03:13:44] [02miraheze/ssl] 07MirahezeSSLBot 03048ad85 - Bot: Update SSL cert for wiki.worlduniversityandschool.org [03:15:17] PROBLEM - www.publictestwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'publictestwiki.com' expires in 15 day(s) (Wed 10 May 2023 02:57:07 GMT +0000). [03:15:21] PROBLEM - wiki.meeusen.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.meeusen.net' expires in 15 day(s) (Wed 10 May 2023 02:58:03 GMT +0000). [03:15:33] PROBLEM - www.zenbuddhism.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.zenbuddhism.info' expires in 15 day(s) (Wed 10 May 2023 02:57:56 GMT +0000). [03:15:33] PROBLEM - wiki.macc.nyc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.macc.nyc' expires in 15 day(s) (Wed 10 May 2023 02:59:26 GMT +0000). [03:15:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/048ad850f1a8...03a0580966c1 [03:15:46] [02miraheze/ssl] 07MirahezeSSLBot 0303a0580 - Bot: Update SSL cert for wiki.autocountsoft.com [03:18:32] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/03a0580966c1...c00b7f36bc60 [03:18:33] [02miraheze/ssl] 07MirahezeSSLBot 03c00b7f3 - Bot: Update SSL cert for publictestwiki.com [03:18:59] PROBLEM - wiki.grottocenter.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.grottocenter.org' expires in 15 day(s) (Wed 10 May 2023 02:57:14 GMT +0000). [03:21:10] PROBLEM - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.gesamtschule-nordkirchen.de' expires in 15 day(s) (Wed 10 May 2023 02:59:49 GMT +0000). [03:22:18] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'permanentfuturelab.wiki' expires in 15 day(s) (Wed 10 May 2023 02:56:55 GMT +0000). [03:23:09] PROBLEM - wiki.make717.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.make717.org' expires in 15 day(s) (Wed 10 May 2023 02:57:22 GMT +0000). [03:23:27] PROBLEM - wiki.tulpa.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.tulpa.info' expires in 15 day(s) (Wed 10 May 2023 02:58:32 GMT +0000). [03:25:59] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.62, 3.42, 3.10 [03:26:50] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c00b7f36bc60...deba3e464bda [03:26:52] [02miraheze/ssl] 07MirahezeSSLBot 03deba3e4 - Bot: Update SSL cert for antiguabarbudacalypso.com [03:27:56] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.03, 3.09, 3.03 [03:28:17] PROBLEM - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'thelonsdalebattalion.co.uk' expires in 15 day(s) (Wed 10 May 2023 02:57:37 GMT +0000). [03:28:40] PROBLEM - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'marinebiodiversitymatrix.org' expires in 15 day(s) (Wed 10 May 2023 02:58:40 GMT +0000). [03:29:02] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [03:31:10] RECOVERY - www.allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.org' will expire on Sun 23 Jul 2023 02:06:30 GMT +0000. [03:31:10] RECOVERY - nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nonbinary.wiki' will expire on Sun 23 Jul 2023 02:05:53 GMT +0000. [03:32:11] RECOVERY - allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.org' will expire on Sun 23 Jul 2023 02:06:30 GMT +0000. [03:32:50] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/deba3e464bda...7295d59e4379 [03:32:51] [02miraheze/ssl] 07MirahezeSSLBot 037295d59 - Bot: Update SSL cert for wiki.macc.nyc [03:33:51] RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Sun 23 Jul 2023 02:08:02 GMT +0000. [03:34:11] RECOVERY - wiki.worlduniversityandschool.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worlduniversityandschool.org' will expire on Sun 23 Jul 2023 02:13:34 GMT +0000. [03:35:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.72, 5.76, 4.07 [03:39:12] RECOVERY - www.sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Sun 23 Jul 2023 02:08:02 GMT +0000. [03:39:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7295d59e4379...7b8d47f63ca4 [03:39:37] [02miraheze/ssl] 07MirahezeSSLBot 037b8d47f - Bot: Update SSL cert for marinebiodiversitymatrix.org [03:39:38] RECOVERY - publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Sun 23 Jul 2023 02:18:25 GMT +0000. [03:39:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.19, 3.56, 3.53 [03:40:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7b8d47f63ca4...6202a9ba65c2 [03:40:43] [02miraheze/ssl] 07MirahezeSSLBot 036202a9b - Bot: Update SSL cert for thelonsdalebattalion.co.uk [03:41:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.78, 3.01, 3.33 [03:44:06] RECOVERY - wiki.macc.nyc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.macc.nyc' will expire on Sun 23 Jul 2023 02:32:41 GMT +0000. [03:45:04] RECOVERY - www.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Sun 23 Jul 2023 02:18:25 GMT +0000. [03:47:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6202a9ba65c2...8308889315c5 [03:47:58] [02miraheze/ssl] 07MirahezeSSLBot 038308889 - Bot: Update SSL cert for wiki.grottocenter.org [03:51:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8308889315c5...d1b8bfeef0c8 [03:51:27] [02miraheze/ssl] 07MirahezeSSLBot 03d1b8bfe - Bot: Update SSL cert for www.zenbuddhism.info [03:57:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d1b8bfeef0c8...2d805638e1cf [03:57:50] [02miraheze/ssl] 07MirahezeSSLBot 032d80563 - Bot: Update SSL cert for wiki.gesamtschule-nordkirchen.de [04:00:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2d805638e1cf...1d3a95449c0f [04:00:44] [02miraheze/ssl] 07MirahezeSSLBot 031d3a954 - Bot: Update SSL cert for wiki.make717.org [04:03:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1d3a95449c0f...92dd4df3f3c7 [04:03:17] [02miraheze/ssl] 07MirahezeSSLBot 0392dd4df - Bot: Update SSL cert for wiki.tulpa.info [04:04:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.50, 7.18, 6.56 [04:06:04] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.52, 6.74, 6.49 [04:06:39] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.60, 6.43, 3.50 [04:08:36] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.63, 5.56, 3.55 [04:09:33] RECOVERY - wiki.autocountsoft.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.autocountsoft.com' will expire on Sun 23 Jul 2023 02:15:36 GMT +0000. [04:10:01] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/92dd4df3f3c7...3943ef39e793 [04:10:03] [02miraheze/ssl] 07MirahezeSSLBot 033943ef3 - Bot: Update SSL cert for speleo.wiki [04:12:08] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Sun 23 Jul 2023 02:26:43 GMT +0000. [04:15:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3943ef39e793...f5368d9a9ba7 [04:15:44] [02miraheze/ssl] 07MirahezeSSLBot 03f5368d9 - Bot: Update SSL cert for wiki.meeusen.net [04:15:56] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:16:02] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f5368d9a9ba7...5b5376673c29 [04:16:04] [02miraheze/ssl] 07MirahezeSSLBot 035b53766 - Bot: Update SSL cert for adadevelopersacademy.wiki [04:16:59] RECOVERY - wiki.grottocenter.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.grottocenter.org' will expire on Sun 23 Jul 2023 02:47:50 GMT +0000. [04:18:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5b5376673c29...a32c0cb56781 [04:18:37] [02miraheze/ssl] 07MirahezeSSLBot 03a32c0cb - Bot: Update SSL cert for permanentfuturelab.wiki [04:20:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.83, 6.85, 6.49 [04:22:41] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.41, 6.21, 6.29 [04:26:51] RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Sun 23 Jul 2023 02:40:33 GMT +0000. [04:27:59] RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Sun 23 Jul 2023 02:39:28 GMT +0000. [04:37:43] RECOVERY - adadevelopersacademy.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'adadevelopersacademy.wiki' will expire on Sun 23 Jul 2023 03:15:55 GMT +0000. [04:37:59] RECOVERY - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Sun 23 Jul 2023 03:18:28 GMT +0000. [04:41:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.12, 3.76, 2.74 [04:43:33] RECOVERY - www.zenbuddhism.info - LetsEncrypt on sslhost is OK: OK - Certificate 'www.zenbuddhism.info' will expire on Sun 23 Jul 2023 02:51:18 GMT +0000. [04:43:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.66, 2.88, 2.54 [04:44:52] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [04:44:55] RECOVERY - wiki.meeusen.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meeusen.net' will expire on Sun 23 Jul 2023 03:15:36 GMT +0000. [04:45:04] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 8209 MB (6% inode=99%); [04:49:51] [02MirahezeMagic] 07Reception123 synchronize pull request 03#419: add script to download/upload swift containers for DataDump compatibi… - 13https://github.com/miraheze/MirahezeMagic/pull/419 [04:49:52] [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/3d93ef7f803a...ee02e9804247 [04:49:54] [02miraheze/MirahezeMagic] 07Reception123 03ee02e98 - some fixes per UO [04:50:10] RECOVERY - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gesamtschule-nordkirchen.de' will expire on Sun 23 Jul 2023 02:57:40 GMT +0000. [04:50:16] RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Sun 23 Jul 2023 03:18:28 GMT +0000. [04:51:10] RECOVERY - wiki.make717.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.make717.org' will expire on Sun 23 Jul 2023 03:00:35 GMT +0000. [04:51:27] RECOVERY - wiki.tulpa.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tulpa.info' will expire on Sun 23 Jul 2023 03:03:00 GMT +0000. [04:52:18] miraheze/MirahezeMagic - Reception123 the build has errored. [04:59:00] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:01:04] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7574 MB (5% inode=99%); [05:03:25] PROBLEM - matomo121 HTTPS on matomo121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.023 second response time [05:06:24] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 838 bytes in 0.071 second response time [05:08:29] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.793 second response time [05:11:03] RECOVERY - speleo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'speleo.wiki' will expire on Sun 23 Jul 2023 03:09:54 GMT +0000. [05:16:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 19.06, 11.03, 7.48 [05:19:30] RECOVERY - matomo121 HTTPS on matomo121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.298 second response time [05:22:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.32, 3.02, 2.47 [05:24:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.22, 7.72, 7.41 [05:24:50] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:29:53] RECOVERY - cp33 APT on cp33 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [05:30:03] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.37, 5.52, 6.49 [05:34:17] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.92, 3.69, 3.48 [05:35:05] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 9057 MB (6% inode=99%); [05:36:15] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.51, 3.03, 3.26 [05:37:50] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1160632968 secs [05:53:49] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.76, 3.06, 2.77 [05:56:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.98, 3.23, 2.89 [05:58:30] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09370571375 secs [06:08:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.94, 6.47, 6.13 [06:10:02] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.32, 6.25, 6.08 [06:11:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.26, 8.77, 7.35 [06:13:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.82, 8.31, 7.36 [06:16:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.97, 6.50, 6.21 [06:18:02] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.58, 6.20, 6.13 [06:26:32] !log sudo php swiftdump.php --wiki darkangelwiki [06:26:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:31:04] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 14799 MB (11% inode=99%); [06:53:32] !log sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki darkangelwiki [06:53:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:02:41] !log sudo -u www-data php fix_swift.php --wiki dcmultiversewiki [07:02:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:02:51] !log sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/CreateWiki/maintenance/setContainersAccess.php [07:02:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:10:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.02, 6.65, 5.82 [07:11:05] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14594 MB (10% inode=99%); [07:12:02] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.86, 6.16, 5.73 [07:14:18] !log sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /home/reception/fix_swift.php [07:14:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:00:20] [02dns] 07Reception123 closed pull request 03#407: Remove mwcosmos.com and dcmultiversewiki.com zones - 13https://github.com/miraheze/dns/pull/407 [08:00:21] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-2/±0] 13https://github.com/miraheze/dns/compare/a7f5b3c0e156...21c4bac6bcf0 [08:00:22] [02miraheze/dns] 07Universal-Omega 0321c4bac - Remove mwcosmos.com and dcmultiversewiki.com zones (#407) [08:33:04] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 15182 MB (11% inode=99%); [09:10:15] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.85, 5.37, 2.73 [09:12:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.81, 4.58, 2.78 [09:45:56] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:58:06] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:14:52] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [10:27:04] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14600 MB (10% inode=99%); [10:27:12] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [10:50:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.93, 10.41, 9.19 [10:52:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.29, 11.45, 9.74 [10:54:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.42, 10.99, 9.77 [10:56:16] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:58:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.08, 9.55, 9.49 [11:00:42] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [11:12:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.85, 10.16, 9.75 [11:13:55] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/277a8cb8bd34...8c8113459d35 [11:13:57] [02miraheze/CreateWiki] 07translatewiki 038c81134 - Localisation updates from https://translatewiki.net. [11:13:58] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±6] 13https://github.com/miraheze/MirahezeMagic/compare/f80ef674b6bf...25972ef2c2f5 [11:13:59] [02miraheze/MirahezeMagic] 07translatewiki 0325972ef - Localisation updates from https://translatewiki.net. [11:14:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.48, 9.67, 9.61 [11:18:30] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 18.61, 14.22, 11.45 [11:18:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.28, 11.50, 9.89 [11:21:41] miraheze/CreateWiki - translatewiki the build passed. [11:24:18] miraheze/MirahezeMagic - translatewiki the build passed. [11:26:05] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [11:26:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.68, 11.87, 11.04 [11:29:24] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [11:30:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.12, 11.72, 11.15 [11:32:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.48, 11.43, 11.10 [11:34:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.58, 12.54, 11.54 [11:35:34] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1265385151 secs [11:36:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.22, 11.10, 11.15 [11:39:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.16, 10.32, 11.68 [11:41:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.36, 11.49, 11.95 [11:43:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 5.73, 9.25, 11.07 [11:44:41] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.81, 8.20, 9.80 [11:47:32] PROBLEM - wiki.limaru.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.limaru.net All nameservers failed to answer the query. [11:51:34] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.0860927999 secs [11:51:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.85, 8.92, 10.10 [11:57:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.90, 10.02, 10.22 [11:59:06] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=weirdcorecitywiki --no-updates --username-prefix=wikia:weirdcorecity /home/reception/ImportDump/weirdcorecitywiki-20230424115712.xml (START) [11:59:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:59:51] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=tutorialswiki --no-updates --username-prefix=w /home/reception/ImportDump/tutorialswiki-20230423122223.xml (START) [11:59:57] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=tttewikiwiki --no-updates --username-prefix=wikia:ttte /home/reception/ImportDump/tttewikiwiki-20230420163911.xml (START) [11:59:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:00:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:00:18] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=tutorialswiki --no-updates --username-prefix=w /home/reception/ImportDump/tutorialswiki-20230423122223.xml (END - exit=0) [12:00:19] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=tutorialswiki (START) [12:00:20] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=weirdcorecitywiki --no-updates --username-prefix=wikia:weirdcorecity /home/reception/ImportDump/weirdcorecitywiki-20230424115712.xml (END - exit=0) [12:00:21] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=weirdcorecitywiki (START) [12:00:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:00:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:00:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:00:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:00:56] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=weirdcorecitywiki (END - exit=0) [12:00:57] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=weirdcorecitywiki --active --update (END - exit=0) [12:01:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:01:05] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 15146 MB (11% inode=99%); [12:01:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:01:06] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=tutorialswiki (END - exit=0) [12:01:07] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=tutorialswiki --active --update (END - exit=0) [12:01:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:01:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:03:51] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.52, 11.45, 10.78 [12:09:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.83, 11.99, 11.37 [12:10:59] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.36, 3.61, 2.63 [12:11:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.85, 12.57, 11.68 [12:12:15] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.30, 5.59, 3.28 [12:12:59] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.71, 3.58, 2.74 [12:13:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.61, 11.44, 11.37 [12:14:16] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.20, 3.89, 2.92 [12:16:34] RECOVERY - wiki.limaru.net - reverse DNS on sslhost is OK: SSL OK - wiki.limaru.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [12:16:59] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.90, 3.20, 2.79 [12:18:27] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=tttewikiwiki --no-updates --username-prefix=wikia:ttte /home/reception/ImportDump/tttewikiwiki-20230420163911.xml (END - exit=0) [12:18:28] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=tttewikiwiki (START) [12:18:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:18:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:19:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.57, 12.28, 11.59 [12:20:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.03, 11.52, 10.47 [12:21:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 19.14, 9.98, 7.32 [12:25:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.25, 7.43, 6.89 [12:26:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.21, 11.65, 11.00 [12:27:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.10, 6.34, 6.55 [12:30:42] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [12:30:59] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.66, 3.66, 3.23 [12:32:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.58, 10.77, 10.70 [12:34:15] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.65, 5.68, 3.79 [12:36:16] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 18.01, 10.00, 5.58 [12:36:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.89, 11.16, 11.04 [12:36:59] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.25, 3.72, 3.36 [12:38:59] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.69, 3.71, 3.40 [12:40:23] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.56, 11.47, 11.11 [12:42:15] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:42:15] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.81, 7.31, 6.00 [12:44:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.13, 5.50, 5.50 [12:44:16] RECOVERY - cp33 APT on cp33 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [12:48:59] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.91, 3.28, 3.36 [12:51:08] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.102126807 secs [12:58:59] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.46, 3.30, 3.29 [12:59:24] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:00:59] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.80, 3.20, 3.26 [13:06:59] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.41, 3.34, 3.29 [13:07:08] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09985530376 secs [13:10:59] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.27, 3.30, 3.27 [13:14:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.42, 11.35, 11.98 [13:14:59] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.21, 3.42, 3.34 [13:16:59] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.98, 3.27, 3.30 [13:18:15] PROBLEM - pwiki.arkcls.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pwiki.arkcls.com' expires in 15 day(s) (Wed 10 May 2023 13:16:07 GMT +0000). [13:20:34] PROBLEM - wiki.mikrodev.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mikrodev.com' expires in 15 day(s) (Wed 10 May 2023 13:16:47 GMT +0000). [13:21:02] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'enc.for.uz' expires in 15 day(s) (Wed 10 May 2023 13:16:39 GMT +0000). [13:22:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.91, 11.36, 11.62 [13:22:28] PROBLEM - wiki.thunis.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.thunis.eu' expires in 15 day(s) (Wed 10 May 2023 13:18:52 GMT +0000). [13:23:39] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a32c0cb56781...08a248e446ec [13:23:41] [02miraheze/ssl] 07MirahezeSSLBot 0308a248e - Bot: Update SSL cert for pwiki.arkcls.com [13:23:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.25, 10.72, 11.93 [13:25:05] PROBLEM - nonciclopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nonciclopedia.org' expires in 15 day(s) (Wed 10 May 2023 13:17:30 GMT +0000). [13:25:13] PROBLEM - wiki.ciptamedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ciptamedia.org' expires in 15 day(s) (Wed 10 May 2023 13:15:39 GMT +0000). [13:25:19] PROBLEM - www.portalsofphereon.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.portalsofphereon.com' expires in 15 day(s) (Wed 10 May 2023 13:23:00 GMT +0000). [13:25:23] PROBLEM - runzeppelin.ru - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'runzeppelin.ru' expires in 15 day(s) (Wed 10 May 2023 13:20:07 GMT +0000). [13:25:42] PROBLEM - wiki.pablojprieto.com.es - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.pablojprieto.com.es' expires in 15 day(s) (Wed 10 May 2023 13:19:18 GMT +0000). [13:26:03] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/08a248e446ec...a9175ab6918b [13:26:06] [02miraheze/ssl] 07MirahezeSSLBot 03a9175ab - Bot: Update SSL cert for enc.for.uz [13:26:20] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.zymonic.com' expires in 15 day(s) (Wed 10 May 2023 13:20:54 GMT +0000). [13:26:51] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:27:04] PROBLEM - wiki.openhatch.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.openhatch.org' expires in 15 day(s) (Wed 10 May 2023 13:17:06 GMT +0000). [13:27:10] PROBLEM - portalsofphereon.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.portalsofphereon.com' expires in 15 day(s) (Wed 10 May 2023 13:23:00 GMT +0000). [13:27:21] PROBLEM - wiki.scvo.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.scvo.org' expires in 15 day(s) (Wed 10 May 2023 13:20:25 GMT +0000). [13:27:33] PROBLEM - wiki.valkyrienskies.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.valkyrienskies.org' expires in 15 day(s) (Wed 10 May 2023 13:21:55 GMT +0000). [13:27:42] PROBLEM - wiki.graalmilitary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.graalmilitary.com' expires in 15 day(s) (Wed 10 May 2023 13:22:23 GMT +0000). [13:27:48] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.81, 4.22, 2.69 [13:27:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.03, 11.56, 12.00 [13:27:53] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mrlove.wiki' expires in 15 day(s) (Wed 10 May 2023 13:21:43 GMT +0000). [13:28:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.84, 11.63, 11.64 [13:28:29] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a9175ab6918b...55ff7f38fe5a [13:28:31] PROBLEM - electowiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'electowiki.org' expires in 15 day(s) (Wed 10 May 2023 13:17:22 GMT +0000). [13:28:31] [02miraheze/ssl] 07MirahezeSSLBot 0355ff7f3 - Bot: Update SSL cert for nonciclopedia.org [13:29:02] PROBLEM - bebaskanpengetahuan.id - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bebaskanpengetahuan.id' expires in 15 day(s) (Wed 10 May 2023 13:20:16 GMT +0000). [13:29:13] PROBLEM - stablestate.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'stablestate.org' expires in 15 day(s) (Wed 10 May 2023 13:17:45 GMT +0000). [13:29:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.05, 10.16, 11.43 [13:30:15] PROBLEM - www.dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 15 day(s) (Wed 10 May 2023 13:15:59 GMT +0000). [13:31:14] PROBLEM - monarchists.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'monarchists.wiki' expires in 15 day(s) (Wed 10 May 2023 13:17:37 GMT +0000). [13:31:34] PROBLEM - wiki.nowchess.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.nowchess.org' expires in 15 day(s) (Wed 10 May 2023 13:19:30 GMT +0000). [13:31:35] PROBLEM - vitriol.spoon.army - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vitriol.spoon.army' expires in 15 day(s) (Wed 10 May 2023 13:22:46 GMT +0000). [13:31:36] PROBLEM - www.programming.red - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'programming.red' expires in 15 day(s) (Wed 10 May 2023 13:23:12 GMT +0000). [13:32:05] PROBLEM - dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 15 day(s) (Wed 10 May 2023 13:15:59 GMT +0000). [13:32:17] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/55ff7f38fe5a...5f8c86f42355 [13:32:18] [02miraheze/ssl] 07MirahezeSSLBot 035f8c86f - Bot: Update SSL cert for wiki.mikrodev.com [13:32:34] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.41, 3.02, 2.29 [13:32:44] PROBLEM - podpedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'podpedia.org' expires in 15 day(s) (Wed 10 May 2023 13:15:47 GMT +0000). [13:34:20] PROBLEM - ao90.pinho.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ao90.pinho.org' expires in 15 day(s) (Wed 10 May 2023 13:21:28 GMT +0000). [13:34:21] PROBLEM - mh142.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mh142.com' expires in 15 day(s) (Wed 10 May 2023 13:20:47 GMT +0000). [13:34:49] PROBLEM - wiki.conworlds.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.conworlds.org' expires in 15 day(s) (Wed 10 May 2023 13:23:21 GMT +0000). [13:34:58] RECOVERY - cp33 APT on cp33 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [13:35:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5f8c86f42355...b11a4c1b2ae7 [13:35:13] [02miraheze/ssl] 07MirahezeSSLBot 03b11a4c1 - Bot: Update SSL cert for runzeppelin.ru [13:35:36] PROBLEM - wiki.joust.ro - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.joust.ro' expires in 15 day(s) (Wed 10 May 2023 13:17:56 GMT +0000). [13:36:33] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.83, 3.96, 2.80 [13:37:20] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 4.03, 3.31, 2.07 [13:37:20] PROBLEM - wiki.starship.digital - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.starship.digital' expires in 15 day(s) (Wed 10 May 2023 13:22:04 GMT +0000). [13:37:23] PROBLEM - www.lab612.at - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.lab612.at' expires in 15 day(s) (Wed 10 May 2023 13:16:27 GMT +0000). [13:37:34] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 13.60, 8.98, 6.08 [13:37:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.59, 10.69, 10.91 [13:38:02] PROBLEM - www.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Wed 10 May 2023 13:20:37 GMT +0000). [13:38:11] PROBLEM - wiki.whentheycry.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.whentheycry.org' expires in 15 day(s) (Wed 10 May 2023 13:16:55 GMT +0000). [13:38:37] PROBLEM - hi.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hi.gyaanipedia.com' expires in 15 day(s) (Wed 10 May 2023 13:18:10 GMT +0000). [13:38:47] PROBLEM - wiki.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Wed 10 May 2023 13:20:37 GMT +0000). [13:39:20] RECOVERY - puppet141 Current Load on puppet141 is OK: OK - load average: 2.20, 2.87, 2.06 [13:39:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.93, 3.45, 3.58 [13:39:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.06, 10.82, 10.94 [13:40:11] PROBLEM - meta.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'meta.gyaanipedia.com' expires in 15 day(s) (Wed 10 May 2023 13:19:56 GMT +0000). [13:40:19] PROBLEM - guia.cineastas.pt - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'guia.cineastas.pt' expires in 15 day(s) (Wed 10 May 2023 13:21:36 GMT +0000). [13:40:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b11a4c1b2ae7...34174ab44106 [13:40:37] [02miraheze/ssl] 07MirahezeSSLBot 0334174ab - Bot: Update SSL cert for stablestate.org [13:40:42] PROBLEM - kunwok.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kunwok.org' expires in 15 day(s) (Wed 10 May 2023 13:16:16 GMT +0000). [13:41:29] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.25, 7.63, 6.22 [13:41:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.35, 2.78, 3.31 [13:42:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.87, 8.59, 10.12 [13:42:06] PROBLEM - vise.dayid.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vise.dayid.org' expires in 15 day(s) (Wed 10 May 2023 13:19:47 GMT +0000). [13:42:31] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.67, 3.89, 3.24 [13:42:50] PROBLEM - mr.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mr.gyaanipedia.com' expires in 15 day(s) (Wed 10 May 2023 13:18:29 GMT +0000). [13:43:05] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/34174ab44106...ce3392c60bb5 [13:43:07] [02miraheze/ssl] 07MirahezeSSLBot 03ce3392c - Bot: Update SSL cert for wiki.starship.digital [13:43:26] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.23, 6.47, 5.97 [13:44:17] PROBLEM - vmcodex.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vmcodex.net' expires in 15 day(s) (Wed 10 May 2023 13:19:40 GMT +0000). [13:44:31] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.47, 3.05, 3.01 [13:44:38] PROBLEM - www.mh142.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mh142.com' expires in 15 day(s) (Wed 10 May 2023 13:20:47 GMT +0000). [13:45:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.11, 9.06, 10.15 [13:46:03] PROBLEM - tep.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tep.wiki' expires in 15 day(s) (Wed 10 May 2023 13:22:37 GMT +0000). [13:46:28] PROBLEM - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'encyclopediaofastrobiology.org' expires in 15 day(s) (Wed 10 May 2023 13:17:14 GMT +0000). [13:47:05] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14592 MB (10% inode=99%); [13:47:15] PROBLEM - en.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.com' expires in 15 day(s) (Wed 10 May 2023 13:18:03 GMT +0000). [13:47:33] PROBLEM - commons.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'commons.gyaanipedia.com' expires in 15 day(s) (Wed 10 May 2023 13:19:10 GMT +0000). [13:48:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.77, 9.59, 9.99 [13:48:34] PROBLEM - www.cgradegames.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.cgradegames.net All nameservers failed to answer the query. [13:48:50] PROBLEM - programming.red - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'programming.red' expires in 15 day(s) (Wed 10 May 2023 13:23:12 GMT +0000). [13:49:05] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ce3392c60bb5...449fd25c019b [13:49:07] [02miraheze/ssl] 07MirahezeSSLBot 03449fd25 - Bot: Update SSL cert for wiki.graalmilitary.com [13:49:14] PROBLEM - pa.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pa.gyaanipedia.com' expires in 15 day(s) (Wed 10 May 2023 13:18:22 GMT +0000). [13:49:37] RECOVERY - wiki.mikrodev.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mikrodev.com' will expire on Sun 23 Jul 2023 12:32:08 GMT +0000. [13:49:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.34, 10.68, 10.54 [13:50:03] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.23, 11.73, 10.73 [13:50:04] PROBLEM - wiki.s23.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.s23.org' expires in 15 day(s) (Wed 10 May 2023 13:18:40 GMT +0000). [13:50:59] PROBLEM - data.nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'data.nonbinary.wiki' expires in 15 day(s) (Wed 10 May 2023 13:22:12 GMT +0000). [13:51:24] PROBLEM - cyberlaw.ccdcoe.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'cyberlaw.ccdcoe.org' expires in 15 day(s) (Wed 10 May 2023 13:21:16 GMT +0000). [13:51:41] PROBLEM - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mxlinuxusers.de' expires in 15 day(s) (Wed 10 May 2023 13:21:05 GMT +0000). [13:52:30] PROBLEM - wiki.worldsofweary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.worldsofweary.com' expires in 15 day(s) (Wed 10 May 2023 13:22:53 GMT +0000). [13:53:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.88, 3.52, 3.12 [13:54:09] RECOVERY - nonciclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nonciclopedia.org' will expire on Sun 23 Jul 2023 12:28:22 GMT +0000. [13:55:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.16, 3.63, 3.23 [13:56:33] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/449fd25c019b...e669c29ce37e [13:56:36] [02miraheze/ssl] 07MirahezeSSLBot 03e669c29 - Bot: Update SSL cert for meta.gyaanipedia.com [13:57:47] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.67, 3.81, 3.33 [13:58:42] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [13:59:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e669c29ce37e...d2c31e136f8d [13:59:55] [02miraheze/ssl] 07MirahezeSSLBot 03d2c31e1 - Bot: Update SSL cert for wiki.scvo.org [14:01:02] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [14:01:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.63, 3.86, 3.49 [14:02:28] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d2c31e136f8d...03497ce42f91 [14:02:30] [02miraheze/ssl] 07MirahezeSSLBot 0303497ce - Bot: Update SSL cert for pa.gyaanipedia.com [14:03:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.40, 3.00, 3.22 [14:06:40] RECOVERY - wiki.starship.digital - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.starship.digital' will expire on Sun 23 Jul 2023 12:42:55 GMT +0000. [14:10:24] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/03497ce42f91...55fd50534823 [14:10:27] [02miraheze/ssl] 07MirahezeSSLBot 0355fd505 - Bot: Update SSL cert for wiki.thunis.eu [14:10:29] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'apocrypha.wiki' expires in 15 day(s) (Wed 10 May 2023 13:49:35 GMT +0000). [14:13:16] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.06, 3.62, 3.17 [14:14:27] PROBLEM - bristolstudenthousingcoop.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bristolstudenthousingcoop.org' expires in 15 day(s) (Wed 10 May 2023 13:52:45 GMT +0000). [14:15:14] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.69, 3.33, 3.12 [14:15:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/55fd50534823...0022d58fb79b [14:15:58] [02miraheze/ssl] 07MirahezeSSLBot 030022d58 - Bot: Update SSL cert for mh142.com [14:17:42] RECOVERY - pwiki.arkcls.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.arkcls.com' will expire on Sun 23 Jul 2023 12:23:15 GMT +0000. [14:17:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.97, 11.00, 11.80 [14:19:18] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0022d58fb79b...1da9f776492d [14:19:19] [02miraheze/ssl] 07MirahezeSSLBot 031da9f77 - Bot: Update SSL cert for wiki.s23.org [14:19:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.89, 13.05, 12.44 [14:20:06] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Sun 23 Jul 2023 12:25:55 GMT +0000. [14:20:34] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [14:24:10] RECOVERY - runzeppelin.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'runzeppelin.ru' will expire on Sun 23 Jul 2023 12:35:00 GMT +0000. [14:24:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1da9f776492d...aac11e97d072 [14:24:53] [02miraheze/ssl] 07MirahezeSSLBot 03aac11e9 - Bot: Update SSL cert for cyberlaw.ccdcoe.org [14:27:03] RECOVERY - wiki.scvo.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.scvo.org' will expire on Sun 23 Jul 2023 12:59:46 GMT +0000. [14:27:12] RECOVERY - stablestate.org - LetsEncrypt on sslhost is OK: OK - Certificate 'stablestate.org' will expire on Sun 23 Jul 2023 12:40:25 GMT +0000. [14:28:03] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=sagan4masonwiki (END - exit=0) [14:28:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:28:13] !log [alex@mwtask141] starting deploy of {'files': '../mediawiki/w/extensions/SemanticMediaWiki/.smw.json'} to all [14:28:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:28:33] !log [alex@mwtask141] finished deploy of {'files': '../mediawiki/w/extensions/SemanticMediaWiki/.smw.json'} to all - SUCCESS in 20s [14:28:42] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:28:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:30:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/aac11e97d072...2b1af616e1de [14:30:46] [02miraheze/ssl] 07MirahezeSSLBot 032b1af61 - Bot: Update SSL cert for dariawiki.org [14:31:33] Orange_Star: do you have shell access right now? [14:31:44] yeah [14:33:02] RECOVERY - mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Sun 23 Jul 2023 13:15:49 GMT +0000. [14:33:04] Can you run `sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=ted52wiki 'Ciencia Al Poder' 'Ciencia Al Poder (usurped)'`? [14:33:11] A global rename got stuck [14:33:41] Actually, can you run `sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=ted52wiki --logwiki=metawiki 'Ciencia Al Poder' 'Ciencia Al Poder (usurped)'`? [14:33:51] forgot to add the "logwiki" parameter [14:34:29] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=ted52wiki --logwiki=metawiki 'Ciencia Al Poder' 'Ciencia Al Poder (usurped)' [14:34:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:35:05] Thanks [14:35:11] It didn't work [14:35:25] Yeah, still says failed [14:35:37] https://wikitech.wikimedia.org/wiki/Stuckglobalrenames [14:35:47] This article describes how WMF handles stuck global renames [14:36:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2b1af616e1de...27fcbeda524e [14:36:25] [02miraheze/ssl] 07MirahezeSSLBot 0327fcbed - Bot: Update SSL cert for programming.red [14:37:56] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/27fcbeda524e...be27e513aaec [14:37:57] [02miraheze/ssl] 07MirahezeSSLBot 03be27e51 - Bot: Update SSL cert for wiki.pablojprieto.com.es [14:38:34] RECOVERY - meta.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'meta.gyaanipedia.com' will expire on Sun 23 Jul 2023 12:56:26 GMT +0000. [14:38:44] looks like it's a problem with one of the extensions in that wiki? [14:39:18] with voteNY to be more exact [14:40:44] oh, it's a private wiki [14:42:06] Orange_Star: SQL should be fixed if you disable and re-enable the extension on that wiki [14:43:57] I'll give that a shot [14:44:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/be27e513aaec...ed831298acb4 [14:44:27] [02miraheze/ssl] 07MirahezeSSLBot 03ed83129 - Bot: Update SSL cert for tep.wiki [14:44:42] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:45:05] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=ted52wiki --logwiki=metawiki 'Ciencia Al Poder' 'Ciencia Al Poder (usurped)' [14:45:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:45:14] yeah boy [14:45:22] Agent: fixed [14:46:43] RECOVERY - cp33 APT on cp33 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [14:46:51] RECOVERY - programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Sun 23 Jul 2023 13:36:14 GMT +0000. [14:47:40] RECOVERY - pa.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pa.gyaanipedia.com' will expire on Sun 23 Jul 2023 13:02:20 GMT +0000. [14:48:33] RECOVERY - www.cgradegames.net - reverse DNS on sslhost is OK: SSL OK - www.cgradegames.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [14:48:44] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ed831298acb4...425c921a2c98 [14:48:45] [02miraheze/ssl] 07MirahezeSSLBot 03425c921 - Bot: Update SSL cert for kunwok.org [14:49:17] Orange_Star: thank you so much! [14:49:37] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [14:50:51] RECOVERY - wiki.thunis.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thunis.eu' will expire on Sun 23 Jul 2023 13:10:12 GMT +0000. [14:51:33] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/425c921a2c98...e990f015b3d6 [14:51:35] [02miraheze/ssl] 07MirahezeSSLBot 03e990f01 - Bot: Update SSL cert for encyclopediaofastrobiology.org [14:54:41] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.19, 4.97, 3.05 [14:54:51] RECOVERY - wiki.pablojprieto.com.es - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pablojprieto.com.es' will expire on Sun 23 Jul 2023 13:37:48 GMT +0000. [14:55:19] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 4.12, 4.88, 3.13 [14:55:42] RECOVERY - wiki.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.graalmilitary.com' will expire on Sun 23 Jul 2023 12:48:57 GMT +0000. [14:55:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.17, 3.43, 2.72 [14:56:38] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 6.12, 5.58, 3.51 [14:57:20] PROBLEM - puppet141 Current Load on puppet141 is WARNING: WARNING - load average: 1.82, 3.77, 2.93 [14:57:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.95, 3.48, 2.82 [14:58:08] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e990f015b3d6...0f6dee9182eb [14:58:11] [02miraheze/ssl] 07MirahezeSSLBot 030f6dee9 - Bot: Update SSL cert for wiki.valkyrienskies.org [14:58:42] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [14:58:44] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0f6dee9182eb...08d025af8504 [14:58:47] [02miraheze/ssl] 07MirahezeSSLBot 0308d025a - Bot: Update SSL cert for ao90.pinho.org [14:59:19] RECOVERY - puppet141 Current Load on puppet141 is OK: OK - load average: 1.44, 2.92, 2.72 [14:59:34] RECOVERY - www.dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Sun 23 Jul 2023 13:30:36 GMT +0000. [14:59:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.37, 4.75, 3.34 [15:00:05] RECOVERY - dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Sun 23 Jul 2023 13:30:36 GMT +0000. [15:00:23] RECOVERY - www.programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Sun 23 Jul 2023 13:36:14 GMT +0000. [15:02:03] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:04:01] RECOVERY - cp33 APT on cp33 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [15:04:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/08d025af8504...c45efb032ef9 [15:04:57] [02miraheze/ssl] 07MirahezeSSLBot 03c45efb0 - Bot: Update SSL cert for data.nonbinary.wiki [15:05:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.99, 3.28, 3.19 [15:11:34] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.40, 3.65, 3.09 [15:12:27] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c45efb032ef9...9adcd66783d3 [15:12:29] [02miraheze/ssl] 07MirahezeSSLBot 039adcd66 - Bot: Update SSL cert for mr.gyaanipedia.com [15:12:38] RECOVERY - www.mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Sun 23 Jul 2023 13:15:49 GMT +0000. [15:13:32] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.08, 3.51, 3.11 [15:14:04] RECOVERY - tep.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tep.wiki' will expire on Sun 23 Jul 2023 13:44:17 GMT +0000. [15:15:30] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.16, 3.36, 3.10 [15:15:56] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [15:17:02] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9adcd66783d3...186a954b487d [15:17:03] [02miraheze/ssl] 07MirahezeSSLBot 03186a954 - Bot: Update SSL cert for guia.cineastas.pt [15:18:06] RECOVERY - wiki.s23.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.s23.org' will expire on Sun 23 Jul 2023 13:19:08 GMT +0000. [15:18:18] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/186a954b487d...78c1233787f0 [15:18:20] [02miraheze/ssl] 07MirahezeSSLBot 0378c1233 - Bot: Update SSL cert for apocrypha.wiki [15:21:22] RECOVERY - cyberlaw.ccdcoe.org - LetsEncrypt on sslhost is OK: OK - Certificate 'cyberlaw.ccdcoe.org' will expire on Sun 23 Jul 2023 13:24:43 GMT +0000. [15:23:04] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/78c1233787f0...b64e0b5edb95 [15:23:05] [02miraheze/ssl] 07MirahezeSSLBot 03b64e0b5 - Bot: Update SSL cert for wiki.conworlds.org [15:23:50] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1474405825 secs [15:24:14] I didn't know we had wikis like that here [15:24:24] cyberlaw.ccdcoe.org I mean [15:25:20] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b64e0b5edb95...3c04c3b2eced [15:25:21] [02miraheze/ssl] 07MirahezeSSLBot 033c04c3b - Bot: Update SSL cert for bristolstudenthousingcoop.org [15:25:36] RECOVERY - wiki.valkyrienskies.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.valkyrienskies.org' will expire on Sun 23 Jul 2023 13:58:00 GMT +0000. [15:27:05] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 15093 MB (11% inode=99%); [15:27:13] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3c04c3b2eced...197b807a6f57 [15:27:16] [02miraheze/ssl] 07MirahezeSSLBot 03197b807 - Bot: Update SSL cert for monarchists.wiki [15:28:44] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [15:29:09] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/197b807a6f57...3fc83203d0e4 [15:29:10] [02miraheze/ssl] 07MirahezeSSLBot 033fc8320 - Bot: Update SSL cert for vitriol.spoon.army [15:32:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.83, 8.27, 6.56 [15:32:20] RECOVERY - ao90.pinho.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ao90.pinho.org' will expire on Sun 23 Jul 2023 13:58:36 GMT +0000. [15:34:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.74, 7.29, 6.41 [15:36:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3fc83203d0e4...3ae4be851b92 [15:36:28] [02miraheze/ssl] 07MirahezeSSLBot 033ae4be8 - Bot: Update SSL cert for wiki.ciptamedia.org [15:38:03] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.39, 5.98, 6.12 [15:38:21] RECOVERY - guia.cineastas.pt - LetsEncrypt on sslhost is OK: OK - Certificate 'guia.cineastas.pt' will expire on Sun 23 Jul 2023 14:16:55 GMT +0000. [15:38:57] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Sun 23 Jul 2023 14:18:11 GMT +0000. [15:39:29] RECOVERY - kunwok.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kunwok.org' will expire on Sun 23 Jul 2023 13:48:37 GMT +0000. [15:41:50] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07278990746 secs [15:43:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3ae4be851b92...f1b296672a6a [15:43:47] [02miraheze/ssl] 07MirahezeSSLBot 03f1b2966 - Bot: Update SSL cert for en.gyaanipedia.com [15:44:30] RECOVERY - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediaofastrobiology.org' will expire on Sun 23 Jul 2023 13:51:25 GMT +0000. [15:44:52] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:47:03] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f1b296672a6a...421e2ed9b6e8 [15:47:05] [02miraheze/ssl] 07MirahezeSSLBot 03421e2ed - Bot: Update SSL cert for podpedia.org [15:48:59] RECOVERY - data.nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'data.nonbinary.wiki' will expire on Sun 23 Jul 2023 14:04:46 GMT +0000. [15:52:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/421e2ed9b6e8...af88377f3407 [15:52:19] [02miraheze/ssl] 07MirahezeSSLBot 03af88377 - Bot: Update SSL cert for wiki.nowchess.org [15:54:04] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.29, 4.70, 3.41 [15:54:15] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/af88377f3407...be8386a2fd4b [15:54:18] [02miraheze/ssl] 07MirahezeSSLBot 03be8386a - Bot: Update SSL cert for wiki.whentheycry.org [15:56:01] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.82, 3.96, 3.29 [15:56:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/be8386a2fd4b...249ee61f4960 [15:56:37] [02miraheze/ssl] 07MirahezeSSLBot 03249ee61 - Bot: Update SSL cert for www.thesimswiki.com [15:57:57] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1295392513 secs [15:57:58] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.59, 3.21, 3.10 [16:00:07] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 12.00, 7.70, 4.09 [16:00:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/249ee61f4960...ae2c354b903f [16:00:28] [02miraheze/ssl] 07MirahezeSSLBot 03ae2c354 - Bot: Update SSL cert for hi.gyaanipedia.com [16:03:58] RECOVERY - wiki.conworlds.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.conworlds.org' will expire on Sun 23 Jul 2023 14:22:55 GMT +0000. [16:06:02] RECOVERY - www.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Sun 23 Jul 2023 14:56:27 GMT +0000. [16:06:06] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.34, 3.25, 3.20 [16:07:33] RECOVERY - wiki.whentheycry.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.whentheycry.org' will expire on Sun 23 Jul 2023 14:54:07 GMT +0000. [16:08:02] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.21, 3.63, 3.12 [16:08:27] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ae2c354b903f...cc1238df28eb [16:08:30] [02miraheze/ssl] 07MirahezeSSLBot 03cc1238d - Bot: Update SSL cert for vmcodex.net [16:09:50] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09552872181 secs [16:09:57] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.19, 3.42, 3.10 [16:10:50] RECOVERY - mr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mr.gyaanipedia.com' will expire on Sun 23 Jul 2023 14:12:19 GMT +0000. [16:11:43] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.32, 4.05, 3.21 [16:12:15] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.41, 5.71, 2.99 [16:12:27] RECOVERY - bristolstudenthousingcoop.org - LetsEncrypt on sslhost is OK: OK - Certificate 'bristolstudenthousingcoop.org' will expire on Sun 23 Jul 2023 14:25:13 GMT +0000. [16:13:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.71, 9.87, 11.92 [16:13:49] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.95, 3.23, 3.09 [16:13:59] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [16:15:15] RECOVERY - en.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.com' will expire on Sun 23 Jul 2023 14:43:38 GMT +0000. [16:15:19] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 5.20, 5.31, 3.18 [16:15:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cc1238df28eb...8b8ef715429a [16:15:37] [02miraheze/ssl] 07MirahezeSSLBot 038b8ef71 - Bot: Update SSL cert for wiki.joust.ro [16:15:42] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=tttewikiwiki (END - exit=0) [16:15:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:15:53] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=tttewikiwiki --active --update (END - exit=0) [16:16:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:16:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.02, 6.67, 4.15 [16:16:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.56, 7.40, 6.67 [16:17:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8b8ef715429a...3dc3a3743612 [16:17:07] [02miraheze/ssl] 07MirahezeSSLBot 033dc3a37 - Bot: Update SSL cert for www.lab612.at [16:17:20] PROBLEM - puppet141 Current Load on puppet141 is WARNING: WARNING - load average: 1.78, 3.93, 2.93 [16:17:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.46, 11.08, 11.90 [16:18:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3dc3a3743612...f7ef16aacde1 [16:18:12] [02miraheze/ssl] 07MirahezeSSLBot 03f7ef16a - Bot: Update SSL cert for wiki.openhatch.org [16:19:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f7ef16aacde1...5dd09ba66021 [16:19:07] [02miraheze/ssl] 07MirahezeSSLBot 035dd09ba - Bot: Update SSL cert for wiki.worldsofweary.com [16:19:19] RECOVERY - puppet141 Current Load on puppet141 is OK: OK - load average: 1.24, 3.00, 2.71 [16:19:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.45, 10.99, 11.76 [16:21:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.54, 12.39, 12.18 [16:22:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.37, 7.33, 6.95 [16:23:15] RECOVERY - wiki.ciptamedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ciptamedia.org' will expire on Sun 23 Jul 2023 14:36:17 GMT +0000. [16:24:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.96, 7.99, 7.24 [16:25:17] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5dd09ba66021...cc417dc21209 [16:25:20] [02miraheze/ssl] 07MirahezeSSLBot 03cc417dc - Bot: Update SSL cert for wiki.zymonic.com [16:26:27] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.48, 3.47, 3.31 [16:26:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.09, 7.74, 7.24 [16:27:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.14, 11.57, 11.96 [16:28:22] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.66, 3.20, 3.23 [16:29:14] RECOVERY - monarchists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'monarchists.wiki' will expire on Sun 23 Jul 2023 14:27:05 GMT +0000. [16:29:32] RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Sun 23 Jul 2023 14:52:09 GMT +0000. [16:29:35] RECOVERY - vitriol.spoon.army - LetsEncrypt on sslhost is OK: OK - Certificate 'vitriol.spoon.army' will expire on Sun 23 Jul 2023 14:29:01 GMT +0000. [16:30:16] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.74, 6.85, 5.30 [16:30:47] RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Sun 23 Jul 2023 14:46:55 GMT +0000. [16:31:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cc417dc21209...e9009289b835 [16:31:43] [02miraheze/ssl] 07MirahezeSSLBot 03e900928 - Bot: Update SSL cert for bebaskanpengetahuan.id [16:32:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.37, 7.79, 7.43 [16:33:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.51, 3.76, 3.96 [16:34:15] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.37, 7.76, 6.01 [16:34:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.93, 7.63, 7.40 [16:35:17] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.10, 11.32, 11.54 [16:35:47] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e9009289b835...a1512a408a44 [16:35:48] [02miraheze/ssl] 07MirahezeSSLBot 03a1512a4 - Bot: Update SSL cert for electowiki.org [16:36:37] RECOVERY - hi.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'hi.gyaanipedia.com' will expire on Sun 23 Jul 2023 15:00:17 GMT +0000. [16:36:47] RECOVERY - wiki.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Sun 23 Jul 2023 14:56:27 GMT +0000. [16:37:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.06, 9.85, 10.99 [16:38:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.56, 7.68, 7.44 [16:38:16] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.01, 7.83, 6.50 [16:38:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a1512a408a44...cda5f3587a37 [16:38:24] [02miraheze/ssl] 07MirahezeSSLBot 03cda5f35 - Bot: Update SSL cert for mrlove.wiki [16:38:59] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cda5f3587a37...e76c0d47979e [16:39:00] [02miraheze/ssl] 07MirahezeSSLBot 03e76c0d4 - Bot: Update SSL cert for wiki.mxlinuxusers.de [16:40:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.37, 7.25, 7.31 [16:40:15] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.09, 9.32, 7.22 [16:40:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e76c0d47979e...ed84d9399260 [16:40:44] [02miraheze/ssl] 07MirahezeSSLBot 03ed84d93 - Bot: Update SSL cert for commons.gyaanipedia.com [16:41:19] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.29, 4.00, 3.97 [16:42:17] RECOVERY - vmcodex.net - LetsEncrypt on sslhost is OK: OK - Certificate 'vmcodex.net' will expire on Sun 23 Jul 2023 15:08:19 GMT +0000. [16:43:50] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset -0.108797878 secs [16:43:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ed84d9399260...a534c6b2d1df [16:43:55] [02miraheze/ssl] 07MirahezeSSLBot 03a534c6b - Bot: Update SSL cert for www.portalsofphereon.com [16:45:14] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.02, 9.26, 10.18 [16:47:01] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a534c6b2d1df...2ecbac7315db [16:47:03] [02miraheze/ssl] 07MirahezeSSLBot 032ecbac7 - Bot: Update SSL cert for vise.dayid.org [16:47:29] [02MirahezeMagic] 07Reception123 synchronize pull request 03#419: add script to download/upload swift containers for DataDump compatibi… - 13https://github.com/miraheze/MirahezeMagic/pull/419 [16:47:30] [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/ee02e9804247...d05b05303ef2 [16:47:32] [02miraheze/MirahezeMagic] 07Reception123 03d05b053 - Update swiftDump.php [16:47:50] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.0902929306 secs [16:48:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.34, 7.51, 7.27 [16:49:41] RECOVERY - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mxlinuxusers.de' will expire on Sun 23 Jul 2023 15:38:49 GMT +0000. [16:50:15] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 3.62, 7.10, 7.23 [16:50:30] miraheze/MirahezeMagic - Reception123 the build has errored. [16:50:34] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:51:11] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.76, 3.89, 3.99 [16:52:12] !log reception@mwtask141:~/narutoes$ sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php /home/reception/narutoes --wiki=esnarutowiki --user=Miraheze_maintenance_script --summary="Imported from naruto.fandom.com/es" --search-recursively [16:52:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:53:09] PROBLEM - td10.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'td10.org' expires in 15 day(s) (Wed 10 May 2023 16:25:33 GMT +0000). [16:54:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.38, 5.11, 6.49 [16:55:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.41, 2.75, 2.28 [16:55:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.25, 10.72, 11.75 [16:55:53] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Sun 23 Jul 2023 15:38:14 GMT +0000. [16:56:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.61, 7.83, 7.73 [16:56:31] RECOVERY - electowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electowiki.org' will expire on Sun 23 Jul 2023 15:35:40 GMT +0000. [16:56:51] RECOVERY - wiki.openhatch.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.openhatch.org' will expire on Sun 23 Jul 2023 15:18:02 GMT +0000. [16:57:06] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.22, 2.65, 3.39 [16:57:37] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.15, 3.29, 2.54 [16:58:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.74, 8.28, 7.90 [16:58:51] RECOVERY - bebaskanpengetahuan.id - LetsEncrypt on sslhost is OK: OK - Certificate 'bebaskanpengetahuan.id' will expire on Sun 23 Jul 2023 15:31:34 GMT +0000. [16:59:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2ecbac7315db...7295fa41ea0a [16:59:26] [02miraheze/ssl] 07MirahezeSSLBot 037295fa4 - Bot: Update SSL cert for td10.org [17:01:29] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.10, 2.99, 2.61 [17:03:01] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.11, 3.28, 3.43 [17:03:36] RECOVERY - wiki.joust.ro - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.joust.ro' will expire on Sun 23 Jul 2023 15:15:27 GMT +0000. [17:04:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.63, 7.53, 7.83 [17:04:59] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.92, 3.15, 3.37 [17:05:23] RECOVERY - www.lab612.at - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lab612.at' will expire on Sun 23 Jul 2023 15:16:58 GMT +0000. [17:06:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.25, 7.60, 7.81 [17:07:04] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14595 MB (10% inode=99%); [17:07:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.55, 9.78, 9.49 [17:09:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.05, 9.34, 9.35 [17:09:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.06, 10.15, 10.47 [17:13:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.99, 10.63, 10.62 [17:15:33] RECOVERY - commons.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'commons.gyaanipedia.com' will expire on Sun 23 Jul 2023 15:40:34 GMT +0000. [17:17:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.06, 9.34, 10.11 [17:19:38] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [17:20:30] RECOVERY - wiki.worldsofweary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worldsofweary.com' will expire on Sun 23 Jul 2023 15:18:59 GMT +0000. [17:21:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.75, 10.52, 10.40 [17:23:19] RECOVERY - www.portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Sun 23 Jul 2023 15:43:44 GMT +0000. [17:24:20] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Sun 23 Jul 2023 15:25:04 GMT +0000. [17:24:56] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.05, 4.29, 3.10 [17:25:10] RECOVERY - portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Sun 23 Jul 2023 15:43:44 GMT +0000. [17:25:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.65, 9.39, 9.97 [17:26:54] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.96, 3.72, 3.03 [17:30:49] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.29, 2.82, 2.87 [17:32:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.91, 7.06, 7.76 [17:35:34] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1331857145 secs [17:36:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.88, 8.13, 7.96 [17:37:30] [02mw-config] 07AgentIsai opened pull request 03#5208: Changes to CentralAuth config - 13https://github.com/miraheze/mw-config/pull/5208 [17:38:37] miraheze/mw-config - AgentIsai the build passed. [17:40:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.93, 7.50, 7.72 [17:40:08] RECOVERY - vise.dayid.org - LetsEncrypt on sslhost is OK: OK - Certificate 'vise.dayid.org' will expire on Sun 23 Jul 2023 15:46:52 GMT +0000. [17:50:36] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:51:10] RECOVERY - td10.org - LetsEncrypt on sslhost is OK: OK - Certificate 'td10.org' will expire on Sun 23 Jul 2023 15:59:18 GMT +0000. [17:54:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.84, 10.60, 9.87 [17:56:21] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.72, 10.07, 9.75 [17:59:00] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:02:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.89, 8.02, 7.63 [18:02:11] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.75, 11.32, 10.45 [18:04:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.87, 7.15, 7.36 [18:06:12] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.93, 3.32, 3.07 [18:07:34] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.07452994585 secs [18:07:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.07, 11.47, 10.75 [18:08:07] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.64, 3.00, 2.98 [18:09:38] [02mw-config] 07Reception123 closed pull request 03#5208: Changes to CentralAuth config - 13https://github.com/miraheze/mw-config/pull/5208 [18:09:39] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/49164c23a8d3...988bf73edef5 [18:09:41] [02miraheze/mw-config] 07AgentIsai 03988bf73 - Changes to CentralAuth config (#5208) [18:09:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.86, 11.27, 10.76 [18:10:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.73, 7.25, 7.25 [18:10:26] [02MirahezeMagic] 07Reception123 closed pull request 03#420: Cleanup and organize group names - 13https://github.com/miraheze/MirahezeMagic/pull/420 [18:10:28] [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/25972ef2c2f5...58223ca242bd [18:10:30] [02miraheze/MirahezeMagic] 07AgentIsai 0358223ca - Cleanup and organize group names (#420) [18:10:38] miraheze/mw-config - Reception123 the build passed. [18:11:31] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.65, 3.77, 2.93 [18:11:49] !log [@mwtask141] starting deploy of {'config': True} to all [18:11:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.86, 13.50, 11.62 [18:11:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:11:58] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [18:12:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.89, 7.26, 7.24 [18:12:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:14:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.62, 7.55, 7.34 [18:15:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.79, 10.46, 9.61 [18:17:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.96, 3.99, 3.32 [18:17:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.58, 10.11, 9.59 [18:19:24] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.47, 4.15, 3.46 [18:20:46] miraheze/MirahezeMagic - Reception123 the build passed. [18:21:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.80, 3.96, 3.47 [18:21:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.43, 10.54, 11.23 [18:25:20] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.18, 3.94, 3.56 [18:25:28] [02mw-config] 07Universal-Omega opened pull request 03#5209: Remove some configs - 13https://github.com/miraheze/mw-config/pull/5209 [18:25:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.10, 10.72, 11.11 [18:26:21] miraheze/mw-config - Universal-Omega the build passed. [18:26:48] !log [@test131] starting deploy of {'config': True} to all [18:26:49] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 1s [18:26:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:27:18] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.54, 3.81, 3.56 [18:27:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.81, 10.08, 10.84 [18:28:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.86, 7.65, 7.74 [18:29:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.65, 10.68, 9.95 [18:31:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.06, 10.29, 9.90 [18:32:01] [02ssl] 07Universal-Omega opened pull request 03#646: Remove certs - 13https://github.com/miraheze/ssl/pull/646 [18:33:46] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.42, 11.80, 10.47 [18:33:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.28, 11.93, 11.37 [18:35:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.19, 11.49, 10.56 [18:35:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.90, 11.68, 11.35 [18:36:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.87, 9.26, 8.26 [18:37:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.08, 12.41, 11.67 [18:38:16] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.51, 3.59, 2.58 [18:39:08] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.18, 4.10, 3.72 [18:40:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.73, 3.63, 2.71 [18:42:11] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.79, 3.24, 2.67 [18:43:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.62, 11.75, 10.91 [18:45:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.57, 10.88, 10.69 [18:45:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.93, 11.30, 11.82 [18:49:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.81, 3.99, 3.89 [18:51:03] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.42, 4.25, 3.97 [18:51:41] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.70, 9.43, 10.06 [18:55:15] [02ssl] 07Reception123 closed pull request 03#646: Remove certs - 13https://github.com/miraheze/ssl/pull/646 [18:55:18] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-3/±2] 13https://github.com/miraheze/ssl/compare/7295fa41ea0a...798336626900 [18:55:21] [02miraheze/ssl] 07Universal-Omega 037983366 - Remove certs (#646) [18:55:23] [02mw-config] 07Reception123 closed pull request 03#5209: Remove some configs - 13https://github.com/miraheze/mw-config/pull/5209 [18:55:25] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/988bf73edef5...ebb640c459dc [18:55:28] [02miraheze/mw-config] 07Universal-Omega 03ebb640c - Remove some configs (#5209) [18:55:46] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.zulaclub.net. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:56:20] miraheze/mw-config - Reception123 the build passed. [18:56:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.67, 3.81, 3.19 [18:56:57] !log [@test131] starting deploy of {'config': True} to all [18:56:58] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [18:57:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:58:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.22, 3.25, 3.06 [18:59:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.48, 10.95, 11.03 [19:01:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.33, 10.41, 10.86 [19:05:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.66, 8.76, 10.02 [19:07:27] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.18, 3.67, 3.23 [19:09:25] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.75, 2.99, 3.03 [19:11:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.30, 11.36, 10.64 [19:12:34] !log [@mwtask141] starting deploy of {'config': True} to all [19:12:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:12:47] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 13s [19:12:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:13:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.63, 9.93, 10.20 [19:20:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.31, 10.94, 9.96 [19:21:15] [02ManageWiki] 07Universal-Omega opened pull request 03#394: Add hooks to modify ManageWikiCore - 13https://github.com/miraheze/ManageWiki/pull/394 [19:22:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.03, 9.35, 9.50 [19:24:45] miraheze/ManageWiki - Universal-Omega the build passed. [19:26:08] [02ManageWiki] 07Universal-Omega synchronize pull request 03#394: Add hooks to modify ManageWikiCore - 13https://github.com/miraheze/ManageWiki/pull/394 [19:28:06] [02ManageWiki] 07Universal-Omega synchronize pull request 03#394: Add hooks to modify ManageWikiCore - 13https://github.com/miraheze/ManageWiki/pull/394 [19:29:10] miraheze/ManageWiki - Universal-Omega the build has errored. [19:29:34] [02ManageWiki] 07Universal-Omega synchronize pull request 03#394: Add hooks to modify ManageWikiCore - 13https://github.com/miraheze/ManageWiki/pull/394 [19:30:44] what if we replace [19:30:49] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.11, 3.47, 3.97 [19:30:50] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:31:22] CentralAuth with a login wiki running OAuth and the rest of the wikis running WSOAuth + PlugabbleAuth? [19:31:30] [02ManageWiki] 07Universal-Omega synchronize pull request 03#394: Add hooks to modify ManageWikiCore - 13https://github.com/miraheze/ManageWiki/pull/394 [19:31:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.59, 11.37, 10.32 [19:31:35] miraheze/ManageWiki - Universal-Omega the build has errored. [19:31:36] just a random thought I had [19:32:17] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.58, 11.26, 10.13 [19:32:27] [02ManageWiki] 07Universal-Omega synchronize pull request 03#394: Add hooks to modify ManageWikiCore - 13https://github.com/miraheze/ManageWiki/pull/394 [19:32:59] miraheze/ManageWiki - Universal-Omega the build has errored. [19:33:08] would kill things like CreateWiki (who would also take ManageWiki with it), current OAuth consumers and probably a million other things, but at least we get rid of CentralAuth and have true SSO (better support for WebAuthn) [19:33:31] [02ManageWiki] 07Universal-Omega synchronize pull request 03#394: Add hooks to modify ManageWikiCore - 13https://github.com/miraheze/ManageWiki/pull/394 [19:33:47] it would also kill global renames since that's a CentralAuth thing [19:34:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.74, 11.81, 10.49 [19:35:00] wonder if it would even be possible to do here without breaking everything [19:35:12] miraheze/ManageWiki - Universal-Omega the build has errored. [19:35:22] miraheze/ManageWiki - Universal-Omega the build has errored. [19:35:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.74, 10.96, 10.43 [19:36:49] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.46, 2.62, 3.40 [19:37:17] miraheze/ManageWiki - Universal-Omega the build passed. [19:41:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.72, 12.51, 11.18 [19:43:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.14, 11.72, 11.04 [19:44:00] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.04, 8.93, 9.83 [19:45:15] Orange_Star: eh [19:45:31] That’s kind of what CA will be in the future [19:45:41] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [19:45:44] SUL2 isn’t well supported or liked by browsers [19:45:47] With WSOAuth, the future is now!! [19:45:57] But why the effort [19:46:13] We don’t have a single wiki that’s truth for accounts [19:46:35] just delete every account and have everyone re-register at the new loginwiki, ez /s [19:46:46] Erm something is going very wrong [19:46:49] With swift [19:46:56] Look at the job queue [19:47:15] I thought it was just another refreshLinks storm [19:47:25] There used to be a grafana thing that showed which wiki [19:47:50] can't really see Grafana cause it says no data in the job queue metrics : [19:47:51] :/ [19:48:23] Which thing you looking at [19:48:29] oh it's a lot of thumbnailRender jobs [19:48:34] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [19:48:35] Orange_Star: you should look at logs [19:48:37] the top unclaimed jobs [19:48:44] but sometimes it doesn't load for me [19:48:46] Thumbnails aren’t generating based on that [19:48:50] Oh ye [19:48:57] That panel would be useful now [19:49:02] You got a link to it [19:49:14] But I suggest we open an incident [19:49:18] it's at the bottom, at the link Grafana posted [19:49:31] It never loads on mobile [19:49:39] Can you link the actual panel [19:49:47] https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki?orgId=1&viewPanel=59 [19:50:46] Orange_Star: try runJobs for esnarutowiki [19:51:14] Then tttewikiwiki [19:53:23] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=esnarutowiki (START) [19:53:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.06, 9.48, 10.19 [19:53:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:36] it's running RhinosF1 [19:53:41] @Site Reliability Engineers [19:53:56] Orange_Star: ty [19:54:00] Look for errors [19:54:04] I remember seeing these wikis upload an insane amount of images on #miraheze-feed before [19:54:07] Hopefully it will error [19:54:15] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.02, 6.55, 5.72 [19:54:16] Tttewiki has before [19:54:27] Orange_Star: run ttte at same time [19:54:40] forgor to run screen, so that will have to wait [19:55:09] Just open another terminal [19:55:18] I guess... [19:55:22] You can be logged in to a server more than once [19:55:55] #miraheze-feed is 70 minutes behind [19:56:15] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.63, 6.42, 5.77 [19:56:53] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=tttewikiwiki (START) [19:56:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:14] Orange_Star: any sign of errors or is it chugging along? [19:57:19] running just fine [19:57:38] Curve looks to be flattening [19:57:56] Which is a sign we’re maybe not on fire and just under pressure [19:58:13] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:58:17] That’s a steady build [19:58:38] Which kind of implies we can’t process jobs quick enough [19:58:41] For thumbnails [19:58:44] Bleh [19:59:28] #miraheze-feed looks just fine for me though? [19:59:40] I see a lot of images uploaded via msUpload on esnarutowiki [19:59:45] that's been going on all day then [19:59:58] It’s definitely behind [20:00:05] Match the edit times up to your current one [20:00:30] We’ve had a few spikes for jobs today [20:00:38] But it looks like it went super high very soon [20:00:49] Job queue likes to fall flat on its face if it gets busy though [20:00:52] And do nothing [20:00:57] Which is why we have the alert [20:01:24] Job Queue has 2 modes [20:01:26] On fire [20:01:30] Or world is lovely [20:01:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.39, 10.36, 9.55 [20:01:46] you're right, miraheze-feed its behind [20:02:49] I think on esnarutowiki, since the import was taking a while to be processed (https://phabricator.miraheze.org/T10753), they must have attempted to upload all images via MsUpload [20:02:51] @paladox @Reception123 @Orduin: infra could do with restarting the feed bot ^ [20:03:25] Orange_Star: how lovely of them [20:03:40] but what's the deal with tttewikiwiki then? [20:03:41] Is MsUpload job queue [20:03:45] Or? [20:03:47] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1138929129 secs [20:03:50] I don’t remember how it works [20:04:00] Orange_Star: no idea but there’s a lot of images on it [20:05:07] That’s a very big wiki [20:05:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.55, 11.24, 10.06 [20:05:44] Only a few users [20:05:52] Probably a fork of a fandom wiki [20:06:12] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.23, 10.47, 10.17 [20:07:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.13, 10.36, 9.90 [20:10:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.09, 11.84, 10.76 [20:11:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.52, 11.73, 10.54 [20:11:46] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09570938349 secs [20:12:50] Orange_Star: how much does esnarutowiki have to clear [20:13:14] tttewiki is clearing well [20:13:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.85, 10.50, 10.23 [20:13:46] looks like it's nowhere near finished there [20:14:13] thumbnail jobs on those wikis have only increased, judging by Grafana [20:14:25] *that wiki [20:15:30] I'm seeing that the causeActor for these jobs is the Miraheze maintenance script, just how big was that image import? [20:15:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.56, 11.81, 11.17 [20:16:19] probably still ongoing [20:16:26] Orange_Star: can you check [20:16:32] htop should tell you [20:16:37] Might be worth killing it [20:16:46] It'll be running as www-data [20:16:51] And restart with a delay [20:18:53] yep, import is still running [20:19:23] Orange_Star: kill it and either add --sleep=1 [20:19:27] Or increase it [20:19:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.64, 12.85, 11.66 [20:20:17] looks like it just finished, can't see it anymore [20:20:36] spoke too soon oops [20:21:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.35, 11.18, 10.55 [20:22:09] Orange_Star: you ok killing it? [20:23:21] no, amount of unclaimed jobs is nothing compared with the insane amounts I saw on a SMW wiki [20:23:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.11, 10.31, 10.33 [20:23:37] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [20:23:41] so while Grafana can complain all it wants, it's not really dangerous right now [20:25:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.71, 11.14, 11.37 [20:25:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.09, 9.73, 10.12 [20:25:56] Orange_Star: it really is [20:26:12] You've not seen how quickly the job queue can get a lot worse [20:26:23] And it can impact everyone then [20:26:30] And take hours to recover [20:26:43] The alerts are paging for good reason [20:27:12] Me and Zppix have many stories of babysitting the job runners for hours to get backlogs down [20:27:44] Job speed matters a lot [20:28:08] Some jobs will clear in ms [20:28:14] Thumbnails won't [20:28:23] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=esnarutowiki (END - exit=65280) [20:28:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:28:34] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.46, 3.62, 2.91 [20:28:37] OOM'd [20:28:45] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=esnarutowiki (START) [20:28:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:29:03] That's not a surprise [20:29:09] There's a max memory option [20:29:47] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.49, 3.40, 1.94 [20:30:06] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.69, 3.73, 2.94 [20:30:34] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.96, 3.29, 2.88 [20:30:51] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.42, 3.39, 3.24 [20:31:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.67, 8.00, 9.96 [20:32:49] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.92, 3.18, 3.17 [20:33:46] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.59, 3.66, 2.48 [20:35:46] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.66, 2.62, 2.24 [20:37:54] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.47, 3.33, 3.09 [20:40:49] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.14, 3.61, 3.35 [20:42:49] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.11, 3.51, 3.35 [20:46:49] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.36, 2.95, 3.17 [20:52:06] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.51, 7.11, 6.32 [20:54:06] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.11, 6.85, 6.33 [20:54:34] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.21, 4.46, 3.21 [20:54:47] tttewikiwiki keeps uploading images, though they shouldn't be a problem for now. The fact that they have to go through the webservers to upload it allows us to keep up with it (though we can't actually clear the jobqueue from their jobs) [20:56:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.00, 10.39, 9.47 [20:57:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.98, 10.27, 9.07 [20:58:06] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.65, 6.02, 6.14 [20:58:22] esnarutowiki's memory usage on the jobqueue has been growing steadily, but no major effect then, as the other jobs on the queue aren't really high priority [20:58:55] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=esnarutowiki (END - exit=65280) [20:58:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:59:06] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=esnarutowiki (START) [20:59:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:59:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.34, 10.32, 9.22 [21:00:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.33, 9.54, 9.34 [21:00:34] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.93, 3.32, 3.16 [21:01:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.97, 9.34, 8.99 [21:03:09] it really does look like tttewikiwiki is just uploading an image dump through MsUpload [21:04:08] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=tttewikiwiki (END - exit=0) [21:04:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.82, 10.42, 9.72 [21:04:53] We should probably ToU ban for this, as if no-one was online to run jobs on that wiki it would have taken over the job queue completely [21:05:51] But that wouldn't crash the farm or cause a 5xx, would it? [21:06:01] PROBLEM - www.mcpk.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.mcpk.wiki' expires in 15 day(s) (Wed 10 May 2023 21:02:33 GMT +0000). [21:06:02] also we're actually faster than the image upload, though the script ended the job queue is filling back up with jobs from that wiki [21:06:11] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=tttewikiwiki (START) [21:06:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:07:15] Agent: You could lock their accounts, or IP block them on the webserver, don't see how that would crash anything [21:07:43] Orange_Star: I'm talking about their activities [21:07:57] T&S recommends not invoking the ToU unless whatever is being done will crash the entire farm [21:08:12] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.14, 3.44, 3.06 [21:08:22] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.28, 9.19, 9.41 [21:08:22] otherwise, they recommend doing lesser things like, perhaps in this case, revoking image uploading privs temporarily and sending the bureaucrat a message [21:08:35] Well if you let the jobqueue just fill completely it would stop other jobs from running I imagine, or slow them down a lot, like the wiki creation job [21:09:00] PROBLEM - es.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'es.gyaanipedia.com' expires in 15 day(s) (Wed 10 May 2023 21:03:54 GMT +0000). [21:09:07] not a crash, but could have nasty effects [21:09:11] PROBLEM - wiki.villagecollaborative.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.villagecollaborative.net' expires in 15 day(s) (Wed 10 May 2023 21:03:47 GMT +0000). [21:09:17] PROBLEM - wiki.thehall.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.thehall.xyz' expires in 15 day(s) (Wed 10 May 2023 21:05:26 GMT +0000). [21:09:30] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.02, 3.87, 3.35 [21:09:41] PROBLEM - spcodex.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'spcodex.wiki' expires in 15 day(s) (Wed 10 May 2023 21:03:24 GMT +0000). [21:09:51] If you think it's necessary, you could temporarily disable image uploads on the wiki [21:09:58] PROBLEM - wiki.minkyu.kim - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.minkyu.kim' expires in 15 day(s) (Wed 10 May 2023 21:04:37 GMT +0000). [21:10:01] and send the crat a message [21:10:02] don't care enough [21:10:08] lol [21:10:09] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.70, 2.74, 2.85 [21:10:10] as I can keep up with the uploads manually [21:10:38] But yeah, I think the bureaucrat of tttewiki is trying to upload their 100GB+ image dump manually [21:10:41] PROBLEM - fc.songcontests.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fc.songcontests.eu' expires in 15 day(s) (Wed 10 May 2023 21:01:59 GMT +0000). [21:10:50] PROBLEM - pubwiki.lab.co.pl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pubwiki.lab.co.pl' expires in 15 day(s) (Wed 10 May 2023 21:03:04 GMT +0000). [21:10:56] their what? [21:11:02] PROBLEM - www.iceria.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.iceria.org' expires in 15 day(s) (Wed 10 May 2023 21:04:05 GMT +0000). [21:11:04] 100GB? [21:11:30] PROBLEM - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ricochetuniverse.com' expires in 15 day(s) (Wed 10 May 2023 21:03:36 GMT +0000). [21:11:37] PROBLEM - n64brew.dev - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'n64brew.dev' expires in 15 day(s) (Wed 10 May 2023 21:07:14 GMT +0000). [21:11:46] https://phabricator.miraheze.org/T10760 [21:11:48] I see [21:11:54] PROBLEM - vedopedia.witches-empire.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vedopedia.witches-empire.com' expires in 15 day(s) (Wed 10 May 2023 21:01:52 GMT +0000). [21:12:03] well that changes everything [21:12:09] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/798336626900...1871f261d898 [21:12:10] [02miraheze/ssl] 07MirahezeSSLBot 031871f26 - Bot: Update SSL cert for www.mcpk.wiki [21:12:15] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=tttewikiwiki (END - exit=0) [21:12:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:12:29] Orange_Star: They're migrating ttte.fandom.com which had over 150GBs of images [21:12:32] Per https://ttte.fandom.com/wiki/Special:MediaStatistics [21:12:55] PROBLEM - iceria.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.iceria.org' expires in 15 day(s) (Wed 10 May 2023 21:04:05 GMT +0000). [21:13:43] I don't know if we have that space in Swift though [21:14:19] PROBLEM - wiki.vinesh.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.vinesh.eu.org' expires in 15 day(s) (Wed 10 May 2023 21:04:58 GMT +0000). [21:14:34] PROBLEM - savagepedia.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'savagepedia.wiki' expires in 15 day(s) (Wed 10 May 2023 21:08:37 GMT +0000). [21:15:09] PROBLEM - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.rebirthofthenight.com' expires in 15 day(s) (Wed 10 May 2023 21:07:45 GMT +0000). [21:15:14] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.20, 3.87, 3.56 [21:15:56] I'm going to be suspending uploads on that wiki [21:16:39] PROBLEM - archive.stellurgists.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'archive.stellurgists.wiki' expires in 15 day(s) (Wed 10 May 2023 21:05:33 GMT +0000). [21:16:50] 100GB is insane, the swift object servers are already 70+ disk usage, with one at 90+ [21:16:52] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1527184546 secs [21:18:09] PROBLEM - wiki.climatechange.ai - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.climatechange.ai' expires in 15 day(s) (Wed 10 May 2023 21:06:36 GMT +0000). [21:18:53] PROBLEM - history.sdtef.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'history.sdtef.org' expires in 15 day(s) (Wed 10 May 2023 21:07:34 GMT +0000). [21:18:56] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1871f261d898...40c4ac0fc565 [21:18:58] [02miraheze/ssl] 07MirahezeSSLBot 0340c4ac0 - Bot: Update SSL cert for vedopedia.witches-empire.com [21:19:04] PROBLEM - fr.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fr.gyaanipedia.com' expires in 15 day(s) (Wed 10 May 2023 21:04:26 GMT +0000). [21:19:10] PROBLEM - index.pdcommunity.ir - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'index.pdcommunity.ir' expires in 15 day(s) (Wed 10 May 2023 21:03:16 GMT +0000). [21:19:26] PROBLEM - wiki.counterculturelabs.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.counterculturelabs.org' expires in 15 day(s) (Wed 10 May 2023 21:06:12 GMT +0000). [21:20:12] PROBLEM - aman.awiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'aman.awiki.org' expires in 15 day(s) (Wed 10 May 2023 21:06:00 GMT +0000). [21:20:13] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/40c4ac0fc565...ca5a69b17964 [21:20:15] [02miraheze/ssl] 07MirahezeSSLBot 03ca5a69b - Bot: Update SSL cert for history.sdtef.org [21:20:42] PROBLEM - de.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'de.gyaanipedia.com' expires in 15 day(s) (Wed 10 May 2023 21:09:24 GMT +0000). [21:20:45] PROBLEM - franchise.franchising.org.ua - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'franchise.franchising.org.ua' expires in 15 day(s) (Wed 10 May 2023 21:04:46 GMT +0000). [21:20:55] PROBLEM - mediawiki.ecfin.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mediawiki.ecfin.eu' expires in 15 day(s) (Wed 10 May 2023 21:09:17 GMT +0000). [21:20:57] uploads suspended on tttewikiwiki [21:20:59] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.31, 3.32, 3.39 [21:21:13] PROBLEM - wiki.triplescripts.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.triplescripts.org' expires in 15 day(s) (Wed 10 May 2023 21:09:35 GMT +0000). [21:21:22] or not, changing settings on ManageWiki is apparently useless there [21:22:03] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ca5a69b17964...4f6179127aa1 [21:22:05] Agent: Aren't you T&S? Can you try like a centralauth lock or something? [21:22:05] [02miraheze/ssl] 07MirahezeSSLBot 034f61791 - Bot: Update SSL cert for wiki.villagecollaborative.net [21:22:53] PROBLEM - hololive.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hololive.wiki' expires in 15 day(s) (Wed 10 May 2023 21:05:18 GMT +0000). [21:23:08] PROBLEM - wiki.seamly.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.seamly.net' expires in 15 day(s) (Wed 10 May 2023 21:05:42 GMT +0000). [21:23:10] PROBLEM - icclopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'icclopedia.org' expires in 15 day(s) (Wed 10 May 2023 21:04:17 GMT +0000). [21:23:19] PROBLEM - history.estill.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'history.estill.org' expires in 15 day(s) (Wed 10 May 2023 21:09:10 GMT +0000). [21:23:29] PROBLEM - tl.awiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tl.awiki.org' expires in 15 day(s) (Wed 10 May 2023 21:06:48 GMT +0000). [21:23:29] PROBLEM - wiki.shaazzz.ir - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.shaazzz.ir' expires in 15 day(s) (Wed 10 May 2023 21:02:56 GMT +0000). [21:23:43] PROBLEM - wiki.danvs.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.danvs.net' expires in 15 day(s) (Wed 10 May 2023 21:08:05 GMT +0000). [21:24:38] PROBLEM - guia.esporo.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'guia.esporo.net' expires in 15 day(s) (Wed 10 May 2023 21:05:06 GMT +0000). [21:25:59] PROBLEM - wiki.ssij.se - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ssij.se' expires in 15 day(s) (Wed 10 May 2023 21:05:53 GMT +0000). [21:26:00] PROBLEM - infectowiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'infectowiki.com' expires in 15 day(s) (Wed 10 May 2023 21:07:26 GMT +0000). [21:26:07] PROBLEM - www.baharna.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.baharna.org' expires in 15 day(s) (Wed 10 May 2023 21:01:34 GMT +0000). [21:27:40] PROBLEM - opendatascot.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'opendatascot.org' expires in 15 day(s) (Wed 10 May 2023 21:08:25 GMT +0000). [21:28:03] PROBLEM - wiki.warframestat.us - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.warframestat.us' expires in 15 day(s) (Wed 10 May 2023 21:02:44 GMT +0000). [21:28:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4f6179127aa1...bfb9be35501e [21:28:25] [02miraheze/ssl] 07MirahezeSSLBot 03bfb9be3 - Bot: Update SSL cert for archive.stellurgists.wiki [21:29:11] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [21:31:07] PROBLEM - wiki.insideearth.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.insideearth.info' expires in 15 day(s) (Wed 10 May 2023 21:08:16 GMT +0000). [21:32:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bfb9be35501e...86bd380acbb8 [21:32:13] PROBLEM - www.bluepageswiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.bluepageswiki.org' expires in 15 day(s) (Wed 10 May 2023 21:02:08 GMT +0000). [21:32:13] [02miraheze/ssl] 07MirahezeSSLBot 0386bd380 - Bot: Update SSL cert for spcodex.wiki [21:32:15] PROBLEM - wiki.mobilityengineer.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mobilityengineer.com' expires in 15 day(s) (Wed 10 May 2023 21:02:19 GMT +0000). [21:32:25] PROBLEM - baharna.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.baharna.org' expires in 15 day(s) (Wed 10 May 2023 21:01:34 GMT +0000). [21:32:39] PROBLEM - wiki.cdntennis.ca - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cdntennis.ca' expires in 15 day(s) (Wed 10 May 2023 21:01:45 GMT +0000). [21:33:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.16, 11.15, 9.58 [21:34:12] PROBLEM - wiki.mineland.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mineland.eu' expires in 15 day(s) (Wed 10 May 2023 21:09:46 GMT +0000). [21:34:54] PROBLEM - beaconspace.unrestrictedlorefare.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'beaconspace.unrestrictedlorefare.com' expires in 15 day(s) (Wed 10 May 2023 21:07:53 GMT +0000). [21:35:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.35, 10.78, 9.63 [21:35:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.07, 11.72, 9.82 [21:35:43] PROBLEM - wiki.lefrenchmelee.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.lefrenchmelee.fr' expires in 15 day(s) (Wed 10 May 2023 21:07:03 GMT +0000). [21:36:17] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=esnarutowiki (END - exit=65280) [21:36:22] PROBLEM - wiki.minecraftathome.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.minecraftathome.com' expires in 15 day(s) (Wed 10 May 2023 21:09:01 GMT +0000). [21:36:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:36:27] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=esnarutowiki (START) [21:36:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:37:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.14, 12.62, 10.46 [21:38:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/86bd380acbb8...40723afabca9 [21:38:59] [02miraheze/ssl] 07MirahezeSSLBot 0340723af - Bot: Update SSL cert for index.pdcommunity.ir [21:39:13] PROBLEM - wiki.anglish.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.anglish.info' expires in 15 day(s) (Wed 10 May 2023 21:08:49 GMT +0000). [21:41:22] PROBLEM - puppet141 Current Load on puppet141 is WARNING: WARNING - load average: 3.58, 3.53, 2.36 [21:43:22] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 12.95, 6.59, 3.58 [21:43:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.75, 11.99, 10.87 [21:43:40] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.93, 3.67, 3.34 [21:44:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/40723afabca9...d8e8edaeacbb [21:44:46] [02miraheze/ssl] 07MirahezeSSLBot 03d8e8eda - Bot: Update SSL cert for infectowiki.com [21:45:21] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.34, 3.89, 3.05 [21:45:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d8e8edaeacbb...51d3da3b9e2b [21:45:24] [02miraheze/ssl] 07MirahezeSSLBot 0351d3da3 - Bot: Update SSL cert for wiki.minkyu.kim [21:45:35] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.93, 3.94, 3.47 [21:45:42] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [21:47:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.91, 3.56, 3.04 [21:47:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.25, 11.58, 11.18 [21:48:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/51d3da3b9e2b...c0a16e5dff56 [21:48:08] [02miraheze/ssl] 07MirahezeSSLBot 03c0a16e5 - Bot: Update SSL cert for history.estill.org [21:48:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.15, 6.95, 7.92 [21:48:42] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09261333942 secs [21:49:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.88, 3.16, 2.94 [21:49:22] PROBLEM - puppet141 Current Load on puppet141 is WARNING: WARNING - load average: 1.68, 3.85, 3.46 [21:49:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.83, 3.55, 3.43 [21:50:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.81, 7.83, 8.10 [21:51:20] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.96, 3.35, 3.37 [21:51:22] RECOVERY - puppet141 Current Load on puppet141 is OK: OK - load average: 2.05, 3.22, 3.27 [21:53:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.79, 9.33, 10.06 [21:54:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.21, 7.39, 7.90 [21:54:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c0a16e5dff56...d413bc83c65f [21:54:17] [02miraheze/ssl] 07MirahezeSSLBot 03d413bc8 - Bot: Update SSL cert for www.bluepageswiki.org [21:55:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.96, 5.41, 3.50 [21:55:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.14, 10.22, 10.51 [21:56:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.05, 7.80, 8.00 [21:57:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.48, 10.56, 10.59 [21:57:55] !log [alex@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=esnarutowiki (END - exit=0) [21:58:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:58:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.22, 7.78, 7.98 [21:59:12] that import is still ongoing however [21:59:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.86, 11.56, 10.94 [21:59:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d413bc83c65f...03ce01253a77 [21:59:37] [02miraheze/ssl] 07MirahezeSSLBot 0303ce012 - Bot: Update SSL cert for wiki.counterculturelabs.org [22:00:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.56, 7.82, 7.96 [22:01:21] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.46, 3.90, 3.22 [22:02:53] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.45, 3.83, 3.67 [22:03:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.09, 3.27, 3.07 [22:03:24] RECOVERY - www.mcpk.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.mcpk.wiki' will expire on Sun 23 Jul 2023 20:11:59 GMT +0000. [22:03:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.97, 11.71, 11.19 [22:03:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/03ce01253a77...ba7616c2b7b7 [22:03:44] [02miraheze/ssl] 07MirahezeSSLBot 03ba7616c - Bot: Update SSL cert for icclopedia.org [22:05:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.20, 11.37, 10.48 [22:05:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ba7616c2b7b7...c15343aac1d3 [22:05:45] [02miraheze/ssl] 07MirahezeSSLBot 03c15343a - Bot: Update SSL cert for mediawiki.ecfin.eu [22:06:49] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.92, 2.71, 3.24 [22:07:09] RECOVERY - wiki.villagecollaborative.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.villagecollaborative.net' will expire on Sun 23 Jul 2023 20:21:55 GMT +0000. [22:07:22] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.11, 3.89, 3.39 [22:07:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.55, 11.01, 10.46 [22:07:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c15343aac1d3...28963fba2d5b [22:07:48] [02miraheze/ssl] 07MirahezeSSLBot 0328963fb - Bot: Update SSL cert for wiki.ricochetuniverse.com [22:07:50] RECOVERY - spcodex.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spcodex.wiki' will expire on Sun 23 Jul 2023 20:32:03 GMT +0000. [22:09:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.22, 3.81, 3.43 [22:10:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/28963fba2d5b...260fca213c16 [22:10:57] [02miraheze/ssl] 07MirahezeSSLBot 03260fca2 - Bot: Update SSL cert for wiki.danvs.net [22:11:09] RECOVERY - vedopedia.witches-empire.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vedopedia.witches-empire.com' will expire on Sun 23 Jul 2023 20:18:48 GMT +0000. [22:11:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.04, 3.21, 3.26 [22:11:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.96, 9.77, 10.11 [22:12:14] PROBLEM - puppet141 Current Load on puppet141 is WARNING: WARNING - load average: 3.46, 3.56, 2.88 [22:13:57] RECOVERY - archive.stellurgists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'archive.stellurgists.wiki' will expire on Sun 23 Jul 2023 20:28:14 GMT +0000. [22:14:11] RECOVERY - puppet141 Current Load on puppet141 is OK: OK - load average: 1.87, 2.92, 2.73 [22:14:36] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.98, 3.39, 3.22 [22:14:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/260fca213c16...a2541ab48268 [22:14:51] [02miraheze/ssl] 07MirahezeSSLBot 03a2541ab - Bot: Update SSL cert for wiki.mobilityengineer.com [22:16:34] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.56, 3.20, 3.15 [22:17:04] RECOVERY - index.pdcommunity.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'index.pdcommunity.ir' will expire on Sun 23 Jul 2023 20:38:50 GMT +0000. [22:17:06] RECOVERY - history.sdtef.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.sdtef.org' will expire on Sun 23 Jul 2023 20:20:06 GMT +0000. [22:17:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.64, 10.34, 10.26 [22:18:34] RECOVERY - mediawiki.ecfin.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'mediawiki.ecfin.eu' will expire on Sun 23 Jul 2023 21:05:36 GMT +0000. [22:18:34] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.26, 2.80, 3.01 [22:20:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.94, 7.19, 7.82 [22:20:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a2541ab48268...af93e58aef56 [22:20:44] [02miraheze/ssl] 07MirahezeSSLBot 03af93e58 - Bot: Update SSL cert for opendatascot.org [22:21:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.09, 9.60, 10.02 [22:22:38] RECOVERY - history.estill.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.estill.org' will expire on Sun 23 Jul 2023 20:47:58 GMT +0000. [22:23:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.82, 8.80, 9.94 [22:24:27] RECOVERY - infectowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'infectowiki.com' will expire on Sun 23 Jul 2023 20:44:33 GMT +0000. [22:24:34] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.88, 4.13, 3.55 [22:26:34] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.28, 3.88, 3.52 [22:27:47] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/af93e58aef56...8cd7b82daf90 [22:27:49] [02miraheze/ssl] 07MirahezeSSLBot 038cd7b82 - Bot: Update SSL cert for es.gyaanipedia.com [22:28:34] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.99, 4.43, 3.78 [22:30:11] RECOVERY - www.bluepageswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.bluepageswiki.org' will expire on Sun 23 Jul 2023 20:54:07 GMT +0000. [22:30:34] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.22, 3.67, 3.58 [22:30:38] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8cd7b82daf90...c2ee293c6531 [22:30:40] [02miraheze/ssl] 07MirahezeSSLBot 03c2ee293 - Bot: Update SSL cert for de.gyaanipedia.com [22:32:13] RECOVERY - wiki.mobilityengineer.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mobilityengineer.com' will expire on Sun 23 Jul 2023 21:14:40 GMT +0000. [22:32:24] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c2ee293c6531...aca9441daf4b [22:32:27] [02miraheze/ssl] 07MirahezeSSLBot 03aca9441 - Bot: Update SSL cert for wiki.mineland.eu [22:32:34] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.71, 3.09, 3.39 [22:33:15] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/aca9441daf4b...8c81437a3e66 [22:33:18] [02miraheze/ssl] 07MirahezeSSLBot 038c81437 - Bot: Update SSL cert for franchise.franchising.org.ua [22:35:31] RECOVERY - wiki.minkyu.kim - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minkyu.kim' will expire on Sun 23 Jul 2023 20:45:11 GMT +0000. [22:36:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.22, 7.86, 7.80 [22:37:21] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.34, 3.43, 2.90 [22:38:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.03, 7.32, 7.59 [22:38:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8c81437a3e66...2d26bf6606e6 [22:38:54] [02miraheze/ssl] 07MirahezeSSLBot 032d26bf6 - Bot: Update SSL cert for www.iceria.org [22:39:02] RECOVERY - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ricochetuniverse.com' will expire on Sun 23 Jul 2023 21:07:37 GMT +0000. [22:39:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.25, 3.02, 2.82 [22:44:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.48, 8.13, 7.74 [22:45:39] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2d26bf6606e6...d8c9a10c531f [22:45:40] [02miraheze/ssl] 07MirahezeSSLBot 03d8c9a10 - Bot: Update SSL cert for fr.gyaanipedia.com [22:47:26] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.22, 3.55, 2.98 [22:48:09] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d8c9a10c531f...cee24621c740 [22:48:11] [02miraheze/ssl] 07MirahezeSSLBot 03cee2462 - Bot: Update SSL cert for wiki.ssij.se [22:48:53] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cee24621c740...6e837d36c1ed [22:48:56] [02miraheze/ssl] 07MirahezeSSLBot 036e837d3 - Bot: Update SSL cert for wiki.minecraftathome.com [22:49:08] RECOVERY - wiki.counterculturelabs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.counterculturelabs.org' will expire on Sun 23 Jul 2023 20:59:29 GMT +0000. [22:49:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.32, 3.00, 2.85 [22:50:12] RECOVERY - de.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'de.gyaanipedia.com' will expire on Sun 23 Jul 2023 21:30:30 GMT +0000. [22:51:11] RECOVERY - icclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'icclopedia.org' will expire on Sun 23 Jul 2023 21:03:36 GMT +0000. [22:52:46] RECOVERY - wiki.danvs.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.danvs.net' will expire on Sun 23 Jul 2023 21:10:46 GMT +0000. [22:55:40] RECOVERY - opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Sun 23 Jul 2023 21:20:32 GMT +0000. [22:56:05] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.48, 4.93, 3.71 [22:56:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6e837d36c1ed...674188e7fbb0 [22:56:26] [02miraheze/ssl] 07MirahezeSSLBot 03674188e - Bot: Update SSL cert for wiki.insideearth.info [23:02:40] RECOVERY - wiki.mineland.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mineland.eu' will expire on Sun 23 Jul 2023 21:32:16 GMT +0000. [23:02:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/674188e7fbb0...56d1c4047a55 [23:02:56] [02miraheze/ssl] 07MirahezeSSLBot 0356d1c40 - Bot: Update SSL cert for pubwiki.lab.co.pl [23:03:55] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.78, 3.33, 3.54 [23:04:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.18, 7.28, 7.87 [23:04:32] RECOVERY - wiki.minecraftathome.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minecraftathome.com' will expire on Sun 23 Jul 2023 21:48:47 GMT +0000. [23:05:53] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.64, 3.01, 3.38 [23:06:37] RECOVERY - es.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'es.gyaanipedia.com' will expire on Sun 23 Jul 2023 21:27:39 GMT +0000. [23:08:05] RECOVERY - www.iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Sun 23 Jul 2023 21:38:43 GMT +0000. [23:08:17] RECOVERY - iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Sun 23 Jul 2023 21:38:43 GMT +0000. [23:08:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/56d1c4047a55...783b14d3cb56 [23:08:30] [02miraheze/ssl] 07MirahezeSSLBot 03783b14d - Bot: Update SSL cert for beaconspace.unrestrictedlorefare.com [23:09:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.97, 3.57, 3.00 [23:11:18] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/783b14d3cb56...d4d157afa83a [23:11:19] [02miraheze/ssl] 07MirahezeSSLBot 03d4d157a - Bot: Update SSL cert for wiki.lefrenchmelee.fr [23:11:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.46, 2.72, 2.75 [23:11:29] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1156223416 secs [23:13:41] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.35, 3.75, 3.42 [23:15:38] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.29, 3.17, 3.25 [23:16:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.69, 7.87, 7.67 [23:18:36] RECOVERY - fr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fr.gyaanipedia.com' will expire on Sun 23 Jul 2023 21:45:30 GMT +0000. [23:19:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d4d157afa83a...ba5789db74c9 [23:19:24] [02miraheze/ssl] 07MirahezeSSLBot 03ba5789d - Bot: Update SSL cert for tl.awiki.org [23:19:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.51, 11.10, 9.54 [23:20:37] RECOVERY - franchise.franchising.org.ua - LetsEncrypt on sslhost is OK: OK - Certificate 'franchise.franchising.org.ua' will expire on Sun 23 Jul 2023 21:33:07 GMT +0000. [23:21:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.08, 10.56, 9.53 [23:22:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.16, 7.92, 7.81 [23:23:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.80, 9.69, 9.35 [23:24:57] RECOVERY - wiki.ssij.se - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ssij.se' will expire on Sun 23 Jul 2023 21:48:02 GMT +0000. [23:25:24] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.87, 4.75, 3.93 [23:26:56] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ba5789db74c9...453d3520b187 [23:26:59] [02miraheze/ssl] 07MirahezeSSLBot 03453d352 - Bot: Update SSL cert for fc.songcontests.eu [23:27:21] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.58, 4.00, 3.75 [23:28:38] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.70, 5.89, 3.34 [23:29:11] RECOVERY - wiki.insideearth.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.insideearth.info' will expire on Sun 23 Jul 2023 21:56:17 GMT +0000. [23:29:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.68, 4.15, 3.85 [23:30:38] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.99, 4.92, 3.32 [23:33:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.15, 3.85, 3.79 [23:33:38] RECOVERY - beaconspace.unrestrictedlorefare.com - LetsEncrypt on sslhost is OK: OK - Certificate 'beaconspace.unrestrictedlorefare.com' will expire on Sun 23 Jul 2023 22:08:15 GMT +0000. [23:34:02] RECOVERY - wiki.lefrenchmelee.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lefrenchmelee.fr' will expire on Sun 23 Jul 2023 22:11:11 GMT +0000. [23:34:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.61, 8.33, 7.73 [23:34:58] RECOVERY - pubwiki.lab.co.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'pubwiki.lab.co.pl' will expire on Sun 23 Jul 2023 22:02:46 GMT +0000. [23:34:59] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/453d3520b187...e93c9b69d402 [23:35:01] [02miraheze/ssl] 07MirahezeSSLBot 03e93c9b6 - Bot: Update SSL cert for guia.esporo.net [23:36:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.84, 7.15, 7.39 [23:37:26] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07801055908 secs [23:37:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e93c9b69d402...7d86890a4b84 [23:37:37] [02miraheze/ssl] 07MirahezeSSLBot 037d86890 - Bot: Update SSL cert for www.baharna.org [23:39:00] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7d86890a4b84...65ea17b3dedc [23:39:02] [02miraheze/ssl] 07MirahezeSSLBot 0365ea17b - Bot: Update SSL cert for wiki.triplescripts.org [23:39:07] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.37, 2.58, 3.29 [23:43:32] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/65ea17b3dedc...5c0d9b4b10e9 [23:43:35] [02miraheze/ssl] 07MirahezeSSLBot 035c0d9b4 - Bot: Update SSL cert for wiki.anglish.info [23:46:13] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.24, 6.06, 6.75 [23:50:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.15, 6.90, 6.93 [23:50:18] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5c0d9b4b10e9...981be661ebd1 [23:50:19] [02miraheze/ssl] 07MirahezeSSLBot 03981be66 - Bot: Update SSL cert for wiki.warframestat.us [23:52:13] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.86, 6.31, 6.72 [23:52:35] RECOVERY - tl.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'tl.awiki.org' will expire on Sun 23 Jul 2023 22:19:14 GMT +0000. [23:56:33] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/981be661ebd1...7b97099d0b5a [23:56:34] [02miraheze/ssl] 07MirahezeSSLBot 037b97099 - Bot: Update SSL cert for wiki.seamly.net [23:58:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7b97099d0b5a...d6226393c5a8 [23:58:29] [02miraheze/ssl] 07MirahezeSSLBot 03d622639 - Bot: Update SSL cert for wiki.rebirthofthenight.com