[00:00:11] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.85, 9.52, 8.24 [00:02:10] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 13.24, 10.87, 8.89 [00:04:10] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 9.78, 10.54, 9.02 [00:07:10] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.88, 3.17, 1.94 [00:08:34] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.54, 3.65, 2.91 [00:09:09] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 3.00, 3.03, 2.04 [00:10:10] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 7.15, 9.66, 9.22 [00:10:30] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.80, 3.53, 2.94 [00:12:26] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.70, 3.26, 2.92 [00:30:23] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.28, 9.49, 8.84 [00:32:14] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 7.05, 8.57, 8.58 [00:33:07] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.56, 2.91, 2.39 [00:35:07] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.51, 2.78, 2.41 [00:37:52] !log [macfan@test131] starting deploy of {'branch': 'REL1_40', 'world': True, 'l10n': True, 'extensionlist': True, 'force': True, 'ignoretime': True} to all [00:38:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:40:11] !log [macfan@test131] starting deploy of {'branch': 'REL1_40', 'world': True, 'extensionlist': True, 'force': True, 'ignoretime': True} to all [00:40:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:43:15] !log [macfan@test131] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [00:43:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:49:04] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.67, 3.32, 2.85 [00:52:10] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.93, 10.66, 9.00 [00:52:59] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.37, 3.25, 2.92 [00:56:13] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 7.79, 9.40, 8.89 [00:58:47] PROBLEM - ffwiki.org - reverse DNS on sslhost is UNKNOWN: [01:01:58] PROBLEM - wiki.litek.top - reverse DNS on sslhost is UNKNOWN: [01:03:06] PROBLEM - vise.dayid.org - reverse DNS on sslhost is UNKNOWN: PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is UNKNOWN: PROBLEM - o.kaoyu666.cn - reverse DNS on sslhost is UNKNOWN: [01:03:10] PROBLEM - wiki.newnormal.fans - reverse DNS on sslhost is UNKNOWN: [01:03:29] PROBLEM - alternatewiki.tombricks.com - reverse DNS on sslhost is UNKNOWN: PROBLEM - wiki.villagecollaborative.net - reverse DNS on sslhost is UNKNOWN: PROBLEM - podpedia.org - reverse DNS on sslhost is UNKNOWN: [01:03:30] PROBLEM - aman.awiki.org - reverse DNS on sslhost is UNKNOWN: PROBLEM - wiki.virtualjet.net - reverse DNS on sslhost is UNKNOWN: PROBLEM - grk.repository.archiopedia.org - reverse DNS on sslhost is UNKNOWN: [01:03:30] PROBLEM - tech.exputra.com - reverse DNS on sslhost is UNKNOWN: [01:03:30] PROBLEM - wiki.elgeis.com - reverse DNS on sslhost is UNKNOWN: [01:04:02] PROBLEM - burnout.wiki - reverse DNS on sslhost is UNKNOWN: PROBLEM - wiki.puucraft.net - reverse DNS on sslhost is UNKNOWN: [01:05:12] PROBLEM - wmworld.sktz.live - reverse DNS on sslhost is UNKNOWN: [01:05:17] PROBLEM - elephants.ggreig.com - reverse DNS on sslhost is UNKNOWN: [01:05:24] PROBLEM - inourownwords.online - reverse DNS on sslhost is UNKNOWN: [01:05:24] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is UNKNOWN: PROBLEM - glitchcity.wiki - reverse DNS on sslhost is UNKNOWN: PROBLEM - wiki.mc.marochuru.net - reverse DNS on sslhost is UNKNOWN: [01:05:24] PROBLEM - wiki.artmechanicum.com - reverse DNS on sslhost is UNKNOWN: [01:05:24] PROBLEM - mon141 APT on mon141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:05:24] PROBLEM - tintinpedia.miraheze.org - reverse DNS on sslhost is UNKNOWN: [01:05:24] PROBLEM - wiki.milkywayidle.com - reverse DNS on sslhost is UNKNOWN: [01:05:25] PROBLEM - bluearchive.wiki - reverse DNS on sslhost is UNKNOWN: [01:05:26] PROBLEM - www.tintinpedia.fr - reverse DNS on sslhost is UNKNOWN: [01:05:47] PROBLEM - persist.wiki - reverse DNS on sslhost is UNKNOWN: [01:05:47] PROBLEM - wiki.sukotuze.com - reverse DNS on sslhost is UNKNOWN: [01:05:47] PROBLEM - persist.wiki - reverse DNS on sslhost is UNKNOWN: [01:05:51] PROBLEM - pt.graalmilitary.com - reverse DNS on sslhost is UNKNOWN: [01:06:22] PROBLEM - crustypedia.org - reverse DNS on sslhost is UNKNOWN: PROBLEM - wiki.digitaldesignhq.com - reverse DNS on sslhost is UNKNOWN: [01:06:22] PROBLEM - wiki.cyberfurs.org - reverse DNS on sslhost is UNKNOWN: [01:06:22] PROBLEM - miraheze.gq - reverse DNS on sslhost is UNKNOWN: [01:06:43] PROBLEM - eurocom.wiki - reverse DNS on sslhost is UNKNOWN: PROBLEM - schizoidnightmares.miraheze.org - reverse DNS on sslhost is UNKNOWN: [01:06:44] PROBLEM - wiki.yumeka.icu - reverse DNS on sslhost is UNKNOWN: [01:06:45] PROBLEM - partopedia.ru - reverse DNS on sslhost is UNKNOWN: SSL OK - partopedia.ru reverse DNS resolves to cp25.miraheze.org - NS RECORDS OK [01:06:45] PROBLEM - wiki.triplescripts.org - reverse DNS on sslhost is UNKNOWN: SSL OK - wiki.triplescripts.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:08:56] RECOVERY - mon141 APT on mon141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [01:09:28] !log [@mwtask141] starting deploy of {'config': True} to all [01:09:32] PROBLEM - wiki.fediverse.kr - reverse DNS on sslhost is UNKNOWN: SSL OK - wiki.fediverse.kr reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:09:49] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 26s [01:09:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:10:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:12:04] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.05, 10.06, 9.46 [01:13:57] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.19, 10.34, 9.63 [01:14:56] PROBLEM - mw131 Puppet on mw131 is UNKNOWN: NRPE: Unable to read output [01:15:24] PROBLEM - mwtask141 Puppet on mwtask141 is UNKNOWN: NRPE: Unable to read output [01:15:53] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.66, 11.05, 9.97 [01:16:55] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.46, 10.27, 9.44 [01:17:48] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.14, 10.82, 10.02 [01:18:53] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.02, 9.54, 9.28 [01:19:43] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 8.88, 10.12, 9.86 [01:22:29] !log [macfan@test131] starting deploy of {'branch': 'REL1_40', 'world': True, 'extensionlist': True, 'force': True, 'ignoretime': True} to all [01:22:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:22:47] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 9.15, 10.56, 9.87 [01:23:17] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:23:35] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.37, 10.86, 10.24 [01:24:45] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 13.39, 12.05, 10.52 [01:25:16] PROBLEM - test131 MediaWiki Rendering on test131 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [01:25:33] !log [macfan@test131] finished deploy of {'branch': 'REL1_40', 'world': True, 'extensionlist': True, 'force': True, 'ignoretime': True} to all - SUCCESS in 183s [01:25:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:43] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.05, 11.29, 10.40 [01:26:50] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:27:24] !log [@test131] starting deploy of {'config': True} to all [01:27:25] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [01:27:31] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.12, 11.43, 10.64 [01:27:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:27:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:27:52] PROBLEM - ffwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ffwiki.org could not be found [01:29:31] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 8.60, 10.72, 10.50 [01:31:49] PROBLEM - wiki.litek.top - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.litek.top could not be found [01:32:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [01:32:30] RECOVERY - vise.dayid.org - reverse DNS on sslhost is OK: SSL OK - vise.dayid.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:32:43] RECOVERY - o.kaoyu666.cn - reverse DNS on sslhost is OK: SSL OK - o.kaoyu666.cn reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:32:45] RECOVERY - wiki.newnormal.fans - reverse DNS on sslhost is OK: SSL OK - wiki.newnormal.fans reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:33:08] RECOVERY - wiki.villagecollaborative.net - reverse DNS on sslhost is OK: SSL OK - wiki.villagecollaborative.net reverse DNS resolves to cp24.miraheze.org - CNAME OK [01:33:10] RECOVERY - wiki.virtualjet.net - reverse DNS on sslhost is OK: SSL OK - wiki.virtualjet.net reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:33:10] RECOVERY - aman.awiki.org - reverse DNS on sslhost is OK: SSL OK - aman.awiki.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:33:12] RECOVERY - alternatewiki.tombricks.com - reverse DNS on sslhost is OK: SSL OK - alternatewiki.tombricks.com reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:33:13] RECOVERY - podpedia.org - reverse DNS on sslhost is OK: SSL OK - podpedia.org reverse DNS resolves to cp24.miraheze.org - NS RECORDS OK [01:33:14] RECOVERY - wiki.elgeis.com - reverse DNS on sslhost is OK: SSL OK - wiki.elgeis.com reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:33:15] RECOVERY - grk.repository.archiopedia.org - reverse DNS on sslhost is OK: SSL OK - grk.repository.archiopedia.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [01:33:29] RECOVERY - tech.exputra.com - reverse DNS on sslhost is OK: SSL OK - tech.exputra.com reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:33:31] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 7.58, 9.17, 9.91 [01:33:39] RECOVERY - burnout.wiki - reverse DNS on sslhost is OK: SSL OK - burnout.wiki reverse DNS resolves to cp24.miraheze.org - NS RECORDS OK [01:33:39] RECOVERY - wiki.puucraft.net - reverse DNS on sslhost is OK: SSL OK - wiki.puucraft.net reverse DNS resolves to cp24.miraheze.org - CNAME OK [01:34:10] RECOVERY - wmworld.sktz.live - reverse DNS on sslhost is OK: SSL OK - wmworld.sktz.live reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:34:19] RECOVERY - wiki.mc.marochuru.net - reverse DNS on sslhost is OK: SSL OK - wiki.mc.marochuru.net reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:34:22] RECOVERY - inourownwords.online - reverse DNS on sslhost is OK: SSL OK - inourownwords.online reverse DNS resolves to cp24.miraheze.org - NS RECORDS OK [01:34:23] RECOVERY - elephants.ggreig.com - reverse DNS on sslhost is OK: SSL OK - elephants.ggreig.com reverse DNS resolves to cp24.miraheze.org - CNAME OK [01:34:26] RECOVERY - tintinpedia.miraheze.org - reverse DNS on sslhost is OK: SSL OK - tintinpedia.miraheze.org reverse DNS resolves to cp24.miraheze.org - NS RECORDS OK [01:34:27] RECOVERY - wiki.milkywayidle.com - reverse DNS on sslhost is OK: SSL OK - wiki.milkywayidle.com reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:34:28] RECOVERY - glitchcity.wiki - reverse DNS on sslhost is OK: SSL OK - glitchcity.wiki reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:34:29] RECOVERY - wiki.mineland.eu - reverse DNS on sslhost is OK: SSL OK - wiki.mineland.eu reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:34:31] RECOVERY - wiki.artmechanicum.com - reverse DNS on sslhost is OK: SSL OK - wiki.artmechanicum.com reverse DNS resolves to cp24.miraheze.org - CNAME OK [01:34:36] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.57, 10.06, 10.20 [01:35:02] RECOVERY - wiki.sukotuze.com - reverse DNS on sslhost is OK: SSL OK - wiki.sukotuze.com reverse DNS resolves to cp24.miraheze.org - CNAME OK [01:35:02] RECOVERY - persist.wiki - reverse DNS on sslhost is OK: SSL OK - persist.wiki reverse DNS resolves to cp24.miraheze.org - NS RECORDS OK [01:35:21] RECOVERY - www.tintinpedia.fr - reverse DNS on sslhost is OK: SSL OK - www.tintinpedia.fr reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [01:35:22] RECOVERY - bluearchive.wiki - reverse DNS on sslhost is OK: SSL OK - bluearchive.wiki reverse DNS resolves to cp24.miraheze.org - CNAME OK [01:35:43] RECOVERY - crustypedia.org - reverse DNS on sslhost is OK: SSL OK - crustypedia.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:35:43] RECOVERY - wiki.cyberfurs.org - reverse DNS on sslhost is OK: SSL OK - wiki.cyberfurs.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:35:49] RECOVERY - miraheze.gq - reverse DNS on sslhost is OK: SSL OK - miraheze.gq reverse DNS resolves to cp24.miraheze.org - NS RECORDS OK [01:35:50] RECOVERY - pt.graalmilitary.com - reverse DNS on sslhost is OK: SSL OK - pt.graalmilitary.com reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:36:17] RECOVERY - wiki.digitaldesignhq.com - reverse DNS on sslhost is OK: SSL OK - wiki.digitaldesignhq.com reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:36:24] RECOVERY - schizoidnightmares.miraheze.org - reverse DNS on sslhost is OK: SSL OK - schizoidnightmares.miraheze.org reverse DNS resolves to cp25.miraheze.org - NS RECORDS OK [01:36:25] RECOVERY - eurocom.wiki - reverse DNS on sslhost is OK: SSL OK - eurocom.wiki reverse DNS resolves to cp24.miraheze.org - NS RECORDS OK [01:36:28] RECOVERY - wiki.yumeka.icu - reverse DNS on sslhost is OK: SSL OK - wiki.yumeka.icu reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:36:28] RECOVERY - partopedia.ru - reverse DNS on sslhost is OK: SSL OK - partopedia.ru reverse DNS resolves to cp25.miraheze.org - NS RECORDS OK [01:36:28] RECOVERY - wiki.triplescripts.org - reverse DNS on sslhost is OK: SSL OK - wiki.triplescripts.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [01:38:30] RECOVERY - wiki.fediverse.kr - reverse DNS on sslhost is OK: SSL OK - wiki.fediverse.kr reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:40:32] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [01:40:55] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:54:20] [02miraheze/MirahezeMagic] 07MacFan4000 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/MirahezeMagic/compare/3aceccbb8bf4...28560c0aa593 [01:54:22] [02miraheze/MirahezeMagic] 07MacFan4000 0328560c0 - Add mergeMessageFileList.php to MM [02:00:19] [02miraheze/mediawiki] 07MacFan4000 pushed 031 commit to 03REL1_41 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/5335f40f941a...b96fdf0a6f82 [02:00:19] [02miraheze/mediawiki] 07MacFan4000 03b96fdf0 - Update MM [02:05:16] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.63, 3.13, 1.88 [02:05:24] miraheze/MirahezeMagic - MacFan4000 the build passed. [02:07:13] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.10, 2.83, 1.93 [02:26:55] [02miraheze/mediawiki] 07MacFan4000 pushed 031 commit to 03REL1_41 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/b96fdf0a6f82...fd7e87c2fdad [02:26:57] [02miraheze/mediawiki] 07MacFan4000 03fd7e87c - Downgrade EmbedVudeo [02:41:07] [02miraheze/mediawiki] 07MacFan4000 pushed 032 commits to 03REL1_41 [+0/-0/±3] 13https://github.com/miraheze/mediawiki/compare/fd7e87c2fdad...492ae0a66f59 [02:41:08] [02miraheze/mediawiki] 07MacFan4000 032eaaf76 - Downgrade EmbedVideo [02:41:10] [02miraheze/mediawiki] 07MacFan4000 03492ae0a - Downgrade Mermaid and SemanticFormsSelect [02:44:49] PROBLEM - test131 Puppet on test131 is WARNING: WARNING: Puppet is currently disabled, message: MacFan4000, last run 17 minutes ago with 0 failures [02:51:31] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.95, 8.97, 7.48 [02:53:31] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 8.61, 9.01, 7.69 [03:05:01] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 12.59, 10.32, 6.59 [03:07:00] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 7.46, 9.35, 6.69 [03:08:55] !log [macfan@test131] starting deploy of {'branch': 'REL1_41', 'world': True, 'l10n': True, 'extensionlist': True, 'force': True, 'ignoretime': True} to all [03:09:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:14:52] !log [macfan@test131] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [03:14:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:15:31] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.46, 11.16, 9.22 [03:16:10] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.54, 11.23, 9.21 [03:16:45] PROBLEM - theagonian.com - LetsEncrypt on sslhost is CRITICAL: connect to address theagonian.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [03:19:32] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 8.76, 9.87, 9.15 [03:19:43] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.44, 4.10, 3.30 [03:20:10] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 9.25, 10.02, 9.19 [03:20:21] !log [macfan@test131] starting deploy of {'branch': 'REL1_41', 'world': True, 'l10n': True, 'extensionlist': True, 'force': True, 'ignoretime': True} to all [03:20:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:21:39] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.18, 3.68, 3.24 [03:23:38] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.21, 3.25, 3.14 [03:25:32] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.90, 10.57, 9.59 [03:29:31] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 7.13, 10.11, 9.77 [03:29:41] !log [macfan@test131] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [03:29:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:30:10] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.58, 9.81, 9.22 [03:31:17] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 1.83, 2.61, 3.91 [03:32:10] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.07, 9.23, 9.08 [03:33:09] !log [macfan@test131] starting deploy of {'branch': 'REL1_41', 'world': True, 'l10n': True, 'extensionlist': True, 'force': True, 'ignoretime': True} to all [03:33:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:35:17] RECOVERY - mon141 Current Load on mon141 is OK: LOAD OK - total load average: 2.19, 2.07, 3.37 [03:35:57] !log [macfan@test131] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [03:36:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:36:59] !log [macfan@test131] starting deploy of {'branch': 'REL1_41', 'world': True, 'l10n': True, 'extensionlist': True, 'force': True, 'ignoretime': True} to all [03:37:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:38:16] [02miraheze/MirahezeMagic] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/28560c0aa593...ebee49702a37 [03:38:18] [02miraheze/MirahezeMagic] 07MacFan4000 03ebee497 - fix path [03:38:33] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.56, 3.75, 3.30 [03:40:29] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.42, 3.26, 3.17 [03:41:54] [02miraheze/mediawiki] 07MacFan4000 pushed 031 commit to 03REL1_41 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/492ae0a66f59...6ca359d35f11 [03:41:56] [02miraheze/mediawiki] 07MacFan4000 036ca359d - fix [03:44:06] [02miraheze/mediawiki] 07MacFan4000 pushed 031 commit to 03REL1_41 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/6ca359d35f11...ce6708ab0116 [03:44:07] [02miraheze/mediawiki] 07MacFan4000 03ce6708a - Update MM [03:46:38] PROBLEM - theagonian.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'theagonian.com' expires in 14 day(s) (Mon 06 Nov 2023 09:22:19 GMT +0000). [03:49:50] miraheze/MirahezeMagic - MacFan4000 the build passed. [03:59:50] !log [macfan@test131] finished deploy of {'branch': 'REL1_41', 'world': True, 'l10n': True, 'extensionlist': True, 'force': True, 'ignoretime': True} to all - SUCCESS in 1371s [03:59:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:06:22] !log [macfan@test131] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/beta.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/AbuseFilter/db_patches/mysql/patch-add-af_actor.sql (START) [04:06:23] !log [macfan@test131] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/beta.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/AbuseFilter/db_patches/mysql/patch-add-af_actor.sql (END - exit=65280) [04:06:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:06:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:06:35] !log [macfan@test131] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/beta.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/AbuseFilter/db_patches/mysql/patch-add-afh_actor.sql (START) [04:06:36] !log [macfan@test131] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/beta.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/AbuseFilter/db_patches/mysql/patch-add-afh_actor.sql (END - exit=65280) [04:06:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:06:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:06:58] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/add63d4d7493 [04:06:59] [02miraheze/puppet] 07The-Voidwalker 03add63d4 - thumbnail generation: don't include timestamp of archived files [04:07:02] [02puppet] 07The-Voidwalker created branch 03The-Voidwalker-patch-1 - 13https://github.com/miraheze/puppet [04:07:04] [02puppet] 07The-Voidwalker opened pull request 03#3473: thumbnail generation: don't include timestamp of archived files - 13https://github.com/miraheze/puppet/pull/3473 [04:08:00] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 3.31, 5.40, 3.65 [04:08:11] !log [macfan@test131] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/beta.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/LoginNotify/sql/mysql/tables-generated.sql (START) [04:08:12] !log [macfan@test131] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/beta.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/LoginNotify/sql/mysql/tables-generated.sql (END - exit=65280) [04:08:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:08:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:15:36] PROBLEM - mail121 Current Load on mail121 is CRITICAL: LOAD CRITICAL - total load average: 10.89, 8.33, 3.76 [04:21:06] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: connect to address 2a10:6740::6:406 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [04:23:06] PROBLEM - test131 MediaWiki Rendering on test131 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [04:27:18] PROBLEM - mail121 Current Load on mail121 is WARNING: LOAD WARNING - total load average: 0.06, 0.90, 1.82 [04:29:18] RECOVERY - mail121 Current Load on mail121 is OK: LOAD OK - total load average: 0.00, 0.60, 1.59 [04:33:37] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.zulaclub.net. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:33:42] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.62, 3.05, 2.58 [04:35:40] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.56, 2.87, 2.58 [04:45:15] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 700 bytes in 8.863 second response time [04:46:45] !log [macfan@test131] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/beta.json /srv/mediawiki/w/maintenance/sql.php composer require lcobucci/jwt:4.1.5 --update-no-dev (START) [04:46:50] !log [macfan@test131] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/beta.json /srv/mediawiki/w/maintenance/sql.php composer require lcobucci/jwt:4.1.5 --update-no-dev (END - exit=256) [04:46:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:46:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:47:26] !log [macfan@test131] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/beta.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/AbuseFilter/db_patches/mysql/patch-add-af_actor.sql (START) [04:47:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:47:35] !log [macfan@test131] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/beta.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/AbuseFilter/db_patches/mysql/patch-add-af_actor.sql (END - exit=0) [04:47:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:47:42] !log [macfan@test131] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/beta.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/AbuseFilter/db_patches/mysql/patch-add-afh_actor.sql (START) [04:47:47] !log [macfan@test131] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/beta.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/AbuseFilter/db_patches/mysql/patch-add-afh_actor.sql (END - exit=0) [04:47:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:47:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:48:09] !log [macfan@test131] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/beta.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/LoginNotify/sql/mysql/tables-generated.sql (START) [04:48:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:48:14] !log [macfan@test131] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/beta.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/LoginNotify/sql/mysql/tables-generated.sql (END - exit=0) [04:48:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:01:09] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.206 second response time [05:02:12] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp25.miraheze.org - CNAME OK [05:03:38] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.50, 3.22, 2.64 [05:05:38] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.17, 3.06, 2.64 [05:09:28] [02puppet] 07MacFan4000 opened pull request 03#3474: switch test131 to mw 1.41 - 13https://github.com/miraheze/puppet/pull/3474 [05:59:17] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 0.56, 1.64, 3.87 [06:03:17] RECOVERY - mon141 Current Load on mon141 is OK: LOAD OK - total load average: 1.16, 1.39, 3.25 [06:27:13] PROBLEM - ndgkb.nena.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ndgkb.nena.org' expires in 15 day(s) (Wed 08 Nov 2023 06:21:59 GMT +0000). [06:30:13] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 4.60, 5.21, 3.54 [06:34:10] RECOVERY - mon141 Current Load on mon141 is OK: LOAD OK - total load average: 2.46, 3.39, 3.15 [07:15:47] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 2.62, 5.81, 3.90 [07:28:28] PROBLEM - kb.nena.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kb.nena.org' expires in 15 day(s) (Wed 08 Nov 2023 07:00:57 GMT +0000). [09:13:02] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [09:17:02] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [09:30:42] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [09:31:17] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 1.21, 2.45, 3.88 [09:35:17] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 7.86, 4.96, 4.46 [09:40:24] PROBLEM - wiki.wisrail.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wisrail.de could not be found [09:45:17] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 2.47, 3.03, 3.89 [09:47:17] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 30.11, 16.49, 8.84 [10:00:43] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [10:17:17] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 2.95, 2.96, 3.92 [10:23:17] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 12.26, 7.26, 5.18 [10:31:17] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 1.11, 2.88, 3.89 [10:35:17] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 13.07, 5.84, 4.65 [10:39:17] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 1.65, 3.48, 3.95 [10:43:17] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 27.86, 8.61, 5.46 [11:16:42] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/f36676cf45f7...3530db18c5a2 [11:16:44] [02miraheze/CreateWiki] 07translatewiki 033530db1 - Localisation updates from https://translatewiki.net. [11:16:46] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/landing/compare/77c5af97468d...914f8ac4e5c7 [11:16:48] [02miraheze/landing] 07translatewiki 03914f8ac - Localisation updates from https://translatewiki.net. [11:16:49] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/MirahezeMagic/compare/ebee49702a37...8fe6508ac030 [11:16:50] [02miraheze/MirahezeMagic] 07translatewiki 038fe6508 - Localisation updates from https://translatewiki.net. [11:16:52] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/5b4864d1cb3a...3ac3e7b69b91 [11:16:55] [02miraheze/ManageWiki] 07translatewiki 033ac3e7b - Localisation updates from https://translatewiki.net. [11:16:56] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/cadb84261545...4c2b91653888 [11:16:57] [02miraheze/WikiDiscover] 07translatewiki 034c2b916 - Localisation updates from https://translatewiki.net. [11:16:58] [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/e803e6ca40bf...c6edba0bf518 [11:17:01] [02miraheze/DataDump] 07translatewiki 03c6edba0 - Localisation updates from https://translatewiki.net. [11:22:02] miraheze/landing - translatewiki the build passed. [11:22:42] miraheze/WikiDiscover - translatewiki the build passed. [11:23:41] miraheze/DataDump - translatewiki the build passed. [11:24:00] miraheze/ManageWiki - translatewiki the build passed. [11:30:01] miraheze/MirahezeMagic - translatewiki the build passed. [11:30:16] miraheze/CreateWiki - translatewiki the build passed. [11:30:58] [02puppet] 07paladox closed pull request 03#3473: thumbnail generation: don't include timestamp of archived files - 13https://github.com/miraheze/puppet/pull/3473 [11:31:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fd061c2a0f68...9bd62f7b7915 [11:31:03] [02miraheze/puppet] 07The-Voidwalker 039bd62f7 - thumbnail generation: don't include timestamp of archived files (#3473) [11:31:06] [02miraheze/puppet] 07paladox deleted branch 03The-Voidwalker-patch-1 [11:31:08] [02puppet] 07paladox deleted branch 03The-Voidwalker-patch-1 - 13https://github.com/miraheze/puppet [11:51:07] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 700 bytes in 0.869 second response time [11:53:07] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.966 second response time [12:06:13] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/f0f521d5e10a [12:06:16] [02miraheze/mw-config] 07paladox 03f0f521d - set wgCommentTempTableSchemaMigrationStage to SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_OLD on beta [12:06:18] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [12:06:25] [02mw-config] 07paladox opened pull request 03#5344: set wgCommentTempTableSchemaMigrationStage to SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_OLD on beta - 13https://github.com/miraheze/mw-config/pull/5344 [12:07:16] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f0f521d5e10a...173cb1e2c09e [12:07:19] [02miraheze/mw-config] 07github-actions 03173cb1e - CI: lint code to MediaWiki standards [12:07:20] [02mw-config] 07github-actions[bot] synchronize pull request 03#5344: set wgCommentTempTableSchemaMigrationStage to SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_OLD on beta - 13https://github.com/miraheze/mw-config/pull/5344 [12:08:02] miraheze/mw-config - paladox the build passed. [12:08:21] [02mw-config] 07paladox synchronize pull request 03#5344: set wgCommentTempTableSchemaMigrationStage to SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_OLD on beta - 13https://github.com/miraheze/mw-config/pull/5344 [12:08:25] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/173cb1e2c09e...d1fb3279393f [12:08:27] [02miraheze/mw-config] 07paladox 03d1fb327 - Update LocalSettings.php [12:10:06] [02mw-config] 07paladox closed pull request 03#5344: set wgCommentTempTableSchemaMigrationStage to SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_OLD on beta - 13https://github.com/miraheze/mw-config/pull/5344 [12:10:08] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1533a223f975...4fba1af17088 [12:10:10] [02miraheze/mw-config] 07paladox 034fba1af - set wgCommentTempTableSchemaMigrationStage to SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_OLD on beta (#5344) [12:10:13] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [12:10:15] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [12:10:25] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [12:10:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:10:43] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 18s [12:10:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:10:50] !log [paladox@mwtask141] starting deploy of {'config': True} to all [12:10:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:11:00] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 10s [12:11:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:11:13] miraheze/mw-config - paladox the build passed. [12:13:47] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4fba1af17088...b70d159ddb62 [12:13:50] [02miraheze/mw-config] 07paladox 03b70d159 - Set wgExternalLinksSchemaMigrationStage to SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_OLD on beta [12:13:51] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [12:13:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:14:01] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [12:14:03] !log [paladox@mwtask141] starting deploy of {'config': True} to all [12:14:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:14:10] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 6s [12:14:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:14:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:14:55] miraheze/mw-config - paladox the build passed. [12:15:22] PROBLEM - mail121 Current Load on mail121 is CRITICAL: LOAD CRITICAL - total load average: 8.38, 3.81, 1.64 [12:21:17] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 1.33, 1.73, 3.88 [12:21:54] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.51, 3.62, 3.25 [12:23:19] RECOVERY - mail121 Current Load on mail121 is OK: LOAD OK - total load average: 0.09, 1.38, 1.44 [12:23:41] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b70d159ddb62...0c4cb11ce564 [12:23:43] [02miraheze/mw-config] 07paladox 030c4cb11 - Set mirabeta to use SCHEMA_COMPAT_READ_NEW [12:23:55] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [12:23:56] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [12:23:57] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [12:24:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:12] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 16s [12:24:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:18] !log [paladox@mwtask141] starting deploy of {'config': True} to all [12:24:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:27] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [12:24:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:49] miraheze/mw-config - paladox the build passed. [12:25:46] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.05, 3.63, 3.31 [12:27:17] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 7.71, 5.89, 4.89 [12:29:38] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.70, 3.70, 3.42 [12:29:42] !log [paladox@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/maintenance/archives/patch-externallinks-el_to_path.sql [12:29:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:31:50] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/7c7a2424c390 [12:31:51] [02miraheze/mw-config] 07paladox 037c7a242 - Set wgCommentTempTableSchemaMigrationStage to SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_OLD [12:31:53] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [12:31:54] [02mw-config] 07paladox opened pull request 03#5345: Set wgCommentTempTableSchemaMigrationStage to SCHEMA_COMPAT_WRITE_BOT… - 13https://github.com/miraheze/mw-config/pull/5345 [12:32:18] [02mw-config] 07paladox closed pull request 03#5345: Set wgCommentTempTableSchemaMigrationStage to SCHEMA_COMPAT_WRITE_BOT… - 13https://github.com/miraheze/mw-config/pull/5345 [12:32:21] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0c4cb11ce564...c72425173420 [12:32:24] [02miraheze/mw-config] 07paladox 03c724251 - Set wgCommentTempTableSchemaMigrationStage to SCHEMA_COMPAT_WRITE_BOT… (#5345) [12:32:27] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [12:32:30] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [12:32:59] !log [paladox@mwtask141] v [12:33:01] !log [paladox@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/migrateRevisionCommentTemp.php [12:33:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:05] miraheze/mw-config - paladox the build passed. [12:33:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:21] miraheze/mw-config - paladox the build passed. [12:33:38] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.27, 3.80, 3.52 [12:34:08] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [12:34:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:34:20] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [12:34:26] !log [paladox@mwtask141] starting deploy of {'config': True} to all [12:34:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:34:37] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 11s [12:34:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:34:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:35:38] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.63, 3.59, 3.47 [12:36:49] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.90, 9.85, 9.16 [12:38:45] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 8.09, 9.51, 9.13 [12:43:38] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.42, 3.18, 3.31 [12:46:19] [02CreateWiki] 07paladox closed pull request 03#444: Update mediawiki/mediawiki-codesniffer requirement from 41.0.0 to 42.0.0 - 13https://github.com/miraheze/CreateWiki/pull/444 [12:46:22] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/3530db18c5a2...6f542a332191 [12:46:24] [02miraheze/CreateWiki] 07dependabot[bot] 036f542a3 - Update mediawiki/mediawiki-codesniffer requirement from 41.0.0 to 42.0.0 (#444) [12:46:25] [02miraheze/CreateWiki] 07paladox deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-42.0.0 [12:46:27] [02CreateWiki] 07paladox deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-42.0.0 - 13https://github.com/miraheze/CreateWiki [12:46:30] [02CreateWiki] 07dependabot[bot] edited pull request 03#439: Update mediawiki/mediawiki-phan-config requirement from 0.12.1 to 0.13.0 - 13https://github.com/miraheze/CreateWiki/pull/439 [12:46:31] [02CreateWiki] 07dependabot[bot] edited pull request 03#444: Update mediawiki/mediawiki-codesniffer requirement from 41.0.0 to 42.0.0 - 13https://github.com/miraheze/CreateWiki/pull/444 [12:47:03] [02CreateWiki] 07paladox closed pull request 03#445: Bump grunt-banana-checker from 0.11.0 to 0.11.1 - 13https://github.com/miraheze/CreateWiki/pull/445 [12:47:04] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/6f542a332191...7591287473ea [12:47:06] [02miraheze/CreateWiki] 07dependabot[bot] 037591287 - Bump grunt-banana-checker from 0.11.0 to 0.11.1 (#445) [12:47:09] [02miraheze/CreateWiki] 07paladox deleted branch 03dependabot/npm_and_yarn/grunt-banana-checker-0.11.1 [12:47:12] [02CreateWiki] 07paladox deleted branch 03dependabot/npm_and_yarn/grunt-banana-checker-0.11.1 - 13https://github.com/miraheze/CreateWiki [12:47:14] [02CreateWiki] 07dependabot[bot] edited pull request 03#436: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.1 - 13https://github.com/miraheze/CreateWiki/pull/436 [12:51:05] [02MirahezeMagic] 07paladox closed pull request 03#426: generateExtensionDatabaseList: support a --directory option - 13https://github.com/miraheze/MirahezeMagic/pull/426 [12:51:08] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/8fe6508ac030...494ba858cb44 [12:51:11] [02miraheze/MirahezeMagic] 07paladox 03494ba85 - generateExtensionDatabaseList: support a --directory option (#426) [12:52:11] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-2 [12:52:12] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/MirahezeMagic [12:53:05] !log [paladox@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/linter.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/Linter/sql/patch-linter-fix-params-null-definition.sql [12:53:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:55:42] !log [paladox@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/growthexperiments.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/GrowthExperiments/sql/mysql/growthexperiments_user_impact.sql [12:55:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:56:02] !log [paladox@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/growthexperiments.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/GrowthExperiments/sql/mysql/patch-add_gemm_mentee_is_active.sql [12:56:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:56:33] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.26, 3.76, 3.42 [12:56:58] !log [paladox@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/flaggedrevs.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/FlaggedRevs/backend/schema/mysql/patch-flaggedrevs_tracking-unique-to-pk.sql [12:57:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:57:48] miraheze/CreateWiki - paladox the build has errored. [12:58:29] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.62, 3.74, 3.46 [12:58:30] !log [paladox@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/wikibaserepo.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/Wikibase/repo/sql/mysql/archives/patch-wb_id_counters-unique-to-pk.sql [12:58:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:00:04] PROBLEM - wiki.litek.top - reverse DNS on sslhost is UNKNOWN: [13:00:29] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.80, 4.25, 3.66 [13:01:01] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.57, 10.68, 9.29 [13:01:43] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is UNKNOWN: [13:02:00] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.19, 3.35, 1.90 [13:02:26] PROBLEM - mon141 grafana.miraheze.org HTTPS on mon141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:02:28] PROBLEM - mon141 grafana.miraheze.org HTTPS on mon141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:02:52] PROBLEM - lgbtqia.wiki - reverse DNS on sslhost is UNKNOWN: [13:03:45] miraheze/MirahezeMagic - paladox the build passed. [13:04:38] PROBLEM - wiki.cdntennis.ca - reverse DNS on sslhost is UNKNOWN: PROBLEM - fid.koymi.net - reverse DNS on sslhost is UNKNOWN: PROBLEM - schizoidnightmares.miraheze.org - reverse DNS on sslhost is UNKNOWN: [13:04:58] [02puppet] 07paladox commented on pull request 03#3474: switch test131 to mw 1.41 - 13https://github.com/miraheze/puppet/pull/3474#issuecomment-1775148640 [13:05:02] [02puppet] 07paladox closed pull request 03#3474: switch test131 to mw 1.41 - 13https://github.com/miraheze/puppet/pull/3474 [13:05:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9bd62f7b7915...4f9a9bc43dde [13:05:04] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.85, 2.53, 1.83 [13:05:04] PROBLEM - template.miraheze.org - reverse DNS on sslhost is UNKNOWN: [13:05:04] PROBLEM - mon141 APT on mon141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:05:04] [02miraheze/puppet] 07MacFan4000 034f9a9bc - switch test131 to mw 1.41 (#3474) [13:05:09] RECOVERY - mon141 grafana.miraheze.org HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 200 OK - 43411 bytes in 0.222 second response time [13:06:05] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:06:06] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.10, 9.06, 9.00 [13:07:17] PROBLEM - smashbroswiki.com - reverse DNS on sslhost is UNKNOWN: SSL OK - smashbroswiki.com reverse DNS resolves to cp25.miraheze.org - NS RECORDS OK [13:07:18] PROBLEM - data.nonbinary.wiki - reverse DNS on sslhost is UNKNOWN: PROBLEM - programming.red - reverse DNS on sslhost is UNKNOWN: PROBLEM - storytime.jdstroy.cf - reverse DNS on sslhost is UNKNOWN: PROBLEM - hr.petrawiki.org - reverse DNS on sslhost is UNKNOWN: ceeded.>SSL OK - hr.petrawiki.org reverse DNS resolves to cp25.miraheze.org - NS RECORDS OK PROBLEM - johanloopmans.nl - reverse DNS on sslhost is UNKNOWN: SSL OK - johanloopmans.nl reverse DNS resolves to cp24.miraheze.org - NS RECORDS OK PROBLEM - slymods.info - reverse DNS on sslhost is UNKNOWN: ed).> [13:07:22] PROBLEM - pastport.org - reverse DNS on sslhost is UNKNOWN: [13:07:22] PROBLEM - www.trollpasta.com - reverse DNS on sslhost is UNKNOWN: [13:07:22] PROBLEM - adadevelopersacademy.wiki - reverse DNS on sslhost is UNKNOWN: [13:07:32] RECOVERY - mon141 APT on mon141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:09:52] RECOVERY - test131 APT on test131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:11:39] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.20, 3.98, 3.95 [13:13:40] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.33, 4.11, 4.01 [13:15:41] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.28, 3.79, 3.90 [13:17:23] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:19:18] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.285 second response time [13:19:23] !log [@test131] starting deploy of {'config': True} to all [13:19:24] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [13:19:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:19:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:20:11] PROBLEM - test131 APT on test131 is CRITICAL: APT CRITICAL: 528 packages available for upgrade (3 critical updates). [13:23:38] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.15, 3.81, 3.82 [13:25:21] PROBLEM - test131 Puppet on test131 is UNKNOWN: NRPE: Unable to read output [13:25:46] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:27:38] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.27, 3.67, 3.77 [13:29:40] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.20, 10.25, 9.56 [13:29:56] PROBLEM - wiki.litek.top - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.litek.top could not be found [13:30:55] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:32:27] PROBLEM - lgbtqia.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lgbtqia.wiki could not be found [13:32:40] RECOVERY - test131 APT on test131 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [13:33:34] RECOVERY - fid.koymi.net - reverse DNS on sslhost is OK: SSL OK - fid.koymi.net reverse DNS resolves to cp25.miraheze.org - CNAME OK [13:33:35] RECOVERY - wiki.cdntennis.ca - reverse DNS on sslhost is OK: SSL OK - wiki.cdntennis.ca reverse DNS resolves to cp25.miraheze.org - CNAME OK [13:33:36] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 9.07, 9.72, 9.52 [13:34:06] RECOVERY - template.miraheze.org - reverse DNS on sslhost is OK: SSL OK - template.miraheze.org reverse DNS resolves to cp25.miraheze.org - NS RECORDS OK [13:34:16] RECOVERY - schizoidnightmares.miraheze.org - reverse DNS on sslhost is OK: SSL OK - schizoidnightmares.miraheze.org reverse DNS resolves to cp25.miraheze.org - NS RECORDS OK [13:36:42] RECOVERY - data.nonbinary.wiki - reverse DNS on sslhost is OK: SSL OK - data.nonbinary.wiki reverse DNS resolves to cp25.miraheze.org - CNAME OK [13:36:42] RECOVERY - johanloopmans.nl - reverse DNS on sslhost is OK: SSL OK - johanloopmans.nl reverse DNS resolves to cp25.miraheze.org - NS RECORDS OK [13:36:43] RECOVERY - hr.petrawiki.org - reverse DNS on sslhost is OK: SSL OK - hr.petrawiki.org reverse DNS resolves to cp24.miraheze.org - NS RECORDS OK [13:36:45] RECOVERY - smashbroswiki.com - reverse DNS on sslhost is OK: SSL OK - smashbroswiki.com reverse DNS resolves to cp24.miraheze.org - NS RECORDS OK [13:36:45] RECOVERY - programming.red - reverse DNS on sslhost is OK: SSL OK - programming.red reverse DNS resolves to cp25.miraheze.org - NS RECORDS OK [13:36:45] RECOVERY - storytime.jdstroy.cf - reverse DNS on sslhost is OK: SSL OK - storytime.jdstroy.cf reverse DNS resolves to cp24.miraheze.org - CNAME OK [13:36:48] RECOVERY - slymods.info - reverse DNS on sslhost is OK: SSL OK - slymods.info reverse DNS resolves to cp25.miraheze.org - CNAME FLAT [13:36:56] RECOVERY - www.trollpasta.com - reverse DNS on sslhost is OK: SSL OK - www.trollpasta.com reverse DNS resolves to cp24.miraheze.org - NS RECORDS OK [13:36:56] RECOVERY - pastport.org - reverse DNS on sslhost is OK: SSL OK - pastport.org reverse DNS resolves to cp24.miraheze.org - NS RECORDS OK [13:36:58] RECOVERY - adadevelopersacademy.wiki - reverse DNS on sslhost is OK: SSL OK - adadevelopersacademy.wiki reverse DNS resolves to cp24.miraheze.org - NS RECORDS OK [13:37:39] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.04, 2.73, 3.31 [13:42:41] PROBLEM - test131 APT on test131 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [13:45:38] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.48, 3.73, 3.47 [13:46:50] PROBLEM - test131 APT on test131 is CRITICAL: APT CRITICAL: 151 packages available for upgrade (5 critical updates). [13:47:06] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.088 second response time [13:47:31] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.44, 10.12, 9.54 [13:47:38] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.73, 3.76, 3.52 [13:48:24] PROBLEM - test131 php-fpm on test131 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.4' [13:48:54] RECOVERY - test131 APT on test131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:49:31] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 13.17, 10.74, 9.81 [13:49:38] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.75, 4.05, 3.65 [13:51:31] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.96, 10.67, 9.89 [13:52:14] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.28, 10.59, 9.87 [13:55:32] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.16, 11.07, 10.20 [13:56:11] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.75, 9.89, 9.79 [13:57:31] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 6.87, 9.53, 9.75 [13:57:38] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.85, 3.89, 3.80 [14:01:38] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.77, 4.00, 3.84 [14:05:38] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.26, 3.94, 3.88 [14:07:56] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c72425173420...fc0805d9214a [14:07:57] [02miraheze/mw-config] 07paladox 03fc0805d - Lower memcached timeout to 250ms [14:08:24] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [14:08:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:09:05] miraheze/mw-config - paladox the build passed. [14:09:20] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 56s [14:09:27] !log [paladox@mwtask141] starting deploy of {'config': True} to all [14:09:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:09:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:09:38] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 11s [14:09:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:10:03] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.15, 4.03, 2.44 [14:11:06] PROBLEM - test131 JobRunner Service on test131 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [14:11:59] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.00, 3.64, 2.48 [14:13:56] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.66, 2.58, 2.24 [14:15:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4f9a9bc43dde...ae149bd4e590 [14:15:15] [02miraheze/puppet] 07paladox 03ae149bd - Lower nutcracker timeout [14:17:06] RECOVERY - test131 JobRunner Service on test131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [14:17:38] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.29, 2.84, 3.32 [14:19:02] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [14:21:51] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 9.07, 10.29, 9.60 [14:23:02] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [14:23:46] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 8.40, 9.52, 9.39 [14:25:16] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [14:25:42] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.80, 3.08, 2.33 [14:26:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ae149bd4e590...c2b35fe43d7f [14:27:01] [02miraheze/puppet] 07paladox 03c2b35fe - nutcracker: tune [14:27:39] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.76, 3.37, 2.53 [14:29:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c2b35fe43d7f...c3f1f2a68634 [14:29:46] [02miraheze/puppet] 07paladox 03c3f1f2a - test131: switch to php 8.2 [14:31:23] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.56, 3.48, 3.34 [14:31:33] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.85, 2.87, 2.54 [14:33:20] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.46, 3.81, 3.47 [14:36:24] RECOVERY - test131 php-fpm on test131 is OK: PROCS OK: 11 processes with command name 'php-fpm8.2' [14:37:12] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.64, 3.52, 3.46 [14:37:13] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.806 second response time [14:41:37] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.92, 2.80, 2.42 [14:43:00] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.13, 3.35, 3.40 [14:44:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/d40528675dcb [14:44:17] [02miraheze/puppet] 07paladox 03d405286 - mediawiki: use python3-internetarchive deb package on bookworm+ [14:44:19] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [14:44:20] [02puppet] 07paladox opened pull request 03#3475: mediawiki: use python3-internetarchive deb package on bookworm+ - 13https://github.com/miraheze/puppet/pull/3475 [14:44:35] [02puppet] 07paladox closed pull request 03#3475: mediawiki: use python3-internetarchive deb package on bookworm+ - 13https://github.com/miraheze/puppet/pull/3475 [14:44:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c3f1f2a68634...23a9b4061c1a [14:44:39] [02miraheze/puppet] 07paladox 0323a9b40 - mediawiki: use python3-internetarchive deb package on bookworm+ (#3475) [14:44:40] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [14:44:42] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [14:45:32] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.51, 2.95, 2.61 [14:51:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/23a9b4061c1a...82350cec0a22 [14:51:25] [02miraheze/puppet] 07paladox 0382350ce - mediawiki::deploy: fix installing langcodes under bookworm [14:54:48] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:57:17] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 0.91, 1.90, 3.89 [14:59:17] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 8.92, 4.74, 4.68 [14:59:37] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.43, 10.79, 9.42 [15:00:36] [02miraheze/mediawiki] 07MacFan4000 pushed 031 commit to 03REL1_41 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/ce6708ab0116...819348b065cc [15:00:37] [02miraheze/mediawiki] 07MacFan4000 03819348b - Re-update EmbedVideo [15:02:10] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.39, 9.61, 8.85 [15:02:19] !log [macfan@test131] starting deploy of {'world': True, 'l10n': True} to all [15:02:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:03:33] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.39, 9.49, 9.17 [15:04:05] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 9.14, 9.51, 8.91 [15:05:17] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 2.37, 2.97, 3.91 [15:06:13] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.86, 3.32, 2.85 [15:08:10] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.39, 3.36, 2.92 [15:11:17] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 14.45, 9.66, 6.16 [15:12:07] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.82, 4.16, 3.33 [15:14:03] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.73, 3.60, 3.22 [15:15:03] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/commit/aeb23d1839ea [15:15:04] [02miraheze/MirahezeMagic] 07paladox 03aeb23d1 - Fix "trim(): Passing null to parameter #1 ($string) " [15:15:07] [02MirahezeMagic] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/MirahezeMagic [15:15:09] [02MirahezeMagic] 07paladox opened pull request 03#439: Fix "trim(): Passing null to parameter #1 ($string) " - 13https://github.com/miraheze/MirahezeMagic/pull/439 [15:16:00] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.05, 3.83, 3.35 [15:17:17] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 12.20, 10.26, 9.52 [15:18:32] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 9.68, 10.25, 9.66 [15:19:07] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.69, 3.02, 2.20 [15:19:15] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 9.18, 9.62, 9.37 [15:19:56] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.23, 3.97, 3.54 [15:20:27] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 8.42, 9.71, 9.54 [15:21:08] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.60, 3.52, 2.48 [15:21:53] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.76, 3.15, 3.29 [15:22:06] [02mw-config] 07JerryHan3 opened pull request 03#5346: Setup dlfmwiki for system messages' translation. - 13https://github.com/miraheze/mw-config/pull/5346 [15:23:10] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 14.38, 11.86, 10.32 [15:25:08] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.08, 11.24, 10.26 [15:26:48] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.52, 4.07, 3.57 [15:29:04] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 12.78, 11.61, 10.59 [15:29:34] miraheze/MirahezeMagic - paladox the build passed. [15:30:43] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [15:30:57] miraheze/mw-config - JerryHan3 the build passed. [15:31:02] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.85, 11.48, 10.69 [15:32:50] PROBLEM - test131 Puppet on test131 is UNKNOWN: NRPE: Unable to read output [15:34:58] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 7.05, 9.28, 10.00 [15:35:49] PROBLEM - db131 Current Load on db131 is CRITICAL: CRITICAL - load average: 8.17, 7.83, 5.64 [15:37:49] PROBLEM - db131 Current Load on db131 is WARNING: WARNING - load average: 6.62, 7.65, 5.86 [15:41:49] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 6.04, 6.70, 5.86 [15:42:16] !log [macfan@test131] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 2397s [15:42:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:46:14] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.45, 3.60, 3.96 [15:46:31] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.25, 11.19, 10.42 [15:48:10] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.18, 3.81, 4.00 [15:48:27] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 9.02, 10.64, 10.33 [15:50:06] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.83, 3.80, 3.97 [15:50:22] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 8.34, 9.86, 10.08 [15:53:58] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.01, 3.95, 3.99 [15:57:31] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.52, 10.17, 9.70 [15:59:29] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 9.32, 9.56, 9.52 [16:00:42] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:03:17] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 1.98, 2.41, 3.75 [16:07:07] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 1.60, 2.54, 3.92 [16:07:17] RECOVERY - mon141 Current Load on mon141 is OK: LOAD OK - total load average: 1.78, 2.28, 3.40 [16:09:38] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.57, 3.74, 4.00 [16:11:07] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 0.54, 1.43, 3.14 [16:15:38] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.71, 4.00, 3.96 [16:17:38] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.21, 3.72, 3.86 [16:22:12] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 7.31, 12.27, 7.51 [16:25:38] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.96, 2.81, 3.38 [16:30:42] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [16:36:32] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.51, 3.24, 3.27 [16:37:04] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 13.25, 10.27, 8.87 [16:39:02] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 9.93, 10.06, 8.96 [16:40:25] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.71, 3.15, 3.24 [17:00:43] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.gab.pt.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:18:12] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.25, 4.38, 3.51 [17:23:00] PROBLEM - wiki.susqu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.susqu.org' expires in 15 day(s) (Wed 08 Nov 2023 17:10:13 GMT +0000). [17:24:00] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.06, 3.89, 3.59 [17:31:44] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.82, 3.24, 3.40 [17:37:09] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.54, 3.21, 2.87 [17:37:38] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.16, 3.43, 3.43 [17:39:09] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.85, 3.14, 2.89 [17:39:38] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.28, 3.00, 3.28 [17:45:38] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.73, 3.27, 3.28 [17:47:38] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.39, 2.93, 3.15 [17:47:58] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/fc0805d9214a...766988a84178 [17:47:59] [02miraheze/mw-config] 07paladox 03766988a - Set wgCheckUserEventTablesMigrationStage to SCHEMA_COMPAT_OLD [17:48:22] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [17:48:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:48:32] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [17:48:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:48:38] !log [paladox@mwtask141] starting deploy of {'config': True} to all [17:48:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:48:45] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [17:48:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:49:05] miraheze/mw-config - paladox the build passed. [17:49:40] PROBLEM - wiki.mc.marochuru.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mc.marochuru.net' expires in 15 day(s) (Wed 08 Nov 2023 17:21:20 GMT +0000). [17:50:19] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_40 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/36082bebe9ad...750664125a41 [17:50:20] [02miraheze/mediawiki] 07paladox 037506641 - Update CheckUser [17:50:39] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [17:50:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:50:55] PROBLEM - sonicpedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sonicpedia.org' expires in 15 day(s) (Wed 08 Nov 2023 17:40:25 GMT +0000). [17:55:23] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 1.46, 1.83, 3.87 [17:56:35] !log [@test131] starting deploy of {'config': True} to all [17:56:36] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [17:56:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:56:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:58:10] PROBLEM - wiki.rsf.world - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.rsf.world' expires in 15 day(s) (Wed 08 Nov 2023 17:39:39 GMT +0000). [17:59:22] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 8.79, 5.96, 5.00 [17:59:30] !log [paladox@mwtask141] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 531s [17:59:38] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.10, 3.37, 3.16 [17:59:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:01:43] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.91, 3.47, 3.22 [18:03:39] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.00, 3.39, 3.23 [18:19:17] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 2.65, 2.26, 3.88 [18:21:17] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 6.75, 5.55, 5.00 [19:02:02] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.56, 2.63, 1.66 [19:03:58] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.53, 1.80, 1.47 [19:13:45] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.90, 2.78, 1.91 [19:15:42] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.72, 2.76, 2.00 [19:43:38] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.96, 3.47, 3.00 [19:45:39] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.88, 3.28, 2.99 [19:55:17] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 2.27, 2.78, 3.91 [19:57:38] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.03, 3.55, 3.18 [19:59:17] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 19.34, 7.32, 5.16 [19:59:38] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.00, 3.41, 3.18 [20:01:38] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.32, 3.04, 3.08 [20:17:03] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [20:21:02] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [20:49:17] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 1.99, 1.97, 3.98 [20:55:17] RECOVERY - mon141 Current Load on mon141 is OK: LOAD OK - total load average: 1.14, 1.52, 3.13 [21:48:58] PROBLEM - sonicpedia.org - LetsEncrypt on sslhost is CRITICAL: connect to address sonicpedia.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [21:53:21] PROBLEM - cloud13 Puppet on cloud13 is UNKNOWN: NRPE: Unable to read output [21:56:10] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 2.26, 4.64, 3.16 [21:58:09] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 2.55, 3.97, 3.09 [22:00:08] RECOVERY - mon141 Current Load on mon141 is OK: LOAD OK - total load average: 1.38, 3.05, 2.86 [22:04:05] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 36.83, 17.12, 8.23 [22:07:11] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 7.24, 4.37, 2.53 [22:11:09] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.97, 3.54, 2.66 [22:13:09] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.28, 2.76, 2.48 [22:18:55] PROBLEM - sonicpedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sonicpedia.org' expires in 15 day(s) (Wed 08 Nov 2023 17:40:25 GMT +0000). [22:21:21] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:23:55] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 1.65, 2.09, 3.90 [22:27:52] RECOVERY - mon141 Current Load on mon141 is OK: LOAD OK - total load average: 1.65, 1.86, 3.38 [22:37:09] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.94, 3.82, 3.03 [22:43:09] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.17, 3.40, 3.08 [22:44:43] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 30.91, 10.78, 5.62 [22:52:40] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 1.96, 3.56, 3.99 [23:00:36] RECOVERY - mon141 Current Load on mon141 is OK: LOAD OK - total load average: 1.55, 2.55, 3.35 [23:04:26] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 12.77, 7.57, 3.55 [23:04:33] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 3.22, 3.98, 3.83 [23:04:55] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.07, 3.45, 3.06 [23:07:35] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 30.20, 13.92, 9.21 [23:08:26] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 1.56, 6.01, 4.02 [23:08:30] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 7.31, 6.35, 4.79 [23:08:47] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.72, 3.36, 3.14 [23:09:54] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:10:04] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 12.88, 9.60, 5.93 [23:10:14] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 16.16, 13.71, 8.13 [23:10:26] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw131 mw142 [23:10:43] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw131 mw142 [23:11:09] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.94, 3.10, 2.56 [23:11:13] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw142 [23:11:21] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw142 [23:11:28] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 29.40, 22.37, 11.74 [23:12:04] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 6.63, 8.63, 6.05 [23:12:42] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.99, 4.17, 3.52 [23:12:46] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 6.31, 4.12, 3.01 [23:13:04] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 12.49, 11.25, 8.76 [23:13:09] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.79, 3.08, 2.63 [23:13:18] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 13.74, 10.25, 7.56 [23:13:21] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [23:13:55] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.367 second response time [23:14:14] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 6.55, 10.30, 8.05 [23:14:26] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [23:14:27] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 1.68, 3.15, 3.83 [23:14:43] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [23:15:13] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [23:15:49] PROBLEM - db131 Current Load on db131 is CRITICAL: CRITICAL - load average: 9.62, 6.60, 4.31 [23:16:14] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 5.44, 8.66, 7.72 [23:17:00] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.41, 11.53, 9.49 [23:17:18] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 10.38, 10.44, 8.27 [23:17:22] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 3.98, 10.74, 9.92 [23:17:49] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 4.10, 5.49, 4.17 [23:18:25] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 13.41, 4.50, 3.99 [23:18:36] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.23, 3.73, 3.57 [23:19:18] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 9.08, 9.70, 8.25 [23:19:20] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 3.22, 8.19, 9.08 [23:20:24] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 2.70, 3.31, 3.61 [23:21:31] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 9.02, 11.38, 10.82 [23:22:23] RECOVERY - mon141 Current Load on mon141 is OK: LOAD OK - total load average: 2.50, 2.87, 3.40 [23:22:24] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 1.98, 3.75, 3.63 [23:24:53] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 12.90, 10.78, 9.79 [23:25:32] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 14.27, 12.12, 11.16 [23:26:13] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.98, 4.18, 3.80 [23:26:20] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.84, 3.94, 3.66 [23:27:31] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.65, 11.74, 11.16 [23:28:08] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.00, 3.72, 3.68 [23:28:49] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 9.17, 10.58, 9.99 [23:30:47] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.30, 10.16, 9.93 [23:31:58] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 1.98, 2.68, 3.25 [23:35:31] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 7.30, 8.66, 9.95 [23:36:00] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.41, 3.53, 3.72 [23:41:48] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.70, 2.97, 3.40