[00:00:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.41, 6.51, 6.60 [00:00:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 6.87, 7.42, 7.27 [00:00:38] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:03:58] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 17.98, 9.03, 4.04 [00:04:23] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 5.15, 6.17, 6.78 [00:05:52] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 10.61, 7.12, 4.65 [00:15:39] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.41, 3.61, 2.14 [00:15:57] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 4.44, 7.61, 7.15 [00:19:31] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.15, 3.63, 2.50 [00:19:57] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 4.93, 6.47, 6.76 [00:21:26] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.62, 3.40, 2.56 [00:23:18] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 8.67, 7.35, 6.86 [00:25:27] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.20, 3.99, 3.01 [00:25:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.11, 6.95, 6.70 [00:27:16] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 7.73, 7.64, 7.10 [00:27:23] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.71, 3.91, 3.10 [00:27:47] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.82, 6.54, 6.58 [00:29:15] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 8.66, 7.96, 7.27 [00:30:32] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [00:33:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 7.90, 8.00, 7.45 [00:37:01] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 8.68, 5.10, 3.82 [00:38:08] PROBLEM - db131 Current Load on db131 is CRITICAL: CRITICAL - load average: 11.12, 10.87, 6.80 [00:41:09] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 8.32, 7.53, 7.35 [00:42:08] PROBLEM - db131 Current Load on db131 is WARNING: WARNING - load average: 4.16, 7.08, 6.15 [00:43:08] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 7.97, 7.68, 7.43 [00:44:08] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 5.49, 6.39, 5.99 [00:48:56] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.45, 3.74, 3.73 [00:49:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.09, 7.03, 6.63 [00:52:56] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.63, 4.15, 3.89 [00:53:00] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.47, 6.92, 6.71 [00:54:57] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.43, 6.73, 6.66 [00:56:57] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.11, 3.82, 3.84 [01:01:00] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 8.29, 7.83, 7.64 [01:04:58] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 7.29, 7.55, 7.58 [01:06:55] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.24, 4.00, 3.77 [01:22:50] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.96, 5.75, 6.74 [01:24:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/1abc9fabe47f [01:24:22] [02miraheze/puppet] 07paladox 031abc9fa - Update jobrunner-hi: remove links group, decrease parsoid to 2 runners [01:24:23] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [01:24:26] [02puppet] 07paladox opened pull request 03#3500: Update jobrunner-hi: remove links group, decrease parsoid to 2 runners - 13https://github.com/miraheze/puppet/pull/3500 [01:24:53] [02puppet] 07paladox closed pull request 03#3500: Update jobrunner-hi: remove links group, decrease parsoid to 2 runners - 13https://github.com/miraheze/puppet/pull/3500 [01:24:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bf881043969f...e30a7ef48439 [01:24:55] [02miraheze/puppet] 07paladox 03e30a7ef - Update jobrunner-hi: remove links group, decrease parsoid to 2 runners (#3500) [01:24:55] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 1.79, 3.61, 3.99 [01:24:56] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [01:24:58] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [01:30:32] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:30:55] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 0.96, 2.02, 3.14 [01:37:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.03, 7.53, 6.86 [01:39:38] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.92, 6.57, 6.59 [01:48:20] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.70, 2.78, 3.88 [01:51:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.78, 7.18, 6.77 [01:53:12] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.99, 6.78, 6.67 [01:57:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.61, 7.54, 7.01 [01:57:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.52, 6.92, 6.22 [01:58:19] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.21, 2.59, 3.32 [01:59:00] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.98, 7.47, 7.03 [02:01:30] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.92, 6.69, 6.32 [02:06:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.08, 7.23, 7.01 [02:08:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.42, 6.76, 6.85 [02:10:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.32, 6.53, 6.75 [02:14:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.22, 7.25, 6.98 [02:16:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.91, 7.56, 7.13 [02:20:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.91, 7.88, 7.29 [02:22:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.92, 7.58, 7.26 [02:30:32] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [02:32:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.70, 7.03, 6.94 [02:34:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.31, 6.58, 6.77 [02:34:19] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.48, 2.89, 2.65 [02:35:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:36:20] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.88, 2.51, 2.54 [02:42:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [03:02:32] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:12:23] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.gab.pt.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:16:04] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [03:31:34] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [03:58:19] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.52, 3.08, 2.78 [04:00:19] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.66, 2.93, 2.76 [04:00:36] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [04:12:21] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [04:18:04] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [04:32:33] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [05:01:35] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:03:11] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.39, 3.72, 3.04 [05:05:06] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.76, 3.69, 3.10 [05:07:03] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.85, 3.29, 3.02 [05:12:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [05:17:12] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [05:20:30] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.31, 3.51, 3.20 [05:24:20] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.07, 3.35, 3.22 [05:30:38] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [05:42:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [06:12:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [06:18:16] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [06:34:19] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.66, 3.30, 3.01 [06:35:34] PROBLEM - wonderfuleveryday.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wonderfuleveryday.org' expires in 15 day(s) (Wed 29 Nov 2023 06:22:12 GMT +0000). [06:35:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [06:35:47] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ba4d9898b3a1...a289746af89e [06:35:49] [02miraheze/ssl] 07MirahezeSSLBot 03a289746 - Bot: Update SSL cert for wonderfuleveryday.org [06:36:19] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.29, 2.95, 2.92 [06:47:49] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [06:48:23] PROBLEM - jobchron121 Current Load on jobchron121 is CRITICAL: LOAD CRITICAL - total load average: 10.87, 6.97, 3.46 [06:49:30] PROBLEM - mw142 JobRunner Service on mw142 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [06:49:36] PROBLEM - mw141 JobRunner Service on mw141 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [06:49:49] PROBLEM - mw131 JobRunner Service on mw131 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [06:50:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.31, 6.30, 5.70 [06:50:24] PROBLEM - mw132 JobRunner Service on mw132 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [06:50:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [06:50:36] PROBLEM - mail121 Current Load on mail121 is CRITICAL: LOAD CRITICAL - total load average: 3.31, 3.44, 1.58 [06:52:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.11, 5.94, 5.65 [06:52:17] RECOVERY - jobchron121 Current Load on jobchron121 is OK: LOAD OK - total load average: 0.43, 3.27, 2.72 [06:53:44] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.64, 3.45, 3.17 [06:54:36] RECOVERY - mail121 Current Load on mail121 is OK: LOAD OK - total load average: 0.07, 1.56, 1.22 [06:55:39] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.54, 3.19, 3.12 [07:01:30] RECOVERY - mw142 JobRunner Service on mw142 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [07:03:21] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.05, 3.30, 3.13 [07:05:16] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.00, 3.08, 3.06 [07:06:24] RECOVERY - mw132 JobRunner Service on mw132 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [07:07:36] RECOVERY - mw141 JobRunner Service on mw141 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [07:19:20] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [07:33:41] RECOVERY - wonderfuleveryday.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wonderfuleveryday.org' will expire on Sun 11 Feb 2024 05:35:40 GMT +0000. [07:55:30] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [08:12:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [08:23:02] PROBLEM - cloud14 Puppet on cloud14 is UNKNOWN: NRPE: Unable to read output [08:51:02] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [09:03:32] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [09:12:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [09:32:34] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [09:50:19] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.42, 2.95, 2.49 [09:52:20] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.73, 2.78, 2.48 [10:32:21] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.30, 3.37, 2.84 [10:34:19] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.03, 3.31, 2.89 [10:56:10] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f0ab4696d78a...8b08efdf75fd [10:56:13] [02miraheze/mw-config] 07paladox 038b08efd - Revert "Enable warmup on all wikis" [10:57:22] miraheze/mw-config - paladox the build passed. [11:01:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e30a7ef48439...6655a438ee73 [11:01:39] [02miraheze/puppet] 07paladox 036655a43 - jobrunner-hi: increase parsoid workers to 4 [11:03:49] RECOVERY - mw131 JobRunner Service on mw131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [11:11:20] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 6.45, 6.98, 5.54 [11:15:19] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 5.66, 6.56, 5.72 [11:25:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 6.50, 6.81, 6.26 [11:27:12] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 5.85, 6.49, 6.21 [11:40:30] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [11:59:54] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 7.05, 6.58, 6.26 [12:00:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6655a438ee73...fda7df0c3528 [12:00:56] [02miraheze/puppet] 07paladox 03fda7df0 - jobrunner-hi: increase parseroidcache to 8 [12:01:53] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 4.28, 1.34, 0.47 [12:11:47] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.60, 9.48, 5.18 [12:12:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [12:15:37] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/ea58bedbed5e...dfdba17d0191 [12:15:40] [02miraheze/CreateWiki] 07translatewiki 03dfdba17 - Localisation updates from https://translatewiki.net. [12:15:41] [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/34677c88667b...e7eb705e8cc8 [12:15:43] [02miraheze/DataDump] 07translatewiki 03e7eb705 - Localisation updates from https://translatewiki.net. [12:15:46] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.70, 10.81, 6.68 [12:17:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.11, 10.78, 7.17 [12:19:44] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.18, 11.22, 7.76 [12:21:43] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.85, 11.04, 8.12 [12:21:48] miraheze/DataDump - translatewiki the build passed. [12:29:02] miraheze/CreateWiki - translatewiki the build passed. [12:30:55] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.76, 2.98, 2.17 [12:32:56] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.92, 2.87, 2.23 [12:33:37] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 9.20, 9.99, 9.16 [12:42:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [12:59:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.39, 6.86, 6.17 [13:01:30] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.34, 6.60, 6.18 [13:14:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.99, 7.06, 6.72 [13:15:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.82, 6.85, 6.47 [13:17:30] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.59, 6.30, 6.31 [13:30:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.18, 6.61, 6.76 [13:34:13] PROBLEM - phab121 Current Load on phab121 is CRITICAL: LOAD CRITICAL - total load average: 2.77, 2.06, 1.16 [13:34:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.21, 7.30, 7.03 [13:36:13] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 1.14, 1.62, 1.10 [13:44:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.56, 7.40, 7.05 [13:46:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.52, 6.85, 6.89 [13:48:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.36, 6.61, 6.79 [13:49:32] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8b08efdf75fd...83ef1bf58342 [13:49:34] [02miraheze/mw-config] 07paladox 0383ef1bf - Enable warmup on wikis begging with p, q, r and s [13:49:37] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [13:49:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:49:48] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [13:49:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:49:54] !log [paladox@mwtask141] starting deploy of {'config': True} to all [13:49:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:50:01] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [13:50:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:50:24] miraheze/mw-config - paladox the build passed. [13:52:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.72, 6.91, 6.89 [13:56:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.48, 6.38, 6.69 [13:56:44] !log [@test131] starting deploy of {'config': True} to all [13:56:45] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [13:56:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:56:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:12:21] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [14:42:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [14:52:50] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/83ef1bf58342...47d3991a397e [14:52:51] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [14:52:52] [02miraheze/mw-config] 07paladox 0347d3991 - Enable warmup on wikis begining with t [14:52:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:53:02] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [14:53:03] !log [paladox@mwtask141] starting deploy of {'config': True} to all [14:53:09] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 5s [14:53:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:53:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:53:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:53:52] miraheze/mw-config - paladox the build passed. [14:56:40] !log [@test131] starting deploy of {'config': True} to all [14:56:41] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [14:56:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:57:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:15:47] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.87, 3.55, 2.59 [15:17:43] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.68, 3.24, 2.60 [15:20:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.09, 6.93, 6.54 [15:22:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.12, 6.51, 6.43 [15:25:11] PROBLEM - www.pyramidgames.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.pyramidgames.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:48:33] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/47d3991a397e...559c29654969 [15:48:34] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [15:48:35] [02miraheze/mw-config] 07paladox 03559c296 - Enable warmup on wikis beginning with u and v [15:48:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:48:44] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [15:48:45] !log [paladox@mwtask141] starting deploy of {'config': True} to all [15:48:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:48:55] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 9s [15:48:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:49:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:49:33] miraheze/mw-config - paladox the build passed. [15:55:11] PROBLEM - www.pyramidgames.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pyramidgames.wiki' expires in 8 day(s) (Wed 22 Nov 2023 14:28:59 GMT +0000). [15:56:31] !log [@test131] starting deploy of {'config': True} to all [15:56:32] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [15:56:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:56:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:00:19] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.14, 7.98, 6.65 [16:02:13] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.85, 7.28, 6.56 [16:02:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.32, 6.89, 6.58 [16:04:07] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.18, 6.67, 6.43 [16:05:47] PROBLEM - miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address miraheze.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:09:00] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/559c29654969...ee68b0848ade [16:09:01] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [16:09:01] [02miraheze/mw-config] 07paladox 03ee68b08 - Enable warmup on wikis beginning with w and x [16:09:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:09:10] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [16:09:11] !log [paladox@mwtask141] starting deploy of {'config': True} to all [16:09:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:09:17] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 5s [16:09:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:09:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:10:04] miraheze/mw-config - paladox the build passed. [16:10:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.71, 6.35, 6.51 [16:13:33] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query franchising.org.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:16:04] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [16:19:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.88, 7.48, 6.99 [16:22:59] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.80, 6.68, 6.78 [16:26:47] !log [@test131] starting deploy of {'config': True} to all [16:26:47] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [16:26:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:26:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:30:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.07, 7.42, 7.02 [16:31:01] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ee68b0848ade...cfce704c31f0 [16:31:02] [02miraheze/mw-config] 07paladox 03cfce704 - Enable warmup on wikis beginnings with y and z [16:31:13] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [16:31:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:31:22] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [16:31:25] !log [paladox@mwtask141] starting deploy of {'config': True} to all [16:31:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:31:31] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 6s [16:31:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:31:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:31:58] miraheze/mw-config - paladox the build passed. [16:32:41] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.37, 6.91, 6.28 [16:32:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.89, 6.96, 6.89 [16:34:35] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.02, 6.56, 6.22 [16:36:35] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.15, 6.63, 6.78 [16:40:13] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 7.65, 10.49, 7.71 [16:42:13] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 5.24, 8.57, 7.34 [16:43:17] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [16:48:09] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:49:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.05, 7.41, 6.92 [16:51:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.21, 7.06, 6.87 [16:55:06] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.43, 6.50, 6.71 [16:56:22] !log [@test131] starting deploy of {'config': True} to all [16:56:23] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [16:56:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:56:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:58:41] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.97, 3.38, 3.06 [16:58:56] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.85, 7.35, 7.03 [17:00:36] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.55, 3.03, 2.96 [17:00:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.32, 7.58, 7.13 [17:06:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.47, 7.98, 7.45 [17:12:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [17:13:03] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp25.miraheze.org - CNAME OK [17:20:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.13, 7.21, 7.18 [17:22:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.82, 6.68, 6.99 [17:28:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.80, 6.06, 6.65 [17:30:13] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 11.12, 8.82, 7.14 [17:30:26] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.57, 4.35, 3.50 [17:30:43] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.25, 10.00, 7.87 [17:32:13] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 8.71, 8.92, 7.40 [17:32:43] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.62, 9.49, 7.95 [17:36:19] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.27, 3.87, 3.60 [17:40:51] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [17:42:23] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [17:46:19] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.59, 3.18, 3.38 [17:47:17] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [17:47:47] PROBLEM - mw143 Current Load on mw143 is WARNING: WARNING - load average: 10.63, 8.08, 5.30 [17:48:08] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw131 mw134 [17:48:12] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 11.01, 10.49, 6.88 [17:48:27] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.99, 8.34, 3.81 [17:49:47] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 5.44, 7.33, 5.38 [17:50:08] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [17:50:12] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 4.83, 8.36, 6.54 [17:50:27] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.76, 5.73, 3.40 [17:51:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.33, 7.15, 6.72 [17:53:56] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.93, 6.57, 6.55 [17:57:06] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.54, 3.45, 3.37 [17:59:01] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.88, 3.35, 3.35 [18:05:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.18, 6.85, 6.69 [18:09:29] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.35, 6.69, 6.67 [18:12:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [18:15:16] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.98, 6.90, 6.79 [18:17:13] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.35, 6.40, 6.62 [18:26:54] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.23, 7.27, 6.88 [18:32:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.72, 7.38, 6.99 [18:34:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.78, 7.25, 7.00 [18:39:22] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/4dd8be64d21f [18:39:24] [02miraheze/mw-config] 07paladox 034dd8be6 - Update GlobalCache.php [18:39:27] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [18:39:42] [02mw-config] 07paladox opened pull request 03#5372: Update GlobalCache.php - 13https://github.com/miraheze/mw-config/pull/5372 [18:40:05] [02mw-config] 07paladox closed pull request 03#5372: Update GlobalCache.php - 13https://github.com/miraheze/mw-config/pull/5372 [18:40:07] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/cfce704c31f0...665a0923c810 [18:40:08] [02miraheze/mw-config] 07paladox 03665a092 - Update GlobalCache.php (#5372) [18:40:09] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [18:40:10] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [18:40:37] miraheze/mw-config - paladox the build passed. [18:41:04] miraheze/mw-config - paladox the build passed. [18:42:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.gab.pt.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:48:20] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [18:52:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.28, 7.59, 7.17 [18:52:34] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.63, 3.61, 3.28 [18:54:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.55, 7.24, 7.10 [18:56:25] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.75, 3.38, 3.28 [18:56:32] !log [@test131] starting deploy of {'config': True} to all [18:56:33] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [18:56:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:56:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:00:19] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.09, 3.87, 3.50 [19:02:19] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.25, 3.33, 3.35 [19:04:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.25, 6.50, 6.80 [19:09:44] !log [@mwtask141] starting deploy of {'config': True} to all [19:09:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:09:52] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [19:09:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:15:03] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [19:34:21] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.57, 3.17, 2.88 [19:36:21] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.08, 3.04, 2.86 [19:41:44] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=malachiteminers2wiki --no-updates --username-prefix=wikia:malachite-miners-2 /home/macfan/ImportDump/malachiteminers2wiki-20231107013348.xml (START) [19:41:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:44:49] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp24.miraheze.org - CNAME OK [19:47:28] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [19:49:45] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=malachiteminers2wiki --no-updates --username-prefix=wikia:malachite-miners-2 /home/macfan/ImportDump/malachiteminers2wiki-20231107013348.xml (END - exit=0) [19:49:46] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=malachiteminers2wiki (START) [19:49:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:51:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.26, 7.09, 6.70 [19:56:19] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.77, 3.08, 2.74 [19:56:57] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.12, 6.73, 6.67 [20:00:20] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.61, 2.98, 2.81 [20:00:32] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=malachiteminers2wiki (END - exit=0) [20:00:33] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=malachiteminers2wiki --active --update (END - exit=0) [20:00:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:00:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:25] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=ahitlastimpactwikiwiki --no-updates --username-prefix=wikia:ahit-last-impact /home/macfan/ImportDump/ahitlastimpactwikiwiki-20231107021210.xml (START) [20:03:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:44] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=ahitlastimpactwikiwiki --no-updates --username-prefix=wikia:ahit-last-impact /home/macfan/ImportDump/ahitlastimpactwikiwiki-20231107021210.xml (END - exit=0) [20:05:45] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=ahitlastimpactwikiwiki (START) [20:05:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.92, 6.67, 6.61 [20:08:36] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.61, 6.47, 6.54 [20:10:24] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=ahitlastimpactwikiwiki (END - exit=0) [20:10:25] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=ahitlastimpactwikiwiki --active --update (END - exit=0) [20:10:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:10:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.57, 7.12, 6.39 [20:13:30] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.38, 6.79, 6.36 [20:18:06] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=teslorewiki --no-updates --username-prefix=wikia:teslore /home/macfan/ImportDump/teslorewiki-20231107142455.xml (START) [20:18:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:24:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.38, 6.91, 6.64 [20:26:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.12, 6.67, 6.59 [20:39:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.28, 7.03, 6.70 [20:41:04] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/665a0923c810...26db2725daac [20:41:07] [02miraheze/mw-config] 07paladox 0326db272 - Update GlobalCache.php [20:42:07] miraheze/mw-config - paladox the build passed. [20:44:56] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.08, 6.76, 6.74 [20:48:32] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [20:55:58] !log [@test131] starting deploy of {'config': True} to all [20:55:59] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [20:56:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:56:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:09:26] !log [@mwtask141] starting deploy of {'config': True} to all [21:09:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:09:35] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [21:09:41]