[00:03:58] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 13.75, 8.61, 3.99 [00:12:46] PROBLEM - os141 Puppet on os141 is UNKNOWN: NRPE: Unable to read output [00:13:57] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 4.48, 6.87, 6.05 [00:16:24] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5002953708 secs [00:17:42] PROBLEM - os131 Puppet on os131 is UNKNOWN: NRPE: Unable to read output [00:17:57] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.28, 5.42, 5.66 [00:20:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.85, 6.75, 6.47 [00:26:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.59, 6.67, 6.56 [00:26:42] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://github.com/miraheze/MirahezeMagic/commit/07933a5be524 [00:26:45] [02miraheze/MirahezeMagic] 07paladox 0307933a5 - Create jobQueueCollector.php [00:26:47] [02MirahezeMagic] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/MirahezeMagic [00:26:48] [02MirahezeMagic] 07paladox opened pull request 03#441: Create jobQueueCollector.php - 13https://github.com/miraheze/MirahezeMagic/pull/441 [00:28:18] [02miraheze/MirahezeMagic] 07github-actions[bot] pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/07933a5be524...3214d2dc2ba4 [00:28:20] [02miraheze/MirahezeMagic] 07github-actions 033214d2d - CI: lint code to MediaWiki standards [00:28:23] [02MirahezeMagic] 07github-actions[bot] synchronize pull request 03#441: Create jobQueueCollector.php - 13https://github.com/miraheze/MirahezeMagic/pull/441 [00:29:00] [02MirahezeMagic] 07paladox synchronize pull request 03#441: Create jobQueueCollector.php - 13https://github.com/miraheze/MirahezeMagic/pull/441 [00:29:01] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/3214d2dc2ba4...589825a3859f [00:29:04] [02miraheze/MirahezeMagic] 07paladox 03589825a - Update jobQueueCollector.php [00:33:11] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.26, 3.48, 3.10 [00:38:10] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.51, 3.92, 3.06 [00:38:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.98, 6.81, 6.66 [00:38:36] miraheze/MirahezeMagic - paladox the build passed. [00:38:58] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 1.89, 3.43, 3.32 [00:40:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.47, 6.75, 6.65 [00:40:53] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.20, 3.12, 3.22 [00:42:53] [02MirahezeMagic] 07paladox synchronize pull request 03#441: Create jobQueueCollector.php - 13https://github.com/miraheze/MirahezeMagic/pull/441 [00:42:56] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/589825a3859f...f550dadc6cda [00:42:58] [02miraheze/MirahezeMagic] 07paladox 03f550dad - Update jobQueueCollector.php [00:44:17] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.67, 3.82, 3.38 [00:46:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.77, 7.17, 6.86 [00:48:15] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.05, 3.64, 3.39 [00:50:14] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.05, 3.45, 3.35 [00:50:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.38, 7.11, 6.86 [00:52:12] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.13, 3.32, 3.31 [00:52:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.92, 6.54, 6.68 [00:55:15] miraheze/MirahezeMagic - paladox the build passed. [01:12:56] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.43, 7.52, 6.90 [01:14:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.70, 7.11, 6.82 [01:18:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.64, 7.16, 6.86 [01:20:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.27, 7.39, 7.00 [01:22:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.40, 8.07, 7.28 [01:24:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.20, 7.60, 7.20 [01:26:36] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.66, 8.01, 7.39 [01:28:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.99, 7.68, 7.35 [01:32:33] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:34:23] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.09, 5.76, 6.64 [01:51:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.42, 7.01, 6.45 [01:53:30] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.83, 6.67, 6.40 [02:00:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.31, 6.75, 6.60 [02:02:28] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.08, 6.51, 6.54 [02:05:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.90, 7.12, 6.60 [02:06:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.54, 6.84, 6.71 [02:07:30] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.60, 6.57, 6.47 [02:08:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.97, 6.47, 6.58 [02:16:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.77, 7.03, 6.73 [02:22:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.19, 6.77, 6.73 [02:26:34] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 3.51, 5.78, 3.17 [02:30:34] RECOVERY - mon141 Current Load on mon141 is OK: LOAD OK - total load average: 0.82, 3.08, 2.66 [02:30:37] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [02:50:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.31, 6.64, 6.36 [02:52:26] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.06, 6.36, 6.29 [02:56:17] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.99, 6.75, 6.47 [02:58:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.72, 6.51, 6.42 [03:01:16] RECOVERY - db101 Backups SQL on db101 is OK: FILE_AGE OK: /var/log/sql-backup.log is 75 seconds old and 0 bytes [03:20:01] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.01, 6.90, 6.09 [03:20:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 8.00, 7.19, 6.61 [03:21:55] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.78, 6.49, 6.04 [03:22:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.06, 6.31, 6.35 [06:10:35] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 5.03, 6.41, 3.29 [06:12:23] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [06:14:34] RECOVERY - mon141 Current Load on mon141 is OK: LOAD OK - total load average: 0.81, 3.37, 2.75 [06:30:24] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.46, 6.54, 5.90 [06:32:18] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.18, 6.11, 5.82 [06:42:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.gab.pt.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:46:04] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [07:18:04] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [07:35:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.85, 6.44, 5.97 [07:38:55] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.77, 6.18, 5.97 [08:25:02] PROBLEM - cloud14 Puppet on cloud14 is UNKNOWN: NRPE: Unable to read output [08:32:32] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [08:46:47] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [08:53:02] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:18:16] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [09:20:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.18, 6.69, 6.16 [09:22:56] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.20, 6.15, 6.03 [09:30:38] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [09:49:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.88, 6.03, 5.68 [09:51:50] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.47, 5.82, 5.65 [11:51:47] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.47, 6.69, 6.12 [11:53:44] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.14, 6.43, 6.09 [12:01:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.69, 6.57, 6.15 [12:03:26] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.26, 6.37, 6.12 [12:16:04] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [12:46:57] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.52, 3.86, 2.46 [12:47:29] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:48:04] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [12:48:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/5e6af8b11589 [12:48:18] [02miraheze/puppet] 07paladox 035e6af8b - prometheus-gdnsd-stats: drop --url param [12:48:21] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [12:48:23] [02puppet] 07paladox opened pull request 03#3498: prometheus-gdnsd-stats: drop --url param - 13https://github.com/miraheze/puppet/pull/3498 [12:48:58] [02puppet] 07paladox closed pull request 03#3498: prometheus-gdnsd-stats: drop --url param - 13https://github.com/miraheze/puppet/pull/3498 [12:48:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e0be7fdd49f7...2f5a3a955c4a [12:49:00] [02miraheze/puppet] 07paladox 032f5a3a9 - prometheus-gdnsd-stats: drop --url param (#3498) [12:49:01] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [12:49:02] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [12:49:32] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 6.296 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [12:50:53] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.74, 3.45, 2.68 [12:52:52] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.69, 2.51, 2.43 [13:34:21] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.43, 3.19, 2.87 [13:36:19] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.37, 3.32, 2.96 [13:42:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [13:43:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.33, 6.53, 6.33 [13:45:03] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.61, 6.80, 6.47 [14:04:20] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.72, 3.42, 3.04 [14:06:19] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.26, 2.93, 2.90 [14:12:23] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [14:18:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/a436470d2da3 [14:18:14] [02miraheze/puppet] 07paladox 03a436470 - Add a prometheus-jobqueue-stats script [14:18:15] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [14:18:18] [02puppet] 07paladox opened pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [14:21:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a436470d2da3...ab8a6a278f6f [14:21:23] [02miraheze/puppet] 07paladox 03ab8a6a2 - Update redis.pp [14:21:26] [02puppet] 07paladox synchronize pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [14:21:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ab8a6a278f6f...be1fcbb5f89b [14:21:44] [02miraheze/puppet] 07paladox 03be1fcbb - Update prometheus-redis-exporter.systemd.erb [14:21:47] [02puppet] 07paladox synchronize pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [14:22:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/be1fcbb5f89b...cf5d0779d685 [14:22:03] [02miraheze/puppet] 07paladox 03cf5d077 - Delete modules/prometheus/files/redis/jobQueueCollector.lua [14:22:05] [02puppet] 07paladox synchronize pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [14:22:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cf5d0779d685...cb0f1c83f971 [14:22:50] [02miraheze/puppet] 07paladox 03cb0f1c8 - Update jobchron121.yaml [14:22:51] [02puppet] 07paladox synchronize pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [14:23:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cb0f1c83f971...47232aba5811 [14:23:03] [02miraheze/puppet] 07paladox 0347232ab - Update test131.yaml [14:23:04] [02puppet] 07paladox synchronize pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [14:25:09] [02puppet] 07paladox synchronize pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [14:25:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/47232aba5811...80d2a8f87550 [14:25:12] [02miraheze/puppet] 07paladox 0380d2a8f - Update redis.pp [14:25:22] [02puppet] 07paladox synchronize pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [14:25:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/80d2a8f87550...f1ac2362c345 [14:25:25] [02miraheze/puppet] 07paladox 03f1ac236 - Update jobchron121.yaml [14:26:11] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:26:42] [02puppet] 07paladox synchronize pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [14:26:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f1ac2362c345...f46cb8b9f1ba [14:26:47] [02miraheze/puppet] 07paladox 03f46cb8b - Update redis.pp [14:27:32] [02puppet] 07paladox synchronize pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [14:27:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f46cb8b9f1ba...cb8e8dfb87f7 [14:27:36] [02miraheze/puppet] 07paladox 03cb8e8df - Update redis.pp [14:28:12] [02puppet] 07paladox synchronize pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [14:28:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cb8e8dfb87f7...edc599ad0e8e [14:28:15] [02miraheze/puppet] 07paladox 03edc599a - Update redis.pp [14:47:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.69, 7.10, 6.48 [14:48:00] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.04, 7.17, 6.43 [14:49:03] PROBLEM - jobchron121 Puppet on jobchron121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:49:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.93, 8.06, 6.90 [14:51:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.77, 7.58, 6.87 [14:51:48] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.50, 7.70, 6.88 [14:53:03] [02puppet] 07paladox synchronize pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [14:53:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+1/-1/±0] 13https://github.com/miraheze/puppet/compare/edc599ad0e8e...4b34e5ed9b13 [14:53:08] [02miraheze/puppet] 07paladox 034b34e5e - Update and rename prometheus-jobqueue-stats.py to prometheus-jobqueue-stats.py.erb [14:53:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.27, 7.83, 7.01 [14:55:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.43, 7.19, 6.88 [14:57:23] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.97, 6.65, 6.71 [15:03:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.83, 7.96, 7.18 [15:03:31] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.93, 7.71, 7.01 [15:05:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.87, 6.98, 6.91 [15:05:31] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.93, 6.66, 6.72 [15:07:43] PROBLEM - matomo121 Puppet on matomo121 is UNKNOWN: NRPE: Unable to read output [15:09:06] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.12, 6.36, 6.69 [15:16:04] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [15:17:54] PROBLEM - jobchron121 Puppet on jobchron121 is UNKNOWN: NRPE: Unable to read output [15:26:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.22, 7.95, 7.18 [15:27:34] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.04, 7.18, 6.59 [15:28:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.74, 7.65, 7.16 [15:29:30] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.58, 6.30, 6.34 [15:30:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.26, 7.79, 7.26 [15:32:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.35, 7.38, 7.18 [15:40:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.01, 6.05, 6.68 [15:42:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [15:45:16] [02puppet] 07paladox synchronize pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [15:45:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4b34e5ed9b13...1612ed900cb0 [15:45:22] [02miraheze/puppet] 07paladox 031612ed9 - Update redis.pp [15:45:50] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [15:46:14] [02puppet] 07paladox synchronize pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [15:46:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1612ed900cb0...aa8ef8ac9c64 [15:46:18] [02miraheze/puppet] 07paladox 03aa8ef8a - Update prometheus-jobqueue-stats.py.erb [15:48:04] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:48:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/aa8ef8ac9c64...992ee73b09ea [15:48:08] [02miraheze/puppet] 07paladox 03992ee73 - Update prometheus.pp [15:48:10] [02puppet] 07paladox synchronize pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [15:50:02] [02puppet] 07paladox synchronize pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [15:50:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/992ee73b09ea...ff35fa86d7f7 [15:50:04] [02miraheze/puppet] 07paladox 03ff35fa8 - Update redis.pp [15:50:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.76, 6.80, 6.62 [15:52:03] [02puppet] 07paladox synchronize pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [15:52:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ff35fa86d7f7...ffa1fa2940c5 [15:52:07] [02miraheze/puppet] 07paladox 03ffa1fa2 - Update prometheus.pp [15:52:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.12, 6.69, 6.61 [15:56:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ffa1fa2940c5...69f018e99e35 [15:56:09] [02miraheze/puppet] 07paladox 0369f018e - Update test131.yaml [15:56:10] [02puppet] 07paladox synchronize pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [15:56:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.56, 7.28, 6.88 [15:58:11] PROBLEM - test131 Puppet on test131 is UNKNOWN: NRPE: Unable to read output [15:58:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.92, 7.33, 6.96 [16:01:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/69f018e99e35...53afecbfab16 [16:01:33] [02miraheze/puppet] 07paladox 0353afecb - Update redis.pp [16:01:36] [02puppet] 07paladox synchronize pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [16:02:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.86, 6.64, 6.78 [16:07:43] RECOVERY - matomo121 Puppet on matomo121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:08:37] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.86, 3.28, 2.22 [16:10:38] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.75, 3.06, 2.27 [16:12:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [16:18:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.89, 6.47, 6.46 [16:24:31] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.75, 3.05, 2.79 [16:26:38] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.97, 6.60, 6.58 [16:28:22] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.09, 3.13, 2.87 [16:34:01] [02puppet] 07paladox closed pull request 03#3499: Add a prometheus-jobqueue-stats script - 13https://github.com/miraheze/puppet/pull/3499 [16:34:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±4] 13https://github.com/miraheze/puppet/compare/2f5a3a955c4a...b92579b9d3cd [16:34:04] [02miraheze/puppet] 07paladox 03b92579b - Add a prometheus-jobqueue-stats script (#3499) [16:34:06] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [16:34:08] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [16:40:20] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.62, 3.17, 2.95 [16:40:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:42:19] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.91, 3.09, 2.95 [16:50:56] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.00, 6.94, 6.61 [16:52:53] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.56, 6.35, 6.43 [17:16:46] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [17:21:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.38, 7.15, 6.49 [17:23:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.46, 6.94, 6.50 [17:26:56] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.80, 6.73, 6.55 [17:27:45] PROBLEM - os141 Puppet on os141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:30:00] PROBLEM - os141 Puppet on os141 is UNKNOWN: NRPE: Unable to read output [17:30:12] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 13.39, 9.55, 6.39 [17:30:28] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 13.41, 9.35, 4.18 [17:30:34] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 5.27, 3.51, 1.91 [17:30:41] PROBLEM - bast141 Current Load on bast141 is CRITICAL: LOAD CRITICAL - total load average: 3.04, 1.85, 0.74 [17:31:10] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.50, 5.45, 2.84 [17:32:12] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 4.76, 7.49, 6.01 [17:32:28] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.08, 6.35, 3.70 [17:32:34] RECOVERY - mon141 Current Load on mon141 is OK: LOAD OK - total load average: 1.13, 2.50, 1.74 [17:32:41] RECOVERY - bast141 Current Load on bast141 is OK: LOAD OK - total load average: 0.41, 1.23, 0.64 [17:33:10] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 0.60, 3.64, 2.49 [17:35:10] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.29, 2.48, 2.20 [17:37:59] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.67, 3.55, 3.12 [17:39:54] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.78, 3.25, 3.06 [17:41:47] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:42:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [17:43:16] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.96, 4.66, 3.15 [17:43:46] PROBLEM - mwtask141 Puppet on mwtask141 is UNKNOWN: NRPE: Unable to read output [17:44:47] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 6.85, 11.68, 7.06 [17:45:14] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.94, 3.18, 2.79 [17:45:35] [Grafana] !sre RESOLVED: MediaWiki Exception Rate https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [17:48:16] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [17:48:41] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 0.46, 5.69, 5.64 [17:51:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.96, 7.66, 6.93 [17:56:58] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.78, 6.64, 6.69 [18:04:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b92579b9d3cd...ba8c50a608df [18:04:34] [02miraheze/puppet] 07paladox 03ba8c50a - jobrunner-hi: increase html runners to 2 [18:07:38] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [18:08:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ba8c50a608df...b3d23a8dfb14 [18:08:44] [02miraheze/puppet] 07paladox 03b3d23a8 - jobrunner-hi: Split html group [18:12:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [18:14:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b3d23a8dfb14...e80e938b1b57 [18:14:12] [02miraheze/puppet] 07paladox 03e80e938 - Update jobrunner-hi.json.erb [18:17:50] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [18:18:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.61, 7.02, 6.70 [18:18:39] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ce14119f6350...c9bdcdcfd8d3 [18:18:42] [02miraheze/mw-config] 07paladox 03c9bdcdc - Enable warmup for wikis beginning with 0-9 and a [18:19:02] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [18:19:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:19:10] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [18:19:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:19:16] !log [paladox@mwtask141] starting deploy of {'config': True} to all [18:19:22] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 6s [18:19:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:19:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:19:43] miraheze/mw-config - paladox the build passed. [18:22:35] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 10.44, 8.18, 5.83 [18:26:16] !log [@test131] starting deploy of {'config': True} to all [18:26:17] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [18:26:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:32:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.42, 7.10, 6.95 [18:32:35] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:32:51] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.52, 3.18, 2.96 [18:34:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.09, 6.66, 6.81 [18:34:46] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.27, 2.87, 2.88 [18:36:58] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c9bdcdcfd8d3...7169f9fafa31 [18:37:00] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [18:37:00] [02miraheze/mw-config] 07paladox 037169f9f - Enable warmup for wikis beginign with b and c [18:37:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:37:07] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [18:37:09] !log [paladox@mwtask141] starting deploy of {'config': True} to all [18:37:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:37:15] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 5s [18:37:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:37:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:38:02] miraheze/mw-config - paladox the build passed. [18:42:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.56, 6.65, 6.79 [18:50:20] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.79, 3.59, 3.25 [18:52:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.12, 7.48, 7.01 [18:54:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.21, 6.99, 6.89 [18:56:22] !log [@test131] starting deploy of {'config': True} to all [18:56:23] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [18:56:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:56:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:58:19] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.69, 3.92, 3.54 [18:59:34] PROBLEM - test131 APT on test131 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (9 critical updates). [19:00:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.11, 6.96, 6.84 [19:00:19] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.97, 3.60, 3.47 [19:01:37] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [19:02:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.34, 6.64, 6.73 [19:06:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.02, 6.94, 6.79 [19:06:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.89, 7.22, 6.53 [19:08:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.39, 6.80, 6.76 [19:12:13] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.79, 6.75, 6.52 [19:12:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.57, 6.85, 6.82 [19:12:19] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.73, 3.08, 3.32 [19:14:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.33, 6.52, 6.71 [19:18:54] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:20:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.85, 6.57, 6.62 [19:22:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.43, 6.57, 6.62 [19:26:01] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7169f9fafa31...ddf30a57a900 [19:26:03] [02miraheze/mw-config] 07paladox 03ddf30a5 - Enable warmup for wikis beginning with d and e [19:26:03] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [19:26:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:13] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [19:26:15] !log [paladox@mwtask141] starting deploy of {'config': True} to all [19:26:16] !log [@test131] starting deploy of {'config': True} to all [19:26:17] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [19:26:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:22] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [19:26:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:02] miraheze/mw-config - paladox the build passed. [19:27:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.68, 6.87, 6.33 [19:28:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.08, 7.00, 6.81 [19:30:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.78, 6.68, 6.72 [19:32:36] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [19:33:30] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.79, 6.62, 6.44 [19:35:30] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.27, 3.52, 3.30 [19:37:25] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.71, 3.20, 3.21 [19:37:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.47, 7.28, 6.74 [19:43:30] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.79, 6.49, 6.56 [19:45:00] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.25, 7.16, 6.88 [19:46:56] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.32, 6.77, 6.76 [19:49:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.26, 7.06, 6.78 [19:53:30] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.22, 6.28, 6.55 [20:04:50] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ddf30a57a900...bac1dd7d4cd7 [20:04:52] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [20:04:52] [02miraheze/mw-config] 07paladox 03bac1dd7 - Enable warmup on wikis begining with f, g, h, I and j [20:04:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:00] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [20:05:02] !log [paladox@mwtask141] starting deploy of {'config': True} to all [20:05:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:08] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 6s [20:05:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:21] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.68, 3.76, 3.35 [20:05:50] miraheze/mw-config - paladox the build passed. [20:09:11] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.28, 3.19, 3.22 [20:20:54] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.10, 7.00, 6.48 [20:22:50] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.42, 6.61, 6.40 [20:26:40] !log [@test131] starting deploy of {'config': True} to all [20:26:41] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [20:26:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:26:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:30:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 6.77, 7.53, 7.95 [20:30:41] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:32:55] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/bac1dd7d4cd7...1a237ebc2306 [20:32:58] [02miraheze/mw-config] 07paladox 031a237eb - Remove WikiLambdaFunctionCall from PoolCounter [20:33:53] miraheze/mw-config - paladox the build passed. [20:34:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.19, 7.09, 6.71 [20:36:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 8.03, 7.62, 7.83 [20:38:19] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.29, 6.62, 6.59 [20:38:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 6.72, 7.26, 7.67 [20:39:39] !log [@mwtask141] starting deploy of {'config': True} to all [20:39:47] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [20:39:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:39:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:39:59] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1a237ebc2306...30e0fbe85e2c [20:40:00] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [20:40:02] [02miraheze/mw-config] 07paladox 0330e0fbe - Enable warmup on wikis beginning with k, l, n, m and o [20:40:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:40:07] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 6s [20:40:10] !log [paladox@mwtask141] starting deploy of {'config': True} to all [20:40:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:40:16] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 5s [20:40:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:40:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:40:55] miraheze/mw-config - paladox the build passed. [20:42:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.46, 7.07, 6.79 [20:46:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 8.20, 7.59, 7.63 [20:48:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.47, 6.76, 6.74 [20:54:54] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [20:56:09] !log [@test131] starting deploy of {'config': True} to all [20:56:10] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [20:56:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:56:19]