[00:06:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f82fd165475c...7f4aaf252bbe [00:06:02] [02miraheze/puppet] 07paladox 037f4aaf2 - php: decrease pm.max_requests to 5k [00:07:36] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 12.33, 8.10, 4.57 [00:08:14] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.08015853167 secs [00:19:28] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.41, 3.34, 3.96 [00:25:24] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.09, 2.36, 3.31 [00:58:56] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1757440865 secs [01:06:53] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.03, 6.28, 3.86 [01:12:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.11, 3.71, 3.55 [01:14:47] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.66, 2.99, 3.30 [01:20:56] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.04962742329 secs [01:30:36] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.17, 3.49, 3.16 [01:32:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.83, 3.06, 3.05 [01:36:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.63, 3.92, 3.46 [01:40:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.58, 4.83, 3.89 [01:52:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.01, 3.29, 3.98 [01:56:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.73, 2.02, 3.30 [02:04:56] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1572673321 secs [02:08:51] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [02:10:56] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.03193962574 secs [02:11:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.60, 5.45, 4.04 [02:15:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.17, 3.07, 3.38 [03:03:45] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.68, 3.62, 2.23 [03:05:45] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.96, 2.87, 2.12 [03:06:03] PROBLEM - wiki.pixlies.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.pixlies.net could not be found [03:08:05] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:10:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.92, 4.87, 3.30 [03:12:50] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.68, 3.51, 2.99 [03:14:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.86, 3.03, 2.88 [03:18:56] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.4338144362 secs [03:25:55] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7095 MB (5% inode=99%); [03:27:55] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 8350 MB (6% inode=99%); [03:36:56] PROBLEM - cp32 NTP time on cp32 is CRITICAL: NTP CRITICAL: Offset 0.5910427272 secs [03:37:42] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [03:38:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 13.15, 6.85, 3.90 [03:41:44] PROBLEM - db142 Backups SQL on db142 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864239 seconds old and 1621 bytes [03:42:56] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.004704505205 secs [03:46:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.25, 3.18, 3.35 [04:08:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.11, 4.23, 2.96 [04:10:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.16, 3.28, 2.77 [04:11:03] PROBLEM - thenintendowiki.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query thenintendowiki.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:20:56] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1411771476 secs [04:38:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.43, 2.89, 2.17 [04:40:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.98, 4.72, 2.95 [04:44:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.88, 3.05, 2.71 [04:52:56] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.09964463115 secs [05:09:03] RECOVERY - thenintendowiki.com - reverse DNS on sslhost is OK: SSL OK - thenintendowiki.com reverse DNS resolves to cp23.miraheze.org - NS RECORDS OK [05:09:47] PROBLEM - phytinfo.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'phytinfo.cf' expires in 7 day(s) (Tue 25 Jul 2023 05:09:23 GMT +0000). [05:14:36] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.81, 4.28, 3.16 [05:18:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.93, 2.91, 2.91 [05:19:55] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7969 MB (5% inode=99%); [05:42:14] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1376310587 secs [05:42:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.77, 3.70, 2.76 [05:44:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.99, 3.09, 2.65 [05:50:15] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.08381345868 secs [06:12:34] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.98, 3.21, 2.44 [06:14:34] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.93, 3.06, 2.48 [06:25:59] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:37:08] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [06:37:17] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.94, 3.37, 2.28 [06:39:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.12, 3.41, 2.44 [06:40:39] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1293943524 secs [06:41:15] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.52, 4.55, 2.97 [06:43:13] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.92, 3.53, 2.80 [06:45:12] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.02, 2.80, 2.62 [06:46:56] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1811234355 secs [06:55:57] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [07:02:39] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.07521292567 secs [07:08:55] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.51, 4.19, 2.69 [07:10:53] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.36, 3.09, 2.47 [07:10:56] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.04683437943 secs [08:41:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.58, 4.60, 2.64 [08:43:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.38, 3.97, 2.64 [08:47:07] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.34, 2.73, 2.44 [09:43:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.61, 3.64, 2.59 [09:45:29] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.82, 2.65, 2.35 [09:57:43] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:14:08] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.18, 4.82, 3.02 [10:18:05] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.03, 3.88, 3.04 [10:20:04] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.11, 2.84, 2.75 [10:27:43] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [11:08:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.84, 3.28, 2.03 [11:11:53] PROBLEM - cp32 PowerDNS Recursor on cp32 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:13:20] [02miraheze/SpriteSheet] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/SpriteSheet/compare/adce4c6874d9...c71db674e4db [11:13:22] [02miraheze/SpriteSheet] 07translatewiki 03c71db67 - Localisation updates from https://translatewiki.net. [11:13:50] RECOVERY - cp32 PowerDNS Recursor on cp32 is OK: DNS OK: 0.551 seconds response time. miraheze.org returns 2607:f1c0:1800:26f::1,74.208.203.152 [11:19:28] miraheze/SpriteSheet - translatewiki the build passed. [11:20:10] PROBLEM - cp32 SSH on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:20:27] PROBLEM - cp32 PowerDNS Recursor on cp32 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:20:36] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 108.175.15.182/cpweb, 2607:f1c0:1800:8100::1/cpweb [11:22:10] RECOVERY - cp32 SSH on cp32 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:22:25] RECOVERY - cp32 PowerDNS Recursor on cp32 is OK: DNS OK: 1.712 second response time. miraheze.org returns 108.175.15.182,2607:f1c0:1800:26f::1,2607:f1c0:1800:8100::1,74.208.203.152 [11:22:36] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:48:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.68, 1.02, 3.67 [11:50:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.65, 0.92, 3.31 [11:55:51] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 8.56, 12.62, 8.11 [11:57:51] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 4.77, 9.66, 7.55 [12:12:34] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.27, 3.64, 2.73 [12:12:39] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.75, 5.06, 3.40 [12:14:10] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.77, 11.27, 7.08 [12:14:34] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.37, 3.59, 2.83 [12:16:06] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 6.09, 9.18, 6.80 [12:16:36] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.56, 3.87, 3.27 [12:18:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.41, 2.91, 2.98 [12:24:34] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.30, 3.40, 3.12 [12:30:34] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.82, 3.97, 3.42 [12:32:34] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.52, 3.82, 3.43 [12:40:36] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 10.14, 5.61, 3.41 [12:46:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.16, 3.65, 3.50 [12:48:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.17, 3.79, 3.56 [12:50:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.58, 2.88, 3.25 [12:54:34] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.65, 3.14, 3.36 [13:06:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.83, 3.77, 2.79 [13:08:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.26, 3.10, 2.67 [13:14:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.25, 6.34, 4.31 [13:18:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.81, 3.30, 3.52 [13:20:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.65, 2.44, 3.18 [13:24:56] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1767242253 secs [13:40:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.26, 4.37, 3.21 [13:42:59] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.05918374658 secs [13:45:11] [02miraheze/ImportDump] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.16.0 [+0/-0/±2] 13https://github.com/miraheze/ImportDump/commit/2cd6484a4f0a [13:45:12] [02miraheze/ImportDump] 07dependabot[bot] 032cd6484 - Bump stylelint-config-wikimedia from 0.15.0 to 0.16.0 [13:45:15] [02ImportDump] 07dependabot[bot] opened pull request 03#47: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.0 - 13https://github.com/miraheze/ImportDump/pull/47 [13:45:16] [02ImportDump] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.16.0 - 13https://github.com/miraheze/ImportDump [13:45:19] [02ImportDump] 07dependabot[bot] labeled pull request 03#47: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.0 - 13https://github.com/miraheze/ImportDump/pull/47 [13:45:22] [02ImportDump] 07dependabot[bot] labeled pull request 03#47: Bump stylelint-config-wikimedia from 0.15.0 to 0.16.0 - 13https://github.com/miraheze/ImportDump/pull/47 [13:55:48] miraheze/ImportDump - dependabot[bot] the build has errored. [13:56:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.53, 1.79, 3.71 [13:56:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/a23cd3aca75f [13:56:48] [02miraheze/puppet] 07paladox 03a23cd3a - mediawiki: lower php-fpm childs to 24 [13:56:51] [02puppet] 07paladox created branch 03paladox-patch-7 - 13https://github.com/miraheze/puppet [13:56:52] [02puppet] 07paladox opened pull request 03#3298: mediawiki: lower php-fpm childs to 24 - 13https://github.com/miraheze/puppet/pull/3298 [13:57:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a23cd3aca75f...01ed283f3f49 [13:57:57] [02miraheze/puppet] 07paladox 0301ed283 - Update mw132.yaml [13:57:58] [02puppet] 07paladox synchronize pull request 03#3298: mediawiki: lower php-fpm childs to 24 - 13https://github.com/miraheze/puppet/pull/3298 [13:58:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/01ed283f3f49...4853d32057ba [13:58:06] [02miraheze/puppet] 07paladox 034853d32 - Update mw133.yaml [13:58:07] [02puppet] 07paladox synchronize pull request 03#3298: mediawiki: lower php-fpm childs to 24 - 13https://github.com/miraheze/puppet/pull/3298 [13:58:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4853d32057ba...e1fbd66c5061 [13:58:13] [02miraheze/puppet] 07paladox 03e1fbd66 - Update mw134.yaml [13:58:15] [02puppet] 07paladox synchronize pull request 03#3298: mediawiki: lower php-fpm childs to 24 - 13https://github.com/miraheze/puppet/pull/3298 [13:58:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e1fbd66c5061...1c4190220694 [13:58:20] [02miraheze/puppet] 07paladox 031c41902 - Update mw141.yaml [13:58:21] [02puppet] 07paladox synchronize pull request 03#3298: mediawiki: lower php-fpm childs to 24 - 13https://github.com/miraheze/puppet/pull/3298 [13:58:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1c4190220694...90ee48e0879c [13:58:26] [02miraheze/puppet] 07paladox 0390ee48e - Update mw142.yaml [13:58:28] [02puppet] 07paladox synchronize pull request 03#3298: mediawiki: lower php-fpm childs to 24 - 13https://github.com/miraheze/puppet/pull/3298 [13:58:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/90ee48e0879c...4ddc4c824ff3 [13:58:33] [02miraheze/puppet] 07paladox 034ddc4c8 - Update mw143.yaml [13:58:34] [02puppet] 07paladox synchronize pull request 03#3298: mediawiki: lower php-fpm childs to 24 - 13https://github.com/miraheze/puppet/pull/3298 [13:58:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.77, 1.45, 3.34 [13:58:48] [02puppet] 07paladox closed pull request 03#3298: mediawiki: lower php-fpm childs to 24 - 13https://github.com/miraheze/puppet/pull/3298 [13:58:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±7] 13https://github.com/miraheze/puppet/compare/7f4aaf252bbe...599afd96c124 [13:58:53] [02miraheze/puppet] 07paladox 03599afd9 - mediawiki: lower php-fpm childs to 24 (#3298) [13:58:56] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-7 [13:58:57] [02puppet] 07paladox deleted branch 03paladox-patch-7 - 13https://github.com/miraheze/puppet [14:03:12] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [14:03:23] PROBLEM - mw143 APT on mw143 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [14:04:05] PROBLEM - swiftobject111 APT on swiftobject111 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [14:05:23] PROBLEM - mw143 APT on mw143 is CRITICAL: APT CRITICAL: 13 packages available for upgrade (13 critical updates). [14:06:05] PROBLEM - swiftobject111 APT on swiftobject111 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (6 critical updates). [14:06:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.58, 3.04, 3.17 [14:07:11] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [14:08:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.07, 3.05, 3.16 [14:10:57] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [14:24:29] PROBLEM - www.arru.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.arru.xyz reverse DNS resolves to webredir.gandi.net [14:36:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.20, 3.86, 2.59 [14:39:01] RECOVERY - www.arru.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'www.arru.xyz' will expire on Thu 27 Jun 2024 23:59:59 GMT +0000. [14:40:57] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [14:47:18] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.127176106 secs [14:48:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.79, 3.23, 3.38 [15:06:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.81, 2.75, 2.46 [15:08:27] PROBLEM - studyabroad.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'studyabroad.wiki' expires in 7 day(s) (Tue 25 Jul 2023 14:49:19 GMT +0000). [15:08:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.41, 2.91, 2.58 [15:17:18] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07755631208 secs [15:27:32] PROBLEM - mcwiki.kirbygang.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'mcwiki.kirbygang.tk' expires in 7 day(s) (Tue 25 Jul 2023 15:10:40 GMT +0000). [15:27:40] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/d30e2cde153c [15:27:42] [02miraheze/mw-config] 07paladox 03d30e2cd - Set locale to en-GB [15:27:43] [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/mw-config [15:27:45] [02mw-config] 07paladox opened pull request 03#5286: Set locale to en-GB - 13https://github.com/miraheze/mw-config/pull/5286 [15:28:43] miraheze/mw-config - paladox the build passed. [15:28:45] [02mw-config] 07paladox closed pull request 03#5286: Set locale to en-GB - 13https://github.com/miraheze/mw-config/pull/5286 [15:28:48] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e05a6aed148d...79bafd94bb2b [15:28:49] [02miraheze/mw-config] 07paladox 0379bafd9 - Set locale to en-GB (#5286) [15:28:52] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3 [15:28:54] [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/mw-config [15:29:44] miraheze/mw-config - paladox the build passed. [15:36:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.32, 4.11, 2.66 [15:39:13] [02mw-config] 07paladox synchronize pull request 03#5284: Change wgSVGConverter to use rsvg - 13https://github.com/miraheze/mw-config/pull/5284 [15:39:14] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0c30f6873942...4bfcc627f945 [15:39:17] [02miraheze/mw-config] 07paladox 034bfcc62 - Update LocalSettings.php [15:40:18] miraheze/mw-config - paladox the build passed. [15:40:27] !log [@mwtask141] starting deploy of {'config': True} to all [15:40:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:40:37] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 9s [15:40:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:46:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.04, 3.19, 3.18 [15:57:14] !log [@test131] starting deploy of {'config': True} to all [15:57:15] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 1s [15:57:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:57:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:07:20] [02miraheze/puppet] 07paladox pushed 0317 commits to 03paladox-patch-4 [+0/-0/±51] 13https://github.com/miraheze/puppet/compare/d41fa96f8fbe...974265ecd6d0 [16:07:22] [02miraheze/puppet] 07paladox 03974265e - Merge branch 'master' into paladox-patch-4 [16:07:23] [02puppet] 07paladox synchronize pull request 03#3286: Mediawiki::jobrunenr: Add upload runner - 13https://github.com/miraheze/puppet/pull/3286 [16:08:09] [02puppet] 07paladox closed pull request 03#3286: Mediawiki::jobrunenr: Add upload runner - 13https://github.com/miraheze/puppet/pull/3286 [16:08:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/599afd96c124...c3b41e607044 [16:08:13] [02miraheze/puppet] 07paladox 03c3b41e6 - Mediawiki::jobrunenr: Add upload runner (#3286) [16:08:16] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [16:08:19] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [16:08:43] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:15:40] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.59, 3.11, 2.35 [16:19:40] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.67, 3.02, 2.51 [16:37:13] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.69, 3.39, 2.38 [16:38:20] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [16:45:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.51, 3.48, 2.98 [16:47:07] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.41, 3.38, 3.02 [16:49:59] [02miraheze/MatomoAnalytics] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.16.0 [+0/-0/±2] 13https://github.com/miraheze/MatomoAnalytics/commit/2955a26bf413 [16:50:00] [02miraheze/MatomoAnalytics] 07dependabot[bot] 032955a26 - Bump stylelint-config-wikimedia from 0.14.0 to 0.16.0 [16:50:02] [02MatomoAnalytics] 07dependabot[bot] opened pull request 03#104: Bump stylelint-config-wikimedia from 0.14.0 to 0.16.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/104 [16:50:05] [02MatomoAnalytics] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.16.0 - 13https://github.com/miraheze/MatomoAnalytics [16:50:08] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#104: Bump stylelint-config-wikimedia from 0.14.0 to 0.16.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/104 [16:50:09] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#104: Bump stylelint-config-wikimedia from 0.14.0 to 0.16.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/104 [16:50:12] [02MatomoAnalytics] 07dependabot[bot] commented on pull request 03#102: Bump stylelint-config-wikimedia from 0.14.0 to 0.15.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/102#issuecomment-1638511450 [16:50:15] [02MatomoAnalytics] 07dependabot[bot] closed pull request 03#102: Bump stylelint-config-wikimedia from 0.14.0 to 0.15.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/102 [16:50:18] [02miraheze/MatomoAnalytics] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.15.0 [16:50:21] [02MatomoAnalytics] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.15.0 - 13https://github.com/miraheze/MatomoAnalytics [16:54:21] miraheze/MatomoAnalytics - dependabot[bot] the build has errored. [16:54:30] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/0d82422ea3ab [16:54:32] [02miraheze/mw-config] 07paladox 030d82422 - Throttle htmlCacheUpdate jobs to 50/s [16:54:34] [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/mw-config [16:54:37] [02mw-config] 07paladox opened pull request 03#5287: Throttle htmlCacheUpdate jobs to 50/s - 13https://github.com/miraheze/mw-config/pull/5287 [16:54:41] [02mw-config] 07paladox closed pull request 03#5287: Throttle htmlCacheUpdate jobs to 50/s - 13https://github.com/miraheze/mw-config/pull/5287 [16:54:43] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/79bafd94bb2b...3dabd2e97082 [16:54:46] [02miraheze/mw-config] 07paladox 033dabd2e - Throttle htmlCacheUpdate jobs to 50/s (#5287) [16:54:48] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3 [16:54:51] [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/mw-config [16:55:31] miraheze/mw-config - paladox the build passed. [16:55:43] miraheze/mw-config - paladox the build passed. [16:56:58] !log [@test131] starting deploy of {'config': True} to all [16:56:59] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [16:57:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:57:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:57:22] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [16:57:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:58:05] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 43s [16:58:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:38:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.52, 3.90, 3.02 [17:40:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.97, 3.78, 3.10 [17:42:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.31, 3.84, 3.19 [17:44:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.78, 3.65, 3.21 [17:45:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/3afde842c7c8 [17:45:56] [02miraheze/puppet] 07paladox 033afde84 - mediawiki: Add mediawiki-firejail-rsvg-convert script [17:45:57] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [17:46:00] [02puppet] 07paladox opened pull request 03#3299: mediawiki: Add mediawiki-firejail-rsvg-convert script - 13https://github.com/miraheze/puppet/pull/3299 [17:46:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.24, 2.74, 2.92 [17:47:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/3afde842c7c8...ffbbff41c740 [17:47:03] [02miraheze/puppet] 07paladox 03ffbbff4 - Create mediawiki-firejail-rsvg-convert.py [17:47:05] [02puppet] 07paladox synchronize pull request 03#3299: mediawiki: Add mediawiki-firejail-rsvg-convert script - 13https://github.com/miraheze/puppet/pull/3299 [17:49:34] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4bfcc627f945...12815368cda5 [17:49:37] [02miraheze/mw-config] 07paladox 031281536 - Update GlobalSettings.php [17:49:38] [02mw-config] 07paladox synchronize pull request 03#5284: Change wgSVGConverter to use rsvg - 13https://github.com/miraheze/mw-config/pull/5284 [17:50:07] [02miraheze/mw-config] 07paladox pushed 035 commits to 03paladox-patch-1 [+0/-0/±6] 13https://github.com/miraheze/mw-config/compare/12815368cda5...b58632fcd85f [17:50:08] [02miraheze/mw-config] 07paladox 03b58632f - Merge branch 'master' into paladox-patch-1 [17:50:09] [02mw-config] 07paladox synchronize pull request 03#5284: Change wgSVGConverter to use rsvg - 13https://github.com/miraheze/mw-config/pull/5284 [17:50:24] [02mw-config] 07paladox synchronize pull request 03#5284: Change wgSVGConverter to use rsvg - 13https://github.com/miraheze/mw-config/pull/5284 [17:50:26] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b58632fcd85f...848dba6c32d7 [17:50:29] [02miraheze/mw-config] 07paladox 03848dba6 - Update LocalSettings.php [17:51:12] miraheze/mw-config - paladox the build passed. [17:51:33] miraheze/mw-config - paladox the build passed. [17:54:13] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [17:54:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:02:52] !log [paladox@mwtask141] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 519s [18:02:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:07:35] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/commit/8ad1212c4c04 [18:07:36] [02miraheze/ManageWiki] 07paladox 038ad1212 - ManageWikiSettings: Add isset check inside remove [18:07:39] [02ManageWiki] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/ManageWiki [18:07:40] [02ManageWiki] 07paladox opened pull request 03#414: ManageWikiSettings: Add isset check inside remove - 13https://github.com/miraheze/ManageWiki/pull/414 [18:08:34] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.40, 3.27, 2.57 [18:08:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.05, 4.08, 2.94 [18:10:10] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/dceffa25bea9...02100a6c8f6e [18:10:13] [02miraheze/ManageWiki] 07paladox 0302100a6 - ManageWikiSettings: Add isset check inside remove (#414) [18:10:16] [02ManageWiki] 07paladox closed pull request 03#414: ManageWikiSettings: Add isset check inside remove - 13https://github.com/miraheze/ManageWiki/pull/414 [18:10:46] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/22225c7e7e7c...edd2da51ecd9 [18:10:48] [02miraheze/mediawiki] 07paladox 03edd2da5 - Update ManageWiki [18:10:56] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.3640605509 secs [18:11:18] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [18:11:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:12:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.39, 3.64, 3.09 [18:12:48] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_40 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/c47b7e186f72...59b724125378 [18:12:50] [02miraheze/mediawiki] 07paladox 0359b7241 - Update ManageWiki [18:12:56] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.02553030849 secs [18:13:02] !log [paladox@test131] starting deploy of {'world': True, 'l10n': True} to all [18:13:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:13:11] [02miraheze/ManageWiki] 07paladox deleted branch 03paladox-patch-1 [18:13:14] [02ManageWiki] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/ManageWiki [18:14:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.30, 2.87, 2.88 [18:16:32] !log [paladox@mwtask141] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 313s [18:16:34] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.70, 3.97, 3.31 [18:16:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:17:41] miraheze/ManageWiki - paladox the build passed. [18:18:34] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.62, 4.23, 3.49 [18:19:27] PROBLEM - fid.koymi.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - fid.koymi.net All nameservers failed to answer the query. [18:19:27] !log [paladox@test131] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 385s [18:19:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:42] PROBLEM - nexus.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'nexus.wiki' expires in 7 day(s) (Tue 25 Jul 2023 18:16:35 GMT +0000). [18:35:09] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [18:48:56] PROBLEM - fid.koymi.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query fid.koymi.net. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:05:09] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [19:05:13] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.44, 2.55, 2.05 [19:07:16] [02puppet] 07paladox closed pull request 03#3299: mediawiki: Add mediawiki-firejail-rsvg-convert script - 13https://github.com/miraheze/puppet/pull/3299 [19:07:17] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [19:07:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/c3b41e607044...6fac26961bae [19:07:23] [02miraheze/puppet] 07paladox 036fac269 - mediawiki: Add mediawiki-firejail-rsvg-convert script (#3299) [19:07:26] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [19:07:37] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/mw-config/compare/3dabd2e97082...ff83b628bcd6 [19:07:39] [02miraheze/mw-config] 07paladox 03ff83b62 - Change wgSVGConverter to use rsvg (#5284) [19:07:41] [02mw-config] 07paladox closed pull request 03#5284: Change wgSVGConverter to use rsvg - 13https://github.com/miraheze/mw-config/pull/5284 [19:07:44] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [19:07:47] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [19:07:49] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [19:07:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:58] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [19:08:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:21] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:08:39] miraheze/mw-config - paladox the build passed. [19:09:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.91, 3.25, 2.50 [19:12:56] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1003654897 secs [19:14:56] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.0947162807 secs [19:18:24] RECOVERY - fid.koymi.net - reverse DNS on sslhost is OK: SSL OK - fid.koymi.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [19:24:49] [02puppet] 07paladox closed pull request 03#3277: mediawiki: use rsyslog-rotate in logrotate - 13https://github.com/miraheze/puppet/pull/3277 [19:24:50] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [19:24:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/6fac26961bae...e03d72ed1842 [19:24:53] [02miraheze/puppet] 07paladox 03e03d72e - mediawiki: use rsyslog-rotate in logrotate (#3277) [19:24:55] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [19:25:07] [02puppet] 07paladox closed pull request 03#3220: Monitor parsoid warming jobs - 13https://github.com/miraheze/puppet/pull/3220 [19:25:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e03d72ed1842...4018f286deb4 [19:25:12] [02miraheze/puppet] 07RhinosF1 034018f28 - Monitor parsoid warming jobs (#3220) [19:26:34] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.39, 2.95, 3.89 [19:27:31] !log [@test131] starting deploy of {'config': True} to all [19:27:32] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [19:27:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:34:34] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.05, 2.55, 3.32 [19:36:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.84, 2.96, 2.27 [19:37:58] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [19:38:50] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.25, 2.65, 2.24 [19:44:45] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.98, 3.63, 2.76 [19:46:43] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.79, 3.39, 2.78 [20:14:28] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.49, 5.86, 3.80 [20:18:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.49, 3.52, 3.30 [20:20:24] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.35, 2.80, 3.06 [20:30:56] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset -0.1092181206 secs [20:40:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.49, 3.01, 2.53 [20:40:56] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.09495589137 secs [20:42:08] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.24, 3.16, 2.65 [21:00:53] RECOVERY - wiki.pixlies.net - reverse DNS on sslhost is OK: SSL OK - wiki.pixlies.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [21:36:38] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.75, 4.12, 2.55 [21:48:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.07, 3.47, 3.82 [21:52:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.36, 2.58, 3.35 [22:03:18] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1016504765 secs [22:04:56] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.308339119 secs [22:05:09] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [22:05:18] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.08564633131 secs [22:06:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.79, 3.22, 2.97 [22:07:45] PROBLEM - wiki.thundamoo.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.thundamoo.com' expires in 7 day(s) (Tue 25 Jul 2023 21:59:01 GMT +0000). [22:08:58] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [22:10:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.20, 2.76, 2.85 [22:18:56] PROBLEM - cp32 NTP time on cp32 is CRITICAL: NTP CRITICAL: Offset 0.5003845394 secs [22:20:56] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.00294303894 secs [22:35:08] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nj.cn.eu.org could not be found [22:36:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.11, 2.89, 2.25 [22:38:36] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [22:50:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.46, 3.47, 3.47 [22:52:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.96, 3.22, 3.36 [23:36:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.81, 3.94, 2.84