[00:04:17] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 16.56, 10.14, 4.62 [00:04:34] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.41, 2.73, 1.65 [00:05:59] PROBLEM - wiki.manos.pt - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.manos.pt' expires in 15 day(s) (Thu 14 Dec 2023 23:48:36 GMT +0000). [00:06:14] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2c7b5a10cf6a...e4c54eded8ac [00:06:17] [02miraheze/ssl] 07MirahezeSSLBot 03e4c54ed - Bot: Update SSL cert for wiki.manos.pt [00:06:35] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.17, 2.52, 1.72 [00:07:46] PROBLEM - wiki.lewisgill.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.lewisgill.com' expires in 15 day(s) (Thu 14 Dec 2023 23:47:50 GMT +0000). [00:08:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e4c54eded8ac...2c638acfdbe1 [00:08:11] [02miraheze/ssl] 07MirahezeSSLBot 032c638ac - Bot: Update SSL cert for wiki.lewisgill.com [00:11:17] PROBLEM - revi.wiki - PositiveSSLDV on sslhost is WARNING: WARNING - Certificate 'revi.wiki' expires in 30 day(s) (Fri 29 Dec 2023 23:59:59 GMT +0000). [00:12:23] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 14.26, 10.37, 7.57 [00:12:29] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.88, 2.90, 1.73 [00:14:23] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 5.99, 8.66, 7.28 [00:14:28] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 3.29, 3.00, 1.91 [00:14:40] PROBLEM - wiki.sheepservermc.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.sheepservermc.net' expires in 15 day(s) (Thu 14 Dec 2023 23:47:00 GMT +0000). [00:14:53] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2c638acfdbe1...7a08b4af0054 [00:14:55] [02miraheze/ssl] 07MirahezeSSLBot 037a08b4a - Bot: Update SSL cert for wiki.sheepservermc.net [00:15:42] PROBLEM - archives.rezehno.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'archives.rezehno.org' expires in 15 day(s) (Thu 14 Dec 2023 23:47:29 GMT +0000). [00:15:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7a08b4af0054...c9f9cd2fafb2 [00:15:55] [02miraheze/ssl] 07MirahezeSSLBot 03c9f9cd2 - Bot: Update SSL cert for archives.rezehno.org [00:16:17] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.11, 6.48, 6.73 [00:18:54] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:22:29] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.33, 4.15, 2.79 [00:23:39] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.16, 4.07, 2.88 [00:23:54] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.86, 9.67, 7.12 [00:24:23] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 13.91, 11.51, 8.99 [00:24:30] PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 12.92, 10.15, 7.09 [00:24:57] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw132 [00:25:52] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 9.54, 9.56, 7.39 [00:26:10] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [00:26:30] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 9.48, 9.97, 7.41 [00:26:57] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [00:27:39] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.75, 4.00, 3.14 [00:28:23] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 9.93, 11.11, 9.45 [00:29:39] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 3.75, 4.02, 3.25 [00:30:23] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 7.51, 9.98, 9.25 [00:31:41] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.19, 3.50, 3.16 [00:32:30] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.20, 3.69, 3.32 [00:33:39] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.31, 3.97, 3.38 [00:34:28] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.46, 3.38, 3.26 [00:35:39] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.53, 3.25, 3.17 [00:36:37] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 16.10, 12.75, 9.63 [00:36:41] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:37:31] RECOVERY - wiki.lewisgill.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lewisgill.com' will expire on Mon 26 Feb 2024 23:08:02 GMT +0000. [00:38:13] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.53, 4.66, 3.30 [00:38:21] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 12.81, 10.85, 7.26 [00:38:23] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.83, 10.24, 9.31 [00:38:38] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.366 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [00:38:40] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.80, 3.76, 3.44 [00:39:37] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 10.66, 11.57, 8.65 [00:39:41] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 11.13, 12.85, 9.94 [00:40:23] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 7.83, 9.09, 8.99 [00:40:39] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.61, 3.24, 3.28 [00:41:54] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 12.29, 12.12, 8.14 [00:41:56] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:42:44] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw141 [00:43:36] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 7.64, 9.28, 8.36 [00:43:50] RECOVERY - wiki.sheepservermc.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.sheepservermc.net' will expire on Mon 26 Feb 2024 23:14:46 GMT +0000. [00:43:54] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 8.21, 11.80, 8.59 [00:44:08] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:44:23] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 28.70, 16.10, 11.61 [00:44:24] RECOVERY - archives.rezehno.org - LetsEncrypt on sslhost is OK: OK - Certificate 'archives.rezehno.org' will expire on Mon 26 Feb 2024 23:15:47 GMT +0000. [00:44:42] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.11, 3.68, 3.44 [00:45:58] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 20.94, 14.30, 9.80 [00:46:00] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw141 mw142 [00:46:28] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.03, 3.63, 3.33 [00:46:39] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [00:46:40] PROBLEM - mw143 Current Load on mw143 is WARNING: WARNING - load average: 9.76, 11.81, 9.66 [00:47:34] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 6.03, 10.58, 10.34 [00:47:43] PROBLEM - wiki.col6.de - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.col6.de and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:47:59] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [00:48:11] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 5.45, 10.10, 10.44 [00:48:22] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [00:48:26] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.08, 3.05, 3.15 [00:48:38] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 0.95, 2.83, 3.20 [00:48:38] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 13.00, 11.49, 9.75 [00:49:32] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.73, 10.02, 10.16 [00:50:20] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:50:23] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 6.86, 10.72, 10.76 [00:50:34] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw141 mw143 [00:50:48] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw141 mw143 [00:51:00] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 3 backends are down. mw141 mw142 mw143 [00:51:04] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:52:28] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:52:32] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [00:52:44] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [00:52:59] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [00:53:04] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.322 second response time [00:54:02] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.34, 3.94, 3.94 [00:55:53] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 6.60, 8.79, 9.98 [00:56:09] RECOVERY - zomvilles.pixiemeta.com - LetsEncrypt on sslhost is OK: OK - Certificate 'zomvilles.pixiemeta.com' will expire on Mon 26 Feb 2024 22:58:09 GMT +0000. [00:56:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03add-mcrouter [+7/-1/±3] 13https://github.com/miraheze/puppet/compare/9d45c61fa610...013c2d46145a [00:56:13] [02miraheze/puppet] 07paladox 03013c2d4 - Add mcrouter module [00:56:14] [02puppet] 07paladox synchronize pull request 03#3531: Add mcrouter module - 13https://github.com/miraheze/puppet/pull/3531 [00:56:23] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 8.00, 9.22, 10.10 [00:56:39] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 6.30, 11.91, 11.67 [00:57:00] PROBLEM - mw143 Current Load on mw143 is WARNING: WARNING - load average: 5.32, 10.10, 10.74 [00:58:58] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 4.09, 8.37, 10.05 [00:59:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03add-mcrouter [+7/-1/±3] 13https://github.com/miraheze/puppet/compare/013c2d46145a...3af0bac0bc08 [00:59:48] [02miraheze/puppet] 07paladox 033af0bac - Add mcrouter module [00:59:51] [02puppet] 07paladox synchronize pull request 03#3531: Add mcrouter module - 13https://github.com/miraheze/puppet/pull/3531 [01:01:14] PROBLEM - test131 APT on test131 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [01:02:22] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 3.69, 6.91, 9.51 [01:03:15] PROBLEM - test131 APT on test131 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (13 critical updates). [01:03:47] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.33, 2.81, 3.38 [01:04:11] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:05:22] RECOVERY - wiki.manos.pt - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.manos.pt' will expire on Mon 26 Feb 2024 23:06:06 GMT +0000. [01:05:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03add-mcrouter [+7/-1/±3] 13https://github.com/miraheze/puppet/compare/3af0bac0bc08...ea426c4ad471 [01:05:48] [02miraheze/puppet] 07paladox 03ea426c4 - Add mcrouter module [01:05:51] [02puppet] 07paladox synchronize pull request 03#3531: Add mcrouter module - 13https://github.com/miraheze/puppet/pull/3531 [01:08:10] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [01:08:20] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [01:12:47] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [01:14:12] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 11.80, 9.57, 8.75 [01:14:29] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 14.39, 12.35, 10.18 [01:16:07] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 6.70, 8.37, 8.40 [01:16:23] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 6.09, 10.11, 9.63 [01:17:14] RECOVERY - wiki.col6.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.col6.de' will expire on Mon 26 Feb 2024 22:49:53 GMT +0000. [01:22:01] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.98, 3.73, 3.17 [01:23:58] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.85, 3.35, 3.10 [01:27:23] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:38:10] PROBLEM - test131 MediaWiki Rendering on test131 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [01:40:14] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.846 second response time [01:44:08] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:06:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03add-mcrouter [+7/-1/±3] 13https://github.com/miraheze/puppet/compare/ea426c4ad471...c22583cbf9cc [02:07:01] [02miraheze/puppet] 07paladox 03c22583c - Add mcrouter module [02:07:04] [02puppet] 07paladox synchronize pull request 03#3531: Add mcrouter module - 13https://github.com/miraheze/puppet/pull/3531 [02:10:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.08, 6.65, 6.25 [02:12:56] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.20, 6.51, 6.25 [02:18:58] PROBLEM - test131 nutcracker process on test131 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 114 (nutcracker), command name 'nutcracker' [02:24:09] [02puppet] 07paladox closed pull request 03#3531: Add mcrouter module - 13https://github.com/miraheze/puppet/pull/3531 [02:24:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+7/-1/±3] 13https://github.com/miraheze/puppet/compare/7aa92335ce23...4e0f0962ceec [02:24:14] [02miraheze/puppet] 07paladox 034e0f096 - Add mcrouter module (#3531) [02:24:17] [02miraheze/puppet] 07paladox deleted branch 03add-mcrouter [02:24:18] [02puppet] 07paladox deleted branch 03add-mcrouter - 13https://github.com/miraheze/puppet [02:25:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/93b2970c071d [02:25:29] [02miraheze/puppet] 07paladox 0393b2970 - Use mcrouter on test131 [02:25:30] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [02:25:32] [02puppet] 07paladox opened pull request 03#3532: Use mcrouter on test131 - 13https://github.com/miraheze/puppet/pull/3532 [02:26:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/93b2970c071d...c11bae9d5579 [02:26:07] [02miraheze/puppet] 07paladox 03c11bae9 - Update test131.yaml [02:26:09] [02puppet] 07paladox synchronize pull request 03#3532: Use mcrouter on test131 - 13https://github.com/miraheze/puppet/pull/3532 [02:26:25] [02puppet] 07paladox closed pull request 03#3532: Use mcrouter on test131 - 13https://github.com/miraheze/puppet/pull/3532 [02:26:27] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [02:26:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/4e0f0962ceec...f5f2ea140297 [02:26:30] [02miraheze/puppet] 07paladox 03f5f2ea1 - Use mcrouter on test131 (#3532) [02:26:32] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [02:26:58] RECOVERY - test131 nutcracker process on test131 is OK: PROCS OK: 1 process with UID = 114 (nutcracker), command name 'nutcracker' [02:27:27] [02mw-config] 07paladox edited pull request 03#5387: Use mcrouter on beta - 13https://github.com/miraheze/mw-config/pull/5387 [02:28:46] [02mw-config] 07paladox synchronize pull request 03#5387: Use mcrouter on beta - 13https://github.com/miraheze/mw-config/pull/5387 [02:28:49] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/546d16653fb2...b481053330cb [02:28:52] [02miraheze/mw-config] 07paladox 03b481053 - Update GlobalCache.php [02:29:43] miraheze/mw-config - paladox the build passed. [02:29:56] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b481053330cb...d48badac1eb5 [02:29:58] [02miraheze/mw-config] 07paladox 03d48bada - Update SemanticMediaWiki.php [02:30:01] [02mw-config] 07paladox synchronize pull request 03#5387: Use mcrouter on beta - 13https://github.com/miraheze/mw-config/pull/5387 [02:30:42] [02mw-config] 07paladox closed pull request 03#5387: Use mcrouter on beta - 13https://github.com/miraheze/mw-config/pull/5387 [02:30:44] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/8aafc450a45f...71c2a744d450 [02:30:47] [02miraheze/mw-config] 07paladox 0371c2a74 - Use mcrouter on beta (#5387) [02:30:50] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [02:30:51] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [02:31:00] miraheze/mw-config - paladox the build passed. [02:31:25] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [02:31:28] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 2s [02:31:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:31:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:31:35] !log [paladox@test131] starting deploy of {'config': True} to all [02:31:36] !log [paladox@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [02:31:37] miraheze/mw-config - paladox the build passed. [02:31:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:31:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:32:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f5f2ea140297...fc661991f766 [02:32:41] [02miraheze/puppet] 07paladox 03fc66199 - Fix [02:36:58] PROBLEM - test131 nutcracker process on test131 is UNKNOWN: check_procs: User name was not found - nutcrackerUsage:check_procs -w -c [-m metric] [-s state] [-p ppid] [-u user] [-r rss] [-z vsz] [-P %cpu] [-a argument-array] [-C command] [-k] [-t timeout] [-v] [02:39:49] !log [@mwtask141] starting deploy of {'config': True} to all [02:39:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:40:01] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 11s [02:40:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:56:20] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [02:59:23] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/71c2a744d450...1d758c023d33 [02:59:26] [02miraheze/mw-config] 07paladox 031d758c0 - Set MessageCacheError to debug [02:59:54] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [03:00:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:00:05] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [03:00:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:00:11] !log [paladox@mwtask141] starting deploy of {'config': True} to all [03:00:15] miraheze/mw-config - paladox the build passed. [03:00:16] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [03:00:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:00:18] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [03:00:20] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [03:00:21] !log [paladox@test131] starting deploy of {'config': True} to all [03:00:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:00:22] !log [paladox@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [03:00:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:00:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:00:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:00:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:02:25] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1d758c023d33...e3773d87bb5b [03:02:27] [02miraheze/mw-config] 07paladox 03e3773d8 - Set objectcache to warning [03:02:28] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [03:02:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:02:37] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [03:02:39] !log [paladox@mwtask141] starting deploy of {'config': True} to all [03:02:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:02:43] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [03:02:44] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 5s [03:02:45] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [03:02:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:02:47] !log [paladox@test131] starting deploy of {'config': True} to all [03:02:48] !log [paladox@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [03:02:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:03:15] miraheze/mw-config - paladox the build passed. [03:03:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:03:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:03:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:03:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:05:47] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e3773d87bb5b...7b921ff9fc8e [03:05:49] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [03:05:49] [02miraheze/mw-config] 07paladox 037b921ff - Set ParserCache to warning [03:05:51] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [03:05:52] !log [paladox@test131] starting deploy of {'config': True} to all [03:05:53] !log [paladox@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [03:05:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:05:55] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [03:06:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:06:04] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [03:06:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:06:06] !log [paladox@mwtask141] starting deploy of {'config': True} to all [03:06:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:06:11] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 5s [03:06:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:06:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:06:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:06:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:06:37] miraheze/mw-config - paladox the build passed. [03:32:24] when somebody has a chance ircrcbot needs a restart [03:58:14] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [04:09:09] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [04:39:35] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [04:40:37] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [04:56:21] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importImages.php --wiki=nitromepediawiki /home/macfan/images --search-recursively (START) [04:56:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:10:37] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:05:06] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [06:36:30] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [06:40:37] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [06:42:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.78, 7.39, 6.46 [06:44:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.05, 7.06, 6.44 [06:47:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/de22f465b10c [06:47:52] [02miraheze/puppet] 07paladox 03de22f46 - memcached: lower role::memcached::growth_factor to 1.15 [06:47:55] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [06:47:57] [02puppet] 07paladox opened pull request 03#3533: memcached: lower role::memcached::growth_factor to 1.15 - 13https://github.com/miraheze/puppet/pull/3533 [06:48:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/de22f465b10c...a31d7d4965df [06:48:12] [02miraheze/puppet] 07paladox 03a31d7d4 - Update mem141.yaml [06:48:13] [02puppet] 07paladox synchronize pull request 03#3533: memcached: lower role::memcached::growth_factor to 1.15 - 13https://github.com/miraheze/puppet/pull/3533 [06:48:15] [02puppet] 07paladox closed pull request 03#3533: memcached: lower role::memcached::growth_factor to 1.15 - 13https://github.com/miraheze/puppet/pull/3533 [06:48:16] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.53, 6.47, 6.35 [06:48:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/fc661991f766...bfa8c907f51b [06:48:20] [02miraheze/puppet] 07paladox 03bfa8c90 - memcached: lower role::memcached::growth_factor to 1.15 (#3533) [06:48:22] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [06:48:23] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [07:10:37] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [07:25:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.22, 7.08, 6.25 [07:29:19] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.65, 6.35, 6.14 [07:40:37] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [08:10:37] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [08:28:03] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [08:40:37] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [08:52:56] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importImages.php --wiki=nitromepediawiki /home/macfan/images --search-recursively (END - exit=0) [08:53:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:59:59] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [09:30:44] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5000607073 secs [09:40:38] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.gab.pt.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:48:00] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.58, 5.16, 2.76 [09:50:00] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.67, 4.72, 2.91 [10:29:55] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [11:01:51] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [11:11:38] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [11:14:19] PROBLEM - wiki.slimelab.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.slimelab.net could not be found [11:39:38] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [12:40:37] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [13:10:37] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [13:10:39] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.34, 3.20, 1.69 [13:12:39] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.08, 2.35, 1.56 [13:29:39] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 9.37, 10.33, 7.35 [13:31:38] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 4.12, 8.11, 6.89 [13:31:44] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [14:33:39] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:56:56] PROBLEM - db121 Backups SQL on db121 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864150 seconds old and 136074 bytes [15:16:09] @MirahezeRelay [15:40:37] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [15:45:12] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 11.89, 10.51, 7.27 [15:47:11] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.77, 8.88, 7.08 [15:53:52] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.02, 6.36, 5.72 [15:54:39] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.03, 3.51, 2.16 [15:55:47] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.71, 6.39, 5.80 [15:56:41] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.31, 2.89, 2.09 [16:03:37] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [16:04:31] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [16:15:27] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.25, 3.40, 2.84 [16:17:24] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.88, 3.49, 2.94 [16:19:21] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.26, 3.32, 2.94 [16:23:17] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.21, 3.78, 3.21 [16:25:14] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.46, 3.36, 3.13 [16:31:09] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.04, 3.44, 3.25 [16:33:07] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.34, 3.10, 3.15 [16:35:33] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:36:15] paladox: can you please restart ircrcbot when you have a chance? [16:36:30] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:44:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.97, 6.58, 6.30 [16:46:03] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.49, 6.48, 6.29 [16:53:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.77, 6.85, 6.52 [16:55:40] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.78, 6.45, 6.40 [17:10:37] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [17:13:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.13, 7.32, 6.69 [17:15:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.48, 6.85, 6.59 [17:17:19] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.07, 6.66, 6.56 [17:25:53] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.50, 3.01, 2.00 [17:27:51] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.58, 2.85, 2.07 [17:31:56] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.87, 3.28, 2.41 [17:37:53] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 3.17, 3.30, 2.74 [18:10:38] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 15.36, 11.82, 8.18 [18:12:34] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 7.26, 9.86, 7.90 [18:16:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.26, 6.79, 6.31 [18:18:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.59, 6.64, 6.31 [18:28:46] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_41 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/dcf9fb9aad48...9651e95abbba [18:28:48] [02miraheze/mediawiki] 07paladox 039651e95 - Update LinkTitles [18:29:33] !log [paladox@test131] starting deploy of {'world': True} to all [18:29:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:30:39] !log [paladox@test131] finished deploy of {'world': True} to all - SUCCESS in 66s [18:30:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:05:28] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [19:13:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.57, 7.21, 6.58 [19:15:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.47, 6.96, 6.57 [19:19:43] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.59, 6.76, 6.58 [19:20:26] PROBLEM - elsterclopedie.marithaime.nl - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query elsterclopedie.marithaime.nl. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:36:24] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.39, 6.30, 5.86 [19:37:24] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:37:42] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.56, 3.05, 2.56 [19:38:19] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.17, 6.28, 5.91 [19:39:42] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 1.48, 2.54, 2.44 [19:45:37] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.59, 7.22, 6.60 [19:47:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.87, 7.35, 6.72 [19:49:13] RECOVERY - elsterclopedie.marithaime.nl - reverse DNS on sslhost is OK: SSL OK - elsterclopedie.marithaime.nl reverse DNS resolves to cp25.miraheze.org - CNAME OK [19:49:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.94, 8.06, 7.03 [19:51:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.47, 7.47, 6.94 [19:53:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.51, 7.38, 6.94 [19:55:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.69, 6.82, 6.78 [19:59:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.26, 7.27, 6.95 [20:03:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.78, 7.38, 7.09 [20:05:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.03, 7.59, 7.20 [20:07:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.29, 7.18, 7.10 [20:07:42] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.68, 3.28, 2.84 [20:09:42] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.68, 3.12, 2.85 [20:12:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/f76539cf046c [20:12:20] [02miraheze/puppet] 07paladox 03f76539c - mediawiki::php: tune [20:12:22] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [20:12:24] [02puppet] 07paladox opened pull request 03#3534: mediawiki::php: tune - 13https://github.com/miraheze/puppet/pull/3534 [20:15:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.45, 7.96, 7.40 [20:22:53] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.54, 7.05, 6.37 [20:25:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.54, 7.80, 7.78 [20:26:44] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.27, 6.40, 6.29 [20:41:05] [02puppet] 07paladox synchronize pull request 03#3534: mediawiki::php: tune - 13https://github.com/miraheze/puppet/pull/3534 [20:41:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f76539cf046c...3924e06bc93e [20:41:10] [02miraheze/puppet] 07paladox 033924e06 - Update php.pp [20:41:13] [02puppet] 07paladox closed pull request 03#3534: mediawiki::php: tune - 13https://github.com/miraheze/puppet/pull/3534 [20:41:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bfa8c907f51b...8906c1878c56 [20:41:17] [02miraheze/puppet] 07paladox 038906c18 - mediawiki::php: tune (#3534) [20:41:19] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [20:41:19] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.46, 6.32, 6.73 [20:41:21] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [21:05:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.94, 6.75, 6.52 [21:07:20] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.71, 6.62, 6.49 [21:11:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.81, 7.02, 6.69 [21:13:19] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.99, 6.71, 6.62 [21:46:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/4b288082c251 [21:46:05] [02miraheze/puppet] 07paladox 034b28808 - mediawiki::mcrouter: don't replicate [21:46:06] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [21:46:07] [02puppet] 07paladox opened pull request 03#3535: mediawiki::mcrouter: don't replicate - 13https://github.com/miraheze/puppet/pull/3535 [21:46:18] [02puppet] 07paladox edited pull request 03#3535: mediawiki::mcrouter: don't replicate - 13https://github.com/miraheze/puppet/pull/3535 [21:46:50] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/9389077dbb69 [21:46:53] [02miraheze/mw-config] 07paladox 039389077 - Remove $wgWANObjectCache [21:46:55] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [21:46:58] [02mw-config] 07paladox opened pull request 03#5388: Remove $wgWANObjectCache - 13https://github.com/miraheze/mw-config/pull/5388 [21:47:00] [02mw-config] 07paladox closed pull request 03#5388: Remove $wgWANObjectCache - 13https://github.com/miraheze/mw-config/pull/5388 [21:47:03] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7b921ff9fc8e...521a3700ffc1 [21:47:06] [02miraheze/mw-config] 07paladox 03521a370 - Remove $wgWANObjectCache (#5388) [21:47:09] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [21:47:10] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [21:47:12] [02puppet] 07paladox closed pull request 03#3535: mediawiki::mcrouter: don't replicate - 13https://github.com/miraheze/puppet/pull/3535 [21:47:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8906c1878c56...7c3d8551a4ad [21:47:16] [02miraheze/puppet] 07paladox 037c3d855 - mediawiki::mcrouter: don't replicate (#3535) [21:47:19] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [21:47:20] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [21:47:48] miraheze/mw-config - paladox the build passed. [21:47:59] miraheze/mw-config - paladox the build passed. [21:49:09]