[00:02:38] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw142 [00:04:07] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 14.68, 8.20, 3.79 [00:04:21] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.50, 5.90, 4.05 [00:04:38] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [00:05:03] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 5.31, 3.61, 2.38 [00:06:21] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.06, 5.48, 4.15 [00:07:04] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 3.33, 3.29, 2.41 [00:16:07] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.61, 6.33, 6.33 [00:19:46] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.34, 3.70, 2.38 [00:21:56] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.72, 3.01, 2.35 [00:23:48] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.47, 3.85, 2.78 [00:23:56] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.21, 2.99, 2.42 [00:24:46] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.22, 3.43, 3.08 [00:25:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.78, 7.47, 6.39 [00:25:48] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.02, 3.18, 2.66 [00:26:57] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.52, 6.52, 6.17 [00:28:40] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.72, 3.22, 3.08 [00:30:09] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.88, 4.12, 2.87 [00:30:10] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 14.30, 10.08, 6.22 [00:30:22] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/dc028c3a9ebb [00:30:23] [02miraheze/mw-config] 07paladox 03dc028c3 - Set memory_limit to 1.4G [00:30:25] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [00:30:27] [02mw-config] 07paladox opened pull request 03#5386: Set memory_limit to 1.4G - 13https://github.com/miraheze/mw-config/pull/5386 [00:30:29] [02mw-config] 07paladox closed pull request 03#5386: Set memory_limit to 1.4G - 13https://github.com/miraheze/mw-config/pull/5386 [00:30:31] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b6716b9ccaa1...41d48ae78dd4 [00:30:34] [02miraheze/mw-config] 07paladox 0341d48ae - Set memory_limit to 1.4G (#5386) [00:30:35] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [00:30:38] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw131 [00:30:38] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [00:31:06] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [00:31:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:31:17] miraheze/mw-config - paladox the build passed. [00:31:23] miraheze/mw-config - paladox the build passed. [00:31:52] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 14.84, 11.07, 7.96 [00:32:06] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 7.32, 8.96, 6.27 [00:32:08] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.30, 3.64, 2.84 [00:32:38] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [00:32:39] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 93s [00:32:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:33:52] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 8.17, 9.75, 7.85 [00:34:08] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.11, 3.69, 2.94 [00:35:44] !log [paladox@mwtask141] starting deploy of {'config': True} to all [00:36:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:36:07] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.15, 3.42, 2.93 [00:36:08] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 24s [00:36:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:38:08] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.14, 3.62, 3.06 [00:38:23] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.46, 3.42, 3.20 [00:40:21] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.27, 3.61, 3.29 [00:42:18] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.62, 3.53, 3.29 [00:44:11] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.89, 3.90, 3.45 [00:44:15] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.85, 3.23, 3.21 [00:47:01] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:48:10] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.19, 3.23, 3.29 [00:51:04] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 2.331 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [00:56:39] !log [@test131] starting deploy of {'config': True} to all [00:56:40] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [00:56:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:56:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:58:32] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [01:07:46] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 12.91, 9.63, 7.59 [01:08:14] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.65, 3.68, 3.13 [01:08:16] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:09:44] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 9.10, 9.91, 7.99 [01:10:13] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.66, 3.41, 3.11 [01:10:16] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 5.067 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [01:12:12] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.43, 2.34, 2.75 [01:29:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.02, 6.25, 5.78 [01:31:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.25, 6.13, 5.79 [01:35:42] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [01:53:31] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw131 [01:55:31] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [01:59:52] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:00:01] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 3 backends are down. mw141 mw142 mw143 [02:00:19] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw142 mw143 [02:00:52] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 24.64, 15.25, 9.24 [02:00:52] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 21.94, 17.46, 10.70 [02:01:21] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw143 [02:01:27] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 12.66, 10.42, 8.20 [02:01:28] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 20.76, 15.65, 10.62 [02:01:31] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw143 [02:02:00] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.69, 3.49, 1.92 [02:02:07] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:03:27] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 9.44, 10.37, 8.48 [02:04:05] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 32 minutes ago with 0 failures [02:05:12] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [02:05:31] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [02:05:59] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.36, 3.94, 2.53 [02:06:01] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [02:06:04] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [02:07:27] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 6.09, 8.99, 8.43 [02:12:21] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 3 backends are down. mw142 mw134 mw143 [02:12:49] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 4 backends are down. mw141 mw142 mw134 mw143 [02:12:50] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 4 backends are down. mw141 mw142 mw134 mw143 [02:12:56] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 4 backends are down. mw141 mw142 mw134 mw143 [02:13:07] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.34, 11.46, 9.46 [02:13:21] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 12.47, 12.26, 9.46 [02:13:59] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.15, 3.18, 2.77 [02:14:16] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [02:14:44] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [02:14:45] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [02:14:51] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [02:15:17] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 8.10, 10.66, 9.20 [02:16:58] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.57, 11.44, 9.96 [02:17:13] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 5.61, 9.06, 8.79 [02:18:14] PROBLEM - mw143 Current Load on mw143 is WARNING: WARNING - load average: 2.78, 9.25, 11.03 [02:18:53] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 5.67, 9.36, 9.37 [02:20:09] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 4.77, 8.12, 11.31 [02:20:09] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 2.88, 7.16, 10.05 [02:20:50] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 5.19, 8.20, 11.52 [02:24:05] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 3.76, 5.76, 9.61 [02:24:46] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 4.02, 6.06, 9.90 [02:31:03] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [02:36:01] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw142 [02:38:01] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [02:53:31] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw141 mw143 [02:55:31] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [03:03:44] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw141 [03:04:01] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw141 [03:04:28] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 16.75, 14.16, 9.57 [03:04:50] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 15.61, 12.75, 8.39 [03:05:17] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 12.19, 10.80, 7.83 [03:05:39] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [03:06:01] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [03:06:46] PROBLEM - mw143 Current Load on mw143 is WARNING: WARNING - load average: 9.79, 11.44, 8.41 [03:07:15] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 7.44, 9.46, 7.69 [03:08:28] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 5.92, 10.64, 9.25 [03:08:42] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 4.88, 9.01, 7.88 [03:10:28] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 5.23, 8.86, 8.76 [03:28:18] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [03:30:30] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [03:30:57] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp24.miraheze.org - CNAME OK [03:58:19] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [03:58:28] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.98, 4.76, 3.12 [03:58:41] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [04:00:25] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.80, 5.38, 3.59 [04:02:30] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [04:05:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/90ab71a3c24f [04:05:56] [02miraheze/puppet] 07paladox 0390ab71a - Disable jobrunner on some mws and enable it on others [04:05:59] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [04:06:01] [02puppet] 07paladox opened pull request 03#3528: Disable jobrunner on some mws and enable it on others - 13https://github.com/miraheze/puppet/pull/3528 [04:06:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/90ab71a3c24f...0f7250d6d6e3 [04:06:32] [02miraheze/puppet] 07paladox 030f7250d - Update mw133.yaml [04:06:35] [02puppet] 07paladox synchronize pull request 03#3528: Disable jobrunner on some mws and enable it on others - 13https://github.com/miraheze/puppet/pull/3528 [04:06:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0f7250d6d6e3...e8c865562763 [04:06:41] [02miraheze/puppet] 07paladox 03e8c8655 - Update mw134.yaml [04:06:42] [02puppet] 07paladox synchronize pull request 03#3528: Disable jobrunner on some mws and enable it on others - 13https://github.com/miraheze/puppet/pull/3528 [04:07:23] [02puppet] 07paladox synchronize pull request 03#3528: Disable jobrunner on some mws and enable it on others - 13https://github.com/miraheze/puppet/pull/3528 [04:07:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e8c865562763...8b0ac40b6854 [04:07:25] [02miraheze/puppet] 07paladox 038b0ac40 - Update mw134.yaml [04:07:37] [02puppet] 07paladox closed pull request 03#3528: Disable jobrunner on some mws and enable it on others - 13https://github.com/miraheze/puppet/pull/3528 [04:07:40] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [04:07:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/ca176d3e9df0...6f8610c56e6c [04:07:44] [02miraheze/puppet] 07paladox 036f8610c - Disable jobrunner on some mws and enable it on others (#3528) [04:07:47] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [04:12:16] PROBLEM - mw141 JobRunner Service on mw141 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [04:23:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6f8610c56e6c...af9b713a4400 [04:23:29] [02miraheze/puppet] 07paladox 03af9b713 - jobrunner-hi: add refreshLinksPrioritized to html queue [04:25:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/af9b713a4400...5e129edf69b2 [04:25:32] [02miraheze/puppet] 07paladox 035e129ed - jobrunner: exclude RecordLintJob [04:28:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5e129edf69b2...5271ec247501 [04:28:06] [02miraheze/puppet] 07paladox 035271ec2 - fix [04:30:22] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [04:30:52] PROBLEM - mwtask141 JobRunner Service on mwtask141 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [04:35:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5271ec247501...cb4c37d3e638 [04:35:38] [02miraheze/puppet] 07paladox 03cb4c37d - jobrunner: allow running parsoidCachePrewarm but as lower priority [04:45:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cb4c37d3e638...f3ffe735872a [04:45:42] [02miraheze/puppet] 07paladox 03f3ffe73 - fix [04:46:40] RECOVERY - mwtask141 JobRunner Service on mwtask141 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [05:20:42] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [05:53:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/a71e8109a573 [05:53:27] [02miraheze/puppet] 07paladox 03a71e810 - mediawiki::nutcracker: lower timeout to 250ms [05:53:28] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [05:53:29] [02puppet] 07paladox opened pull request 03#3529: mediawiki::nutcracker: lower timeout to 250ms - 13https://github.com/miraheze/puppet/pull/3529 [05:55:31] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [05:56:34] [02puppet] 07paladox closed pull request 03#3529: mediawiki::nutcracker: lower timeout to 250ms - 13https://github.com/miraheze/puppet/pull/3529 [05:56:35] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [05:56:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f3ffe735872a...36e13cf900ae [05:56:40] [02miraheze/puppet] 07paladox 0336e13cf - mediawiki::nutcracker: lower timeout to 250ms (#3529) [05:56:42] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [05:57:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/41d48ae78dd4...f6e45f0c7fcd [05:57:17] [02miraheze/mw-config] 07paladox 03f6e45f0 - Lower memcached time out to 250ms [05:57:51] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [05:57:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:58:00] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [05:58:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:58:06] !log [paladox@mwtask141] starting deploy of {'config': True} to all [05:58:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:58:12] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 6s [05:58:16] miraheze/mw-config - paladox the build passed. [05:58:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:59:57] !log [@test131] starting deploy of {'config': True} to all [05:59:58] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [06:00:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:00:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:04:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/36e13cf900ae...0592d845ccc0 [06:04:38] [02miraheze/puppet] 07paladox 030592d84 - Update jobrunner-hi.json.erb [06:24:52] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:57:29] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw141 [07:57:46] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw141 [07:57:52] PROBLEM - mw143 Current Load on mw143 is WARNING: WARNING - load average: 10.55, 11.22, 6.76 [07:59:29] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [07:59:46] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [07:59:52] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 3.75, 8.40, 6.26 [08:27:46] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [08:59:19] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [09:30:15] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [09:46:44] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [09:59:42] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [10:17:21] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:41:47] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.007178097963 secs [11:39:22] PROBLEM - db131 Backups SQL on db131 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864191 seconds old and 136563 bytes [11:47:39] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [12:17:00] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [13:42:08] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [14:14:08] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:25:29] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [15:04:45] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.101354152 secs [15:12:08] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [15:26:41] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:44:08] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:46:03] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query franchising.org.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:05:22] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 5.09, 3.02, 2.04 [16:09:21] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.39, 3.54, 2.51 [16:13:21] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 3.38, 3.39, 2.69 [16:15:04] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp25.miraheze.org - CNAME OK [16:16:34] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.67, 6.81, 6.24 [16:17:21] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.91, 3.58, 2.92 [16:18:31] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.02, 6.76, 6.31 [16:23:21] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.66, 3.28, 3.01 [16:37:21] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.55, 3.47, 3.13 [16:41:21] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.36, 3.10, 3.07 [16:53:06] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.43, 3.21, 3.13 [16:55:03] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.82, 3.09, 3.10 [17:30:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/555dc39ee1f6 [17:30:43] [02miraheze/puppet] 07paladox 03555dc39 - mediawiki::nutcracker: set nice to -19 for service [17:30:46] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [17:30:49] [02puppet] 07paladox opened pull request 03#3530: mediawiki::nutcracker: set nice to -19 for service - 13https://github.com/miraheze/puppet/pull/3530 [17:30:50] [02puppet] 07paladox closed pull request 03#3530: mediawiki::nutcracker: set nice to -19 for service - 13https://github.com/miraheze/puppet/pull/3530 [17:30:52] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [17:30:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0592d845ccc0...53d38bea4ad6 [17:30:54] [02miraheze/puppet] 07paladox 0353d38be - mediawiki::nutcracker: set nice to -19 for service (#3530) [17:30:57] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [17:34:49] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 10.41, 10.36, 6.70 [17:36:39] PROBLEM - graylog131 Puppet on graylog131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:36:49] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 4.82, 8.34, 6.40 [18:02:38] RECOVERY - graylog131 Puppet on graylog131 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:24:35] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [18:56:08] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:26:28] PROBLEM - wiki.seamly.net - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.seamly.net and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:34:46] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f6e45f0c7fcd...2b7881edd8e8 [19:34:48] [02miraheze/mw-config] 07paladox 032b7881e - Memcached: Set allow_tcp_nagle_delay [19:35:10] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [19:35:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:35:21] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [19:35:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:35:27] !log [paladox@mwtask141] starting deploy of {'config': True} to all [19:35:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:35:46] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 19s [19:35:49] miraheze/mw-config - paladox the build passed. [19:35:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:35:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/53d38bea4ad6...846af10205f7 [19:35:56] [02miraheze/puppet] 07paladox 03846af10 - nutcracker: increase timeout to 500ms [19:37:31] !log [@test131] starting deploy of {'config': True} to all [19:37:32] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [19:37:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:39:01] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.seamly.net could not be found [19:57:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/846af10205f7...7aa92335ce23 [19:57:40] [02miraheze/puppet] 07paladox 037aa9233 - Update nutcracker.pp [20:12:23] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [20:13:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.58, 7.44, 6.48 [20:17:23] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.05, 6.33, 6.26 [20:40:23] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:40:34] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.06, 3.27, 2.33 [20:42:35] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.85, 3.09, 2.38 [21:19:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.63, 6.25, 6.01 [21:21:05] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.58, 6.20, 6.02 [21:51:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.40, 7.13, 6.58 [21:53:09] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.44, 6.61, 6.46 [21:54:03] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [22:04:36] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.20, 3.33, 1.81 [22:12:08] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [22:12:23] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [22:14:35] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.04, 3.54, 2.88 [22:16:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.24, 6.69, 6.34 [22:16:32] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.76, 2.85, 2.70 [22:18:27] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.21, 6.05, 6.14 [22:24:14] PROBLEM - cloud14 Puppet on cloud14 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [22:25:39] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [22:44:08] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [22:52:14] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:56:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.07, 7.53, 6.74 [22:56:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03add-mcrouter [+30/-0/±1] 13https://github.com/miraheze/puppet/commit/ef55ccc09ffc [22:57:01] [02miraheze/puppet] 07paladox 03ef55ccc - Add mcrouter module [22:57:02] [02puppet] 07paladox created branch 03add-mcrouter - 13https://github.com/miraheze/puppet [22:57:11] [02puppet] 07paladox opened pull request 03#3531: Add mcrouter module - 13https://github.com/miraheze/puppet/pull/3531 [22:58:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.25, 7.57, 6.86 [23:02:27] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.34, 6.45, 6.57 [23:05:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03add-mcrouter [+30/-0/±1] 13https://github.com/miraheze/puppet/compare/ef55ccc09ffc...09821669108a [23:05:57] [02miraheze/puppet] 07paladox 030982166 - Add mcrouter module [23:06:00] [02puppet] 07paladox synchronize pull request 03#3531: Add mcrouter module - 13https://github.com/miraheze/puppet/pull/3531 [23:08:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03add-mcrouter [+5/-0/±1] 13https://github.com/miraheze/puppet/compare/09821669108a...a4a38497d1dd [23:08:08] [02miraheze/puppet] 07paladox 03a4a3849 - Add mcrouter module [23:08:10] [02puppet] 07paladox synchronize pull request 03#3531: Add mcrouter module - 13https://github.com/miraheze/puppet/pull/3531 [23:08:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03add-mcrouter [+5/-0/±1] 13https://github.com/miraheze/puppet/compare/a4a38497d1dd...9f30fdea048b [23:08:48] [02miraheze/puppet] 07paladox 039f30fde - Add mcrouter module [23:08:51] [02puppet] 07paladox synchronize pull request 03#3531: Add mcrouter module - 13https://github.com/miraheze/puppet/pull/3531 [23:10:23] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:11:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03add-mcrouter [+5/-0/±1] 13https://github.com/miraheze/puppet/compare/9f30fdea048b...b155f73951cc [23:11:49] [02miraheze/puppet] 07paladox 03b155f73 - Add mcrouter module [23:11:50] [02puppet] 07paladox synchronize pull request 03#3531: Add mcrouter module - 13https://github.com/miraheze/puppet/pull/3531 [23:14:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.40, 7.01, 6.68 [23:16:27] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.67, 6.65, 6.57 [23:17:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03add-mcrouter [+5/-0/±1] 13https://github.com/miraheze/puppet/compare/b155f73951cc...22ed8da8cc7e [23:17:21] [02miraheze/puppet] 07paladox 0322ed8da - Add mcrouter module [23:17:23] [02puppet] 07paladox synchronize pull request 03#3531: Add mcrouter module - 13https://github.com/miraheze/puppet/pull/3531 [23:24:57] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [23:33:37] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2b7881edd8e8...8aafc450a45f [23:33:38] [02miraheze/mw-config] 07paladox 038aafc45 - Set wgStatsCacheType and wgMicroStashType to use memcache [23:34:03] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [23:34:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:34:29] miraheze/mw-config - paladox the build passed. [23:34:32] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 28s [23:34:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:34:39] !log [paladox@mwtask141] starting deploy of {'config': True} to all [23:34:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:34:47] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [23:34:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:35:24] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [23:35:26] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [23:35:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:35:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:35:36] !log [paladox@test131] starting deploy of {'config': True} to all [23:35:37] !log [paladox@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [23:35:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:35:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:42:08] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [23:49:14] PROBLEM - wiki.col6.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.col6.de' expires in 15 day(s) (Thu 14 Dec 2023 23:46:25 GMT +0000). [23:50:02] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/340b56862134...55a7d2f54022 [23:50:05] [02miraheze/ssl] 07MirahezeSSLBot 0355a7d2f - Bot: Update SSL cert for wiki.col6.de [23:55:18] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/546d16653fb2 [23:55:21] [02miraheze/mw-config] 07paladox 03546d166 - Use mcrouter [23:55:22] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [23:55:23] [02mw-config] 07paladox opened pull request 03#5387: Use mcrouter - 13https://github.com/miraheze/mw-config/pull/5387 [23:56:21] miraheze/mw-config - paladox the build passed. [23:56:31] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [23:56:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03add-mcrouter [+5/-0/±2] 13https://github.com/miraheze/puppet/compare/22ed8da8cc7e...9d45c61fa610 [23:57:00] [02miraheze/puppet] 07paladox 039d45c61 - Add mcrouter module [23:57:07] [02puppet] 07paladox synchronize pull request 03#3531: Add mcrouter module - 13https://github.com/miraheze/puppet/pull/3531 [23:58:03] PROBLEM - zomvilles.pixiemeta.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'zomvilles.pixiemeta.com' expires in 15 day(s) (Thu 14 Dec 2023 23:48:13 GMT +0000). [23:58:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/55a7d2f54022...2c7b5a10cf6a [23:58:18] [02miraheze/ssl] 07MirahezeSSLBot 032c7b5a1 - Bot: Update SSL cert for zomvilles.pixiemeta.com