[00:18:14] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 48211MiB (10% inode=99%); [01:27:20] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.94, 6.20, 4.87 [01:29:18] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 4.64, 5.59, 4.81 [01:50:53] PROBLEM - rainverse.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'rainverse.wiki' expires in 14 day(s) (Wed 05 Feb 2025 01:26:05 AM GMT +0000). [01:51:04] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/4f4a164ae8144e36d75f6c731fd4420e67948b8f [01:51:05] 02ssl/03master 07WikiTideSSLBot 034f4a164 Bot: Update SSL cert for rainverse.wiki [02:20:16] RECOVERY - rainverse.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rainverse.wiki' will expire on Mon 21 Apr 2025 12:52:28 AM GMT +0000. [03:40:10] PROBLEM - spiceandwolf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'spiceandwolf.wiki' expires in 14 day(s) (Wed 05 Feb 2025 03:24:43 AM GMT +0000). [03:40:20] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/36d422d52dd98603cd5017403680b2583ceea1bf [03:40:20] 02ssl/03master 07WikiTideSSLBot 0336d422d Bot: Update SSL cert for spiceandwolf.wiki [04:38:28] RECOVERY - spiceandwolf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spiceandwolf.wiki' will expire on Mon 21 Apr 2025 02:41:45 AM GMT +0000. [11:14:07] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 4.14, 2.91, 1.94 [11:16:07] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 2.91, 3.01, 2.11 [11:50:03] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 17.29, 11.30, 7.79 [11:52:00] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 6.17, 9.01, 7.37 [12:07:55] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:08:02] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 13 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw153 mw154 mw163 mw173 mw174 mw183 [12:09:22] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [12:09:25] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:09:25] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 12 backends are down. mw152 mw161 mw162 mw171 mw172 mw181 mw153 mw154 mw163 mw164 mw173 mw183 [12:09:39] Hmm [12:10:03] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4108 bytes in 8.746 second response time [12:10:14] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 26997MiB (5% inode=99%); [12:10:37] PROBLEM - mw184 MediaWiki Rendering on mw184 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:40] PROBLEM - mw184 HTTPS on mw184 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:10:41] PROBLEM - mw174 MediaWiki Rendering on mw174 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:58] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:11:24] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.158 second response time [12:11:28] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.751 second response time [12:12:12] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:40] RECOVERY - mw174 MediaWiki Rendering on mw174 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.420 second response time [12:12:45] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:12:58] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4107 bytes in 5.208 second response time [12:14:08] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.106 second response time [12:14:12] PROBLEM - mw154 HTTPS on mw154 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:14:13] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:14:29] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [12:14:30] RECOVERY - mw184 MediaWiki Rendering on mw184 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.543 second response time [12:14:34] RECOVERY - mw184 HTTPS on mw184 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.051 second response time [12:14:40] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.124 second response time [12:15:10] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:15:10] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:15:45] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:16:10] RECOVERY - mw154 HTTPS on mw154 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.139 second response time [12:16:14] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4108 bytes in 6.424 second response time [12:16:28] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.139 second response time [12:17:00] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:17:04] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.136 second response time [12:17:05] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.486 second response time [12:17:27] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:17:32] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [12:17:42] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.138 second response time [12:18:19] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [12:19:22] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.146 second response time [12:19:31] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.530 second response time [12:20:19] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.516 second response time [12:20:26] PROBLEM - mw174 HTTPS on mw174 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:20:41] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:21:52] PROBLEM - mw184 HTTPS on mw184 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [12:21:52] PROBLEM - mw184 MediaWiki Rendering on mw184 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:21] RECOVERY - mw174 HTTPS on mw174 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.132 second response time [12:22:38] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:22:39] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.385 second response time [12:22:47] PROBLEM - mw154 MediaWiki Rendering on mw154 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.020 second response time [12:23:54] RECOVERY - mw184 MediaWiki Rendering on mw184 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.871 second response time [12:23:54] RECOVERY - mw184 HTTPS on mw184 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 4.651 second response time [12:24:32] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.276 second response time [12:24:36] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:24:47] RECOVERY - mw154 MediaWiki Rendering on mw154 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.611 second response time [12:25:25] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [12:25:48] PROBLEM - cp36 Nginx Backend for mw182 on cp36 is CRITICAL: connect to address localhost and port 8120: Connection refused [12:25:52] PROBLEM - cp36 Nginx Backend for matomo151 on cp36 is CRITICAL: connect to address localhost and port 8203: Connection refused [12:25:54] PROBLEM - hoseki.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address hoseki.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:25:55] PROBLEM - cp36 Nginx Backend for mw151 on cp36 is CRITICAL: connect to address localhost and port 8113: Connection refused [12:26:03] PROBLEM - cp36 Nginx Backend for mw171 on cp36 is CRITICAL: connect to address localhost and port 8117: Connection refused [12:26:05] PROBLEM - rarewarewiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address rarewarewiki.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:26:09] PROBLEM - cp36 Nginx Backend for phorge171 on cp36 is CRITICAL: connect to address localhost and port 8202: Connection refused [12:26:11] PROBLEM - cp36 Nginx Backend for mwtask171 on cp36 is CRITICAL: connect to address localhost and port 8161: Connection refused [12:26:12] PROBLEM - cp36 Nginx Backend for swiftproxy161 on cp36 is CRITICAL: connect to address localhost and port 8206: Connection refused [12:26:17] PROBLEM - cp36 Nginx Backend for mw162 on cp36 is CRITICAL: connect to address localhost and port 8116: Connection refused [12:26:17] PROBLEM - cp36 Nginx Backend for mw154 on cp36 is CRITICAL: connect to address localhost and port 8122: Connection refused [12:26:18] PROBLEM - cp36 Nginx Backend for reports171 on cp36 is CRITICAL: connect to address localhost and port 8205: Connection refused [12:26:20] PROBLEM - cp36 Nginx Backend for swiftproxy171 on cp36 is CRITICAL: connect to address localhost and port 8207: Connection refused [12:26:30] PROBLEM - inourownwords.online - LetsEncrypt on sslhost is CRITICAL: connect to address inourownwords.online and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:26:36] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.436 second response time [12:26:39] PROBLEM - cp36 Nginx Backend for mwtask161 on cp36 is CRITICAL: connect to address localhost and port 8163: Connection refused [12:26:45] PROBLEM - stablestate.org - LetsEncrypt on sslhost is CRITICAL: connect to address stablestate.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:26:48] PROBLEM - cp36 Nginx Backend for mwtask181 on cp36 is CRITICAL: connect to address localhost and port 8160: Connection refused [12:26:51] PROBLEM - cp36 Nginx Backend for mw183 on cp36 is CRITICAL: connect to address localhost and port 8127: Connection refused [12:26:53] PROBLEM - cp36 Nginx Backend for mw181 on cp36 is CRITICAL: connect to address localhost and port 8119: Connection refused [12:26:54] PROBLEM - cp36 Nginx Backend for mw153 on cp36 is CRITICAL: connect to address localhost and port 8121: Connection refused [12:26:58] PROBLEM - cp36 Nginx Backend for mw164 on cp36 is CRITICAL: connect to address localhost and port 8124: Connection refused [12:27:01] PROBLEM - cp36 Nginx Backend for mw152 on cp36 is CRITICAL: connect to address localhost and port 8114: Connection refused [12:27:01] PROBLEM - urbanshade.org - LetsEncrypt on sslhost is CRITICAL: connect to address urbanshade.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:27:02] PROBLEM - beyondtheveilrp.com - LetsEncrypt on sslhost is CRITICAL: connect to address beyondtheveilrp.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:27:06] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.205/cpweb, 2602:294:0:b13::110/cpweb [12:27:11] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.205/cpweb, 2602:294:0:b13::110/cpweb [12:27:48] RECOVERY - cp36 Nginx Backend for mw182 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8120 [12:27:50] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:27:52] RECOVERY - cp36 Nginx Backend for matomo151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8203 [12:27:55] RECOVERY - cp36 Nginx Backend for mw151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8113 [12:28:03] RECOVERY - cp36 Nginx Backend for mw171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8117 [12:28:09] RECOVERY - cp36 Nginx Backend for phorge171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8202 [12:28:11] RECOVERY - cp36 Nginx Backend for mwtask171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8161 [12:28:12] RECOVERY - cp36 Nginx Backend for swiftproxy161 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8206 [12:28:17] RECOVERY - cp36 Nginx Backend for mw162 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8116 [12:28:17] RECOVERY - cp36 Nginx Backend for mw154 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8122 [12:28:18] RECOVERY - cp36 Nginx Backend for reports171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8205 [12:28:20] RECOVERY - cp36 Nginx Backend for swiftproxy171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8207 [12:28:39] RECOVERY - cp36 Nginx Backend for mwtask161 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8163 [12:28:48] RECOVERY - cp36 Nginx Backend for mwtask181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8160 [12:28:49] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4107 bytes in 0.061 second response time [12:28:51] RECOVERY - cp36 Nginx Backend for mw183 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8127 [12:28:53] RECOVERY - cp36 Nginx Backend for mw181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8119 [12:28:54] RECOVERY - cp36 Nginx Backend for mw153 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8121 [12:28:58] RECOVERY - cp36 Nginx Backend for mw164 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8124 [12:29:01] RECOVERY - cp36 Nginx Backend for mw152 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8114 [12:29:06] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:29:11] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:29:25] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:29:25] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 2 backends are down. mw182 mw173 [12:29:49] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.135 second response time [12:31:23] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4130 bytes in 3.552 second response time [12:32:05] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4659 bytes in 0.085 second response time [12:32:45] !log restart db171 [12:32:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:26] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [12:34:00] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.169 second response time [12:34:02] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [12:34:30] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 16.92, 13.68, 9.40 [12:36:27] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.97, 9.42, 8.35 [12:52:01] PROBLEM - changeprop151 Current Load on changeprop151 is WARNING: LOAD WARNING - total load average: 8.75, 13.92, 7.58 [12:54:01] RECOVERY - changeprop151 Current Load on changeprop151 is OK: LOAD OK - total load average: 2.62, 9.91, 6.88 [12:54:41] RECOVERY - rarewarewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'rarewarewiki.com' will expire on Sun 09 Mar 2025 03:52:46 PM GMT +0000. [12:55:31] RECOVERY - inourownwords.online - LetsEncrypt on sslhost is OK: OK - Certificate 'inourownwords.online' will expire on Sat 08 Mar 2025 09:05:57 AM GMT +0000. [12:55:48] RECOVERY - hoseki.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hoseki.wiki' will expire on Sat 08 Mar 2025 01:09:01 PM GMT +0000. [12:56:27] RECOVERY - stablestate.org - LetsEncrypt on sslhost is OK: OK - Certificate 'stablestate.org' will expire on Fri 07 Mar 2025 07:29:55 AM GMT +0000. [12:56:32] RECOVERY - urbanshade.org - LetsEncrypt on sslhost is OK: OK - Certificate 'urbanshade.org' will expire on Mon 24 Mar 2025 02:48:59 PM GMT +0000. [12:56:33] RECOVERY - beyondtheveilrp.com - LetsEncrypt on sslhost is OK: OK - Certificate 'beyondtheveilrp.com' will expire on Fri 21 Feb 2025 07:24:14 PM GMT +0000. [13:06:09] [02CreateWiki] 07BlankEclair created 03fix-missing-cache-sets (+1 new commit) 13https://github.com/miraheze/CreateWiki/commit/450773e01d60 [13:06:09] 02CreateWiki/03fix-missing-cache-sets 07BlankEclair 03450773e Add missing cache timestamp setters [13:06:33] [02CreateWiki] 07BlankEclair opened pull request #650: Add missing cache timestamp setters (03master...03fix-missing-cache-sets) 13https://github.com/miraheze/CreateWiki/pull/650 [13:06:40] [02CreateWiki] 07coderabbitai[bot] commented on pull request #650: --- […] 13https://github.com/miraheze/CreateWiki/pull/650#issuecomment-2604686544 [13:09:02] [02CreateWiki] 07paladox merged 07BlankEclair's pull request #650: Add missing cache timestamp setters (03master...03fix-missing-cache-sets) 13https://github.com/miraheze/CreateWiki/pull/650 [13:09:02] [02CreateWiki] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/CreateWiki/commit/ffe2cb94568f56d496d1112b2c758d149aadd936 [13:09:02] 02CreateWiki/03master 07Claire 03ffe2cb9 Add missing cache timestamp setters (#650) [13:09:03] [02CreateWiki] 07paladox 04deleted 03fix-missing-cache-sets at 03450773e 13https://github.com/miraheze/CreateWiki/commit/450773e [13:09:36] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': ['1.42', '1.43'], 'upgrade_extensions': 'CreateWiki'} to all [13:09:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:12:48] miraheze/CreateWiki - BlankEclair the build has errored. [13:12:54] hah [13:12:57] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': ['1.42', '1.43'], 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 201s [13:12:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:13:18] miraheze/CreateWiki - paladox the build has errored. [13:13:32] > 2025-01-21T13:11:59.0186192Z fatal: unable to access 'https://gerrit.wikimedia.org/r/mediawiki/extensions/MobileFrontend/': Failed to connect to gerrit.wikimedia.org port 443 after 132707 ms: Couldn't connect to server [13:13:36] nvm, not our problem [13:25:29] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/c9a19dbc9c1c [13:25:29] 02puppet/03paladox-patch-2 07paladox 03c9a19db redis: set maxmemory_policy to volatile-lru [13:25:33] [02puppet] 07paladox opened pull request #4141: redis: set maxmemory_policy to volatile-lru (03master...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4141 [13:25:41] [02puppet] 07coderabbitai[bot] commented on pull request #4141: --- […] 13https://github.com/miraheze/puppet/pull/4141#issuecomment-2604732047 [13:25:44] [02puppet] 07paladox merged pull request #4141: redis: set maxmemory_policy to volatile-lru (03master...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4141 [13:25:45] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/f0599356fcb8c63eac45438ef006e589c58429bc [13:25:45] 02puppet/03master 07paladox 03f059935 redis: set maxmemory_policy to volatile-lru (#4141) [13:25:46] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 03c9a19db 13https://github.com/miraheze/puppet/commit/c9a19db [13:27:21] !log increase rdb151 ram to 4gib [13:31:41] !log increase rdb151 ram to 4gib [13:31:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:32:30] sometimes, you just really gotta log to confirm [13:58:30] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.42', '1.43', '1.44'], 'upgrade_extensions': 'CreateWiki'} to test151 [13:58:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:01:32] !log [paladox@test151] finished deploy of {'l10n': True, 'versions': ['1.42', '1.43', '1.44'], 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 182s [14:01:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:59:44] miraheze/CreateWiki - paladox the build passed. [15:48:26] !log run dist-upgrade on rdb151 and reboot [15:48:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:51:45] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 18.07, 11.19, 6.21 [15:52:09] PROBLEM - rdb151 NTP time on rdb151 is WARNING: NTP WARNING: Offset -0.3674421608 secs [15:53:28] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/d7ffb3224463 [15:53:28] 02puppet/03paladox-patch-2 07paladox 03d7ffb32 changeprop151: set redis heap to 3g and set policy to volatile-lru [15:53:32] [02puppet] 07paladox opened pull request #4142: changeprop151: set redis heap to 3g and set policy to volatile-lru (03master...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4142 [15:53:39] [02puppet] 07coderabbitai[bot] commented on pull request #4142: --- […] 13https://github.com/miraheze/puppet/pull/4142#issuecomment-2605110194 [15:53:45] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 5.77, 9.42, 6.21 [15:53:53] [02puppet] 07paladox merged pull request #4142: changeprop151: set redis heap to 3g and set policy to volatile-lru (03master...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4142 [15:53:53] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/43f0fae8672e47d353cb25a7ac735603bb6ff5ad [15:53:53] 02puppet/03master 07paladox 0343f0fae changeprop151: set redis heap to 3g and set policy to volatile-lru (#4142) [15:53:55] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 03d7ffb32 13https://github.com/miraheze/puppet/commit/d7ffb32 [15:54:10] RECOVERY - rdb151 NTP time on rdb151 is OK: NTP OK: Offset -8.681416512e-05 secs [15:54:28] !log run dist-upgrade on changeprop151 and reboot [15:54:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:55:35] !log run dist-upgrade on eventgate181 and reboot [15:55:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:56:09] !log run dist-upgrade on kafka181 and reboot [15:56:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:56:13] good morning tech team, I'm trying to add $wgMinervaNightMode['base'] = true; to localsettings.php to force dark mode on my default skin. How can I access this file for my wiki? [15:59:25] PROBLEM - eventgate181 eventgate on eventgate181 is CRITICAL: connect to address 10.0.18.147 and port 8192: Connection refused [15:59:48] SCIRON: all our config is in https://github.com/miraheze/mw-config. You can open a task if you need tech's help. [15:59:49] [GitHub] Not Found [15:59:52] PROBLEM - changeprop151 changeprop on changeprop151 is CRITICAL: connect to address 10.0.15.148 and port 7200: Connection refused [16:00:14] PROBLEM - kafka181 SSH on kafka181 is CRITICAL: connect to address 10.0.18.146 and port 22: Connection refused [16:00:27] PROBLEM - kafka181 ferm_active on kafka181 is CRITICAL: connect to address 10.0.18.146 port 5666: No route to hostconnect to host 10.0.18.146 port 5666: No route to host [16:00:30] PROBLEM - kafka181 Puppet on kafka181 is CRITICAL: connect to address 10.0.18.146 port 5666: No route to hostconnect to host 10.0.18.146 port 5666: No route to host [16:00:30] PROBLEM - kafka181 Kafka Broker Server on kafka181 is CRITICAL: connect to address 10.0.18.146 port 5666: No route to hostconnect to host 10.0.18.146 port 5666: No route to host [16:00:34] PROBLEM - kafka181 NTP time on kafka181 is CRITICAL: connect to address 10.0.18.146 port 5666: No route to hostconnect to host 10.0.18.146 port 5666: No route to host [16:00:59] Err [16:01:04] That's not supposed to happen [16:01:23] RECOVERY - eventgate181 eventgate on eventgate181 is OK: TCP OK - 0.000 second response time on 10.0.18.147 port 8192 [16:01:29] PROBLEM - kafka181 Disk Space on kafka181 is CRITICAL: connect to address 10.0.18.146 port 5666: No route to hostconnect to host 10.0.18.146 port 5666: No route to host [16:01:29] PROBLEM - ping on kafka181 is CRITICAL: CRITICAL - Host Unreachable (10.0.18.146) [16:01:29] PROBLEM - Host kafka181 is DOWN: CRITICAL - Host Unreachable (10.0.18.146) [16:01:52] RECOVERY - changeprop151 changeprop on changeprop151 is OK: TCP OK - 0.000 second response time on 10.0.15.148 port 7200 [16:03:01] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/9abd3e909839b12e9a7f2ab0aeb8d164f0f89055 [16:03:01] 02puppet/03master 07paladox 039abd3e9 changeprop151: increase heap to 7gb [16:03:26] RECOVERY - Host kafka181 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [16:03:27] RECOVERY - kafka181 Disk Space on kafka181 is OK: DISK OK - free space: / 15992MiB (56% inode=95%); [16:03:31] RECOVERY - ping on kafka181 is OK: PING OK - Packet loss = 0%, RTA = 0.28 ms [16:04:01] RECOVERY - kafka181 Kafka Broker Server on kafka181 is OK: PROCS OK: 1 process with command name 'java', args 'Kafka /opt/kafka/config/server.properties' [16:04:08] RECOVERY - kafka181 Puppet on kafka181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:04:08] RECOVERY - kafka181 SSH on kafka181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u4 (protocol 2.0) [16:04:28] RECOVERY - kafka181 ferm_active on kafka181 is OK: OK ferm input default policy is set [16:04:43] RECOVERY - kafka181 NTP time on kafka181 is OK: NTP OK: Offset 4.321336746e-06 secs [16:06:59] [02mediawiki-repos] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/mediawiki-repos/commit/92566fbe33f8271ada5ff833022c588dbe038a88 [16:07:01] 02mediawiki-repos/03master 07paladox 0392566fb Use master branch for NumberHeadings… [16:07:13] PROBLEM - isocasg.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'isocasg.org' expires in 7 day(s) (Wed 29 Jan 2025 03:51:53 PM GMT +0000). [16:07:42] Ain't that a CF wiki [16:08:38] https://github.com/miraheze/ssl/blob/36d422d52dd98603cd5017403680b2583ceea1bf/certs.yaml#L2040 [16:08:38] [GitHub] [miraheze/ssl] certs.yaml @ 36d422d52dd98603cd5017403680b2583ceea1bf | L2040: ca: 'LetsEncrypt' [16:08:39] no [16:09:01] It's not [16:09:06] But that's not how you check [16:09:20] Can you have a look at why it won't renew @paladox? [16:45:03] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/edc7b30fdf3e116c30895724b5d9d0ba8dc17705 [16:45:03] 02ssl/03master 07MacFan4000 03edc7b30 add backroomszh.wiki [17:29:10] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:29:38] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [19:24:12] [02CreateWiki] 07AgentIsai created 03ai-improvements from 03master (+0 new commit) 13https://github.com/miraheze/CreateWiki/compare/ai-improvements [19:24:40] [02CreateWiki] 07AgentIsai pushed 1 new commit to 03ai-improvements 13https://github.com/miraheze/CreateWiki/commit/4bb901ff2451b90e364bb1e2550c428d61d74af6 [19:24:40] 02CreateWiki/03ai-improvements 07Agent Isai 034bb901f Introduce number of improvements to process [19:29:18] [02CreateWiki] 07AgentIsai opened pull request #651: Introduce improvements to AI process (03master...03ai-improvements) 13https://github.com/miraheze/CreateWiki/pull/651 [19:29:26] [02CreateWiki] 07coderabbitai[bot] commented on pull request #651: --- […] 13https://github.com/miraheze/CreateWiki/pull/651#issuecomment-2605568689 [19:30:12] [02CreateWiki] 07AgentIsai left a review on pull request #651: @coderabbitai review 13https://github.com/miraheze/CreateWiki/pull/651#pullrequestreview-2565524228 [19:31:02] [02CreateWiki] 07AgentIsai commented on pull request #651: @coderabbitai review 13https://github.com/miraheze/CreateWiki/pull/651#issuecomment-2605571656 [19:31:05] [02CreateWiki] 07github-actions[bot] pushed 1 new commit to 03ai-improvements 13https://github.com/miraheze/CreateWiki/commit/fd920222516a0eb6946fb6e414e9ff14d701731c [19:31:05] 02CreateWiki/03ai-improvements 07github-actions 03fd92022 CI: lint code to MediaWiki standards… [19:31:09] [02CreateWiki] 07coderabbitai[bot] commented on pull request #651:

[…] 13https://github.com/miraheze/CreateWiki/pull/651#issuecomment-2605571836 [19:32:34] miraheze/CreateWiki - AgentIsai the build has errored. [19:34:08] [02CreateWiki] 07coderabbitai[bot] left a review on pull request #651: **Actionable comments posted: 7** […] 13https://github.com/miraheze/CreateWiki/pull/651#pullrequestreview-2565531168 [19:34:08] [02CreateWiki] 07coderabbitai[bot] left a file comment in pull request #651 03fd92022: _:warning: Potential issue_ […] 13https://github.com/miraheze/CreateWiki/pull/651#discussion_r1924267694 [19:34:10] [02CreateWiki] 07coderabbitai[bot] left a file comment in pull request #651 03fd92022: _:warning: Potential issue_ […] 13https://github.com/miraheze/CreateWiki/pull/651#discussion_r1924267700 [19:34:12] [02CreateWiki] 07coderabbitai[bot] left a file comment in pull request #651 03fd92022: _:warning: Potential issue_ […] 13https://github.com/miraheze/CreateWiki/pull/651#discussion_r1924267711 [19:34:14] [02CreateWiki] 07coderabbitai[bot] left a file comment in pull request #651 03fd92022: _:warning: Potential issue_ […] 13https://github.com/miraheze/CreateWiki/pull/651#discussion_r1924267702 [19:34:16] [02CreateWiki] 07coderabbitai[bot] left a file comment in pull request #651 03fd92022: _:warning: Potential issue_ […] 13https://github.com/miraheze/CreateWiki/pull/651#discussion_r1924267713 [19:34:18] [02CreateWiki] 07coderabbitai[bot] left a file comment in pull request #651 03fd92022: _:warning: Potential issue_ […] 13https://github.com/miraheze/CreateWiki/pull/651#discussion_r1924267698 [19:34:20] [02CreateWiki] 07coderabbitai[bot] left a file comment in pull request #651 03fd92022: _:warning: Potential issue_ […] 13https://github.com/miraheze/CreateWiki/pull/651#discussion_r1924267717 [19:45:36] [02CreateWiki] 07AgentIsai pushed 1 new commit to 03ai-improvements 13https://github.com/miraheze/CreateWiki/commit/8e63f30f5cd20f44dc610eb2e6a9cb35dd847a28 [19:45:37] 02CreateWiki/03ai-improvements 07Agent Isai 038e63f30 Update [19:47:43] [02CreateWiki] 07github-actions[bot] pushed 1 new commit to 03ai-improvements 13https://github.com/miraheze/CreateWiki/commit/67d3d6458590e503787e3c2646e3c34ddb0b59d3 [19:47:43] 02CreateWiki/03ai-improvements 07github-actions 0367d3d64 CI: lint code to MediaWiki standards… [19:49:05] miraheze/CreateWiki - AgentIsai the build has errored. [20:35:11] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/1b0cf081a9e603db0158eaee20ec8a95a4311091 [20:35:11] 02ssl/03master 07MacFan4000 031b0cf08 Update certs.yaml [20:46:06] PROBLEM - backrooms.sytes.net - Cloudflare on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.803BC9C313150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1605:SSL alert number 40 [22:47:06] PROBLEM - swiftobject161 Disk Space on swiftobject161 is CRITICAL: DISK CRITICAL - free space: / 80105MiB (5% inode=84%); [23:20:04] PROBLEM - changeprop151 Current Load on changeprop151 is WARNING: LOAD WARNING - total load average: 5.93, 14.58, 9.75 [23:22:04] RECOVERY - changeprop151 Current Load on changeprop151 is OK: LOAD OK - total load average: 1.44, 10.15, 8.73