[00:01:27] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/62cb98f9a32b95bc645c1c64d8ceffb2156dfba1 [00:01:27] 02statichelp/03main 07WikiTideBot 0362cb98f Bot: Auto-update Tech namespace pages 2026-09-05 00:01:24 [00:08:50] miraheze/CreateWiki - lihaohong6 the build passed. [00:08:54] miraheze/CreateWiki - lihaohong6 the build passed. [00:16:40] RECOVERY - db161 mariadb_memory on db161 is OK: OK Memory 12% used [00:17:31] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [00:17:38] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 14 backends are down. mw151 mw152 mw162 mw171 mw172 mw182 mw153 mw183 mw191 mw192 mw193 mw201 mw202 mw203 [00:18:15] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [00:18:18] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.010 second response time [00:18:19] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [00:18:25] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [00:18:48] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 14 backends are down. mw151 mw152 mw162 mw171 mw172 mw182 mw153 mw183 mw191 mw192 mw193 mw201 mw202 mw203 [00:18:51] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [00:19:03] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 400.99, 165.53, 63.77 [00:19:13] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.012 second response time [00:19:15] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.010 second response time [00:19:22] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [00:19:35] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 14 backends are down. mw151 mw152 mw162 mw171 mw172 mw182 mw153 mw183 mw191 mw192 mw193 mw201 mw202 mw203 [00:20:19] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [00:20:21] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [00:20:24] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [00:20:24] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [00:20:39] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:42] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:20:42] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [00:20:46] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:46] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:47] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:47] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:48] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:20:48] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:49] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:51] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:20:51] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:52] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [00:20:54] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [00:20:54] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [00:20:57] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:20:57] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [00:20:57] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [00:20:57] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:58] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:21:00] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:21:01] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [00:21:03] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [00:21:04] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:21:04] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:21:06] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [00:21:07] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [00:21:08] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:21:08] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 14 backends are down. mw151 mw152 mw162 mw171 mw172 mw182 mw153 mw183 mw191 mw192 mw193 mw201 mw202 mw203 [00:21:43] !log [petramagna@mwtask181] starting deploy of {'config': True, 'force': True} to all [00:22:01] !log [petramagna@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 18s [00:23:03] okay [00:23:07] PROBLEM - db161 MariaDB Connections on db161 is CRITICAL: Critical connection usage: 91.4%Current connections: 914 [00:23:23] ? [00:24:13] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.196 second response time [00:24:13] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.629 second response time [00:24:17] !log [petramagna@mwtask181] starting deploy of {'config': True, 'force': True} to all [00:24:20] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.084 second response time [00:24:22] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1518 bytes in 4.072 second response time [00:24:25] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.086 second response time [00:24:25] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.088 second response time [00:24:29] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 4.424 second response time [00:24:37] !log [petramagna@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 20s [00:24:41] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.086 second response time [00:24:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:24:42] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.080 second response time [00:24:45] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.211 second response time [00:24:48] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 32 backends are healthy [00:24:48] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 30% [00:24:49] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.209 second response time [00:24:50] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.080 second response time [00:24:51] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.222 second response time [00:24:51] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.218 second response time [00:24:51] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.086 second response time [00:24:53] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.299 second response time [00:24:53] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.225 second response time [00:24:54] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.074 second response time [00:24:54] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.222 second response time [00:24:54] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.065 second response time [00:24:57] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.246 second response time [00:24:57] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.078 second response time [00:24:57] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.231 second response time [00:24:59] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.078 second response time [00:24:59] There we go. Somehow my first deploy didn't go through. [00:25:03] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.254 second response time [00:25:04] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.208 second response time [00:25:04] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.082 second response time [00:25:04] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.219 second response time [00:25:05] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.081 second response time [00:25:05] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.085 second response time [00:25:06] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.223 second response time [00:25:07] RECOVERY - db161 MariaDB Connections on db161 is OK: OK connection usage: 0.2%Current connections: 2 [00:25:07] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.074 second response time [00:25:08] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.236 second response time [00:25:08] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 32 backends are healthy [00:25:09] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.236 second response time [00:25:11] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.080 second response time [00:25:13] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.094 second response time [00:25:14] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.076 second response time [00:25:27] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.090 second response time [00:25:33] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.237 second response time [00:25:35] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 32 backends are healthy [00:25:38] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 32 backends are healthy [00:26:06] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.077 second response time [00:29:46] Repooling db161 now that load is down [00:29:56] !log [petramagna@mwtask181] starting deploy of {'config': True} to all [00:29:57] !log [petramagna@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [00:29:59] !log [petramagna@mwtask181] starting deploy of {'config': True, 'force': True} to all [00:30:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:30:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:30:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:30:18] !log [petramagna@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 18s [00:30:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:28] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 45185MiB (10.0% inode=100%); [00:32:29] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 45316MiB (10.0% inode=100%); [00:36:06] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 44390MiB (9.8% inode=100%); [00:40:06] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 86789MiB (19.1% inode=100%); [00:44:28] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 81804MiB (18.0% inode=100%); [01:12:29] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 81609MiB (18.0% inode=100%); [01:17:03] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 1.39, 1.80, 11.76 [01:21:03] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 1.50, 1.56, 9.37 [02:18:57] !log upgraded EmbedVideo for T15975 [02:19:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:30:20] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/416f53c0e47d3c3570f2828d95ad8c2c1db52424 [03:30:21] 02ssl/03main 07WikiTideBot 03416f53c Bot: Auto-update domain lists [04:13:05] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.26, 12.98, 11.58 [04:15:05] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 11.88, 12.71, 11.66 [05:30:19] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/6d1a91a4fe0bb40074b27142e1ad3f1b66c74661 [05:30:19] 02ssl/03main 07WikiTideBot 036d1a91a Bot: Auto-update domain lists [06:49:28] [02CreateWiki] 07lihaohong6 left a file comment in pull request #852 033ac0db5: I changed things around. The downside is that we no longer get all the `

` entries inside a single `

miraheze/CreateWiki - lihaohong6 the build passed. [11:59:03] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 48.27, 21.18, 9.23 [12:00:48] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 5 backends are down. mw162 mw182 mw183 mw192 mw193 [12:01:03] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [12:01:07] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [12:01:08] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 13 backends are down. mw151 mw162 mw171 mw172 mw182 mw153 mw183 mw191 mw192 mw193 mw201 mw202 mw203 [12:01:22] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:01:23] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.010 second response time [12:01:24] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:01:27] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:01:31] depooling db161 [12:01:35] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 9 backends are down. mw151 mw152 mw171 mw172 mw182 mw153 mw191 mw201 mw202 [12:01:38] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 9 backends are down. mw152 mw162 mw172 mw182 mw153 mw191 mw201 mw202 mw203 [12:02:03] !log [petramagna@mwtask181] starting deploy of {'config': True, 'force': True} to all [12:02:06] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [12:02:18] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:23] !log [petramagna@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 20s [12:02:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:02:59] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 43% [12:03:01] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.091 second response time [12:03:03] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.267 second response time [12:03:07] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [12:03:08] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 32 backends are healthy [12:03:18] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.089 second response time [12:03:20] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.222 second response time [12:03:24] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.081 second response time [12:03:27] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.087 second response time [12:03:35] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 32 backends are healthy [12:03:38] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 32 backends are healthy [12:04:05] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.236 second response time [12:04:15] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.226 second response time [12:04:39] [1/2] @Infrastructure Specialists db161 still down. I'll repool once CPU utilization goes below 10%. Since it happened 2 days in a row we should grab the proclist again. [12:04:40] [2/2] I suspect that they were caused by different things because the CPU user/system split was different last time. [12:04:48] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 32 backends are healthy [12:04:58] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 37% [12:12:18] !log [petramagna@mwtask181] starting deploy of {'config': True, 'force': True} to all [12:12:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:12:27] Repooling db161 [12:12:38] !log [petramagna@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 19s [12:12:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:16:55] !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php MirahezeMagic:AssignImportedEdits --wiki=doorswiki --from=Noobyguy775 --to=Noobyguy775 --import-prefix=doorsgame (END - exit=0) [12:16:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:17:06] !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initEditCount --wiki=doorswiki (END - exit=0) [12:17:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:21:03] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 1.38, 4.22, 11.96 [12:25:03] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 2.11, 3.03, 9.70 [12:30:19] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/a8c3d0786f4e543b060e4f33417ac1b4742192fd [12:30:19] 02ssl/03main 07WikiTideBot 03a8c3d07 Bot: Auto-update domain lists [12:31:07] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [12:58:25] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.08, 13.65, 13.28 [12:58:40] PROBLEM - db161 mariadb_memory on db161 is WARNING: WARN Memory 90% used. Largest process: mariadbd (1770829) = 88.2% [13:06:23] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.06, 14.85, 13.97 [13:08:22] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 15.24, 14.91, 14.10 [13:14:20] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 10.59, 12.87, 13.54 [13:45:10] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.04, 13.41, 12.86 [13:47:09] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 12.38, 13.13, 12.83 [14:25:05] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 15.84, 14.46, 13.24 [14:27:05] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.15, 15.02, 13.60 [14:30:21] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/5881850daf577ee88fff477c6c0cab7d8eb0aabd [14:30:21] 02ssl/03main 07WikiTideBot 035881850 Bot: Auto-update domain lists [14:31:05] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.99, 15.27, 14.08 [14:35:05] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 10.82, 13.28, 13.56 [14:44:03] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.85, 13.85, 13.63 [14:52:00] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 11.32, 12.88, 13.45 [15:08:05] PROBLEM - matomo151 Puppet on matomo151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 0 failures [15:10:09] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.45, 13.23, 12.73 [15:14:00] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 12.45, 13.14, 12.84 [15:41:52] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.22, 13.11, 12.38 [15:43:48] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 13.21, 13.06, 12.44 [16:23:19] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.83, 13.28, 12.62 [16:25:19] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 12.10, 12.91, 12.58 [16:30:19] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/2ebe6160d44f0fc64ba4fbeb58fb4751782fe903 [16:30:19] 02ssl/03main 07WikiTideBot 032ebe616 Bot: Auto-update domain lists [17:02:50] [02MirahezeRequests] 07Reception123 pushed 1 new commit to 03requestaccount 13https://github.com/miraheze/MirahezeRequests/commit/fd08c9d7dda76981de7ede665c2b4ef482ffc185 [17:02:50] 02MirahezeRequests/03requestaccount 07Reception123 03fd08c9d Update en.json [17:23:28] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.01, 13.12, 12.37 [17:29:19] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 11.46, 12.91, 12.60 [17:53:09] [02CreateWiki] 07Universal-Omega left a review on pull request #852: Just some comments, not really requesting changes yet for these, though the one about the message is probably the one one immediately actionable if it works properly. 13https://github.com/miraheze/CreateWiki/pull/852#pullrequestreview-5122110784 [17:53:09] [02CreateWiki] 07Universal-Omega left a file comment in pull request #852 034af185a: This can be simplified more *I think*: […] 13https://github.com/miraheze/CreateWiki/pull/852#discussion_r3941321666 [17:53:09] [02CreateWiki] 07Universal-Omega left a file comment in pull request #852 034af185a: This can probably move to CreateWikiValidator, alongside isEnabled -> isLoadoutEnabled or something, and getLoadoutNames then inject CreateWikiValidator. Only do this if doing the comment I mentioned below also, because my main goal is to reduce what […] 13https://github.com/miraheze/CreateWiki/pull/852#discussion_r3941415902 [17:53:09] [02CreateWiki] 07Universal-Omega left a file comment in pull request #852 034af185a: I am little worried about this. I don't believe we should have to keep the XML on the disk permanently, because we use swift for file storage. My thought is we do like we do for the persistent model: […] 13https://github.com/miraheze/CreateWiki/pull/852#discussion_r3941485088 [17:53:11] [02CreateWiki] 07Universal-Omega left a file comment in pull request #852 034af185a: Feel free to completely ignore this. A lot of it is complicated, I don't think we should yet but for a future cleanup later which is why I still posted this. 13https://github.com/miraheze/CreateWiki/pull/852#discussion_r3941457211 [18:13:31] PROBLEM - db161 mariadb_memory on db161 is CRITICAL: CRIT Memory 95% used. Largest process: mariadbd (1770829) = 93.1% [18:15:31] PROBLEM - db161 mariadb_memory on db161 is WARNING: WARN Memory 94% used. Largest process: mariadbd (1770829) = 93.1% [18:17:31] PROBLEM - db161 mariadb_memory on db161 is CRITICAL: CRIT Memory 95% used. Largest process: mariadbd (1770829) = 93.2% [18:25:19] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.10, 13.34, 12.61 [18:37:19] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 12.85, 13.24, 13.05 [18:44:11] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.86, 13.62, 13.29 [18:46:06] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 13.50, 13.55, 13.30 [18:54:29] [02ManageWiki] 07Universal-Omega created 03cache-reset (+1 new commit) 13https://github.com/miraheze/ManageWiki/commit/3de8b15260b5 [18:54:29] 02ManageWiki/03cache-reset 07CosmicAlpha 033de8b15 Fix cache reset [18:54:35] [02ManageWiki] 07Universal-Omega opened pull request #812: Fix cache reset (03main...03cache-reset) 13https://github.com/miraheze/ManageWiki/pull/812 [18:54:47] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.62, 13.73, 13.51 [18:55:14] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03cache-reset 13https://github.com/miraheze/ManageWiki/commit/803b2ffce2c6bd0d0665dc8a13216b20205f29d4 [18:55:15] 02ManageWiki/03cache-reset 07CosmicAlpha 03803b2ff Update [18:58:03] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03cache-reset 13https://github.com/miraheze/ManageWiki/commit/3507d2b69cd41d79c78235f823024b4b97aaaf0b [18:58:03] 02ManageWiki/03cache-reset 07CosmicAlpha 033507d2b Add [18:58:58] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03cache-reset 13https://github.com/miraheze/ManageWiki/commit/ffd46ffe3300fa27e4de24fe62ce18875eb4ce17 [18:58:58] 02ManageWiki/03cache-reset 07CosmicAlpha 03ffd46ff Add [18:59:32] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03cache-reset 13https://github.com/miraheze/ManageWiki/commit/956f71b5558ea8e60a8d8a741f758e0aa99677d8 [18:59:32] 02ManageWiki/03cache-reset 07CosmicAlpha 03956f71b Update [19:00:32] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 13.21, 13.43, 13.47 [19:03:15] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03cache-reset 13https://github.com/miraheze/ManageWiki/commit/6470d2c408fb107ac9f8f6bf159e1dfaeb221638 [19:03:15] 02ManageWiki/03cache-reset 07CosmicAlpha 036470d2c Add [19:05:39] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03cache-reset 13https://github.com/miraheze/ManageWiki/commit/4314e536f02c8b4de336fb8852e7de86a0b012ee [19:05:39] 02ManageWiki/03cache-reset 07CosmicAlpha 034314e53 Add [19:11:00] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03cache-reset 13https://github.com/miraheze/ManageWiki/commit/c98067abd293c8e42429e734c032f718e5b7f4e3 [19:11:00] 02ManageWiki/03cache-reset 07CosmicAlpha 03c98067a Update [19:11:02] miraheze/ManageWiki - Universal-Omega the build passed. [19:12:05] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.07, 13.72, 13.48 [19:12:15] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03cache-reset 13https://github.com/miraheze/ManageWiki/commit/868e9a3c9802fa4f73f5951453b03ae2bd4127ab [19:12:15] 02ManageWiki/03cache-reset 07CosmicAlpha 03868e9a3 Add [19:15:19] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03cache-reset 13https://github.com/miraheze/ManageWiki/commit/cc5934c8482e0bca7a9f8dcfe5c280479f80df36 [19:15:19] 02ManageWiki/03cache-reset 07CosmicAlpha 03cc5934c Add [19:20:14] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03cache-reset 13https://github.com/miraheze/ManageWiki/commit/d4797f48966c3329fd17d3677c440a67e9762eb5 [19:20:14] 02ManageWiki/03cache-reset 07CosmicAlpha 03d4797f4 Don't do twice on one server [19:20:26] miraheze/ManageWiki - Universal-Omega the build passed. [19:25:43] miraheze/ManageWiki - Universal-Omega the build passed. [19:51:38] [02dns] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/dns/commit/b347841f49ce [19:51:38] 02dns/03paladox-patch-1 07paladox 03b347841 Add matomo211 [19:51:42] [02dns] 07paladox opened pull request #714: Add matomo211 (03main...03paladox-patch-1) 13https://github.com/miraheze/dns/pull/714 [19:52:32] miraheze/dns - paladox the build passed. [19:52:55] [02dns] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/dns/commit/ba0a7920c8a3c7ad4078216abcf20cd297a388fd [19:52:55] 02dns/03paladox-patch-1 07paladox 03ba0a792 Add PTR records for speedscope211 and matomo211 [19:53:14] [02dns] 07paladox merged pull request #714: Add matomo211 (03main...03paladox-patch-1) 13https://github.com/miraheze/dns/pull/714 [19:53:15] [02dns] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/dac012706050b90b3d6ea5aed93e434507f697c5 [19:53:15] 02dns/03main 07paladox 03dac0127 Add matomo211 (#714) [19:53:16] [02dns] 07paladox 04deleted 03paladox-patch-1 at 03ba0a792 13https://api.github.com/repos/miraheze/dns/commit/ba0a792 [19:53:52] miraheze/dns - paladox the build passed. [19:54:04] miraheze/dns - paladox the build passed. [20:34:00] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/2184fd92fd4d [20:34:00] 02puppet/03paladox-patch-2 07paladox 032184fd9 matomo: set proxy config [20:34:04] [02puppet] 07paladox opened pull request #5035: matomo: set proxy config (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/5035 [20:34:52] [02puppet] 07paladox merged pull request #5035: matomo: set proxy config (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/5035 [20:34:54] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/631f1aec82317bf7ea3f2123eb9a876fcbe9311d [20:34:55] 02puppet/03main 07paladox 03631f1ae matomo: set proxy config (#5035) [20:34:57] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 032184fd9 13https://api.github.com/repos/miraheze/puppet/commit/2184fd9 [20:36:47] PROBLEM - matomo211 chrony on matomo211 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.21.173: Connection reset by peer [20:36:49] PROBLEM - matomo211 Disk Space on matomo211 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.21.173: Connection reset by peer [20:36:51] PROBLEM - matomo211 PowerDNS Recursor on matomo211 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.21.173: Connection reset by peer [20:36:58] PROBLEM - matomo211 nftables_active on matomo211 is UNKNOWN: NRPE: Unable to read output [20:37:01] PROBLEM - matomo211 conntrack_table_size on matomo211 is UNKNOWN: NRPE: Unable to read output [20:37:12] PROBLEM - matomo211 Puppet on matomo211 is UNKNOWN: NRPE: Unable to read output [20:37:50] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/2f30a1ab3268 [20:37:50] 02puppet/03paladox-patch-2 07paladox 032f30a1a Allow matomos to connect to redis… [20:38:06] [02puppet] 07paladox opened pull request #5036: Allow matomos to connect to redis (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/5036 [20:38:18] [02puppet] 07paladox merged pull request #5036: Allow matomos to connect to redis (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/5036 [20:38:20] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/03f8a1250656d3abc93f78b1f8e182c8ab44f773 [20:38:22] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 032f30a1a 13https://api.github.com/repos/miraheze/puppet/commit/2f30a1a [20:38:23] 02puppet/03main 07paladox 0303f8a12 Allow matomos to connect to redis (#5036)… [20:38:44] PROBLEM - matomo211 chrony on matomo211 is WARNING: chrony stratum too high: 10 > 5 [20:38:44] RECOVERY - matomo211 Disk Space on matomo211 is OK: DISK OK - free space: / 23692MiB (90.7% inode=95%); [20:38:53] RECOVERY - matomo211 nftables_active on matomo211 is OK: OK nftables input default policy is set [20:38:56] RECOVERY - matomo211 conntrack_table_size on matomo211 is OK: OK: nf_conntrack is 0 % full [20:39:01] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/8b65bedbee80 [20:39:01] 02puppet/03paladox-patch-2 07paladox 038b65bed Install matomo211 [20:39:06] [02puppet] 07paladox opened pull request #5037: Install matomo211 (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/5037 [20:39:10] RECOVERY - matomo211 Puppet on matomo211 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [20:40:59] [02puppet] 07Universal-Omega requested changes on pull request #5037 13https://github.com/miraheze/puppet/pull/5037#pullrequestreview-5122900465 [20:41:01] [02puppet] 07Universal-Omega left a file comment in pull request #5037 038b65bed: Needs: […] 13https://github.com/miraheze/puppet/pull/5037#discussion_r3941990745 [20:41:48] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/a59c31911bd6aa09496b7aea25026764a7498bde [20:41:48] 02puppet/03paladox-patch-2 07paladox 03a59c319 Add enable_redis parameter to matomo role [20:41:58] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/5d203fe4cf316c8424d1733e4ce3d5b4f6e9fac8 [20:41:58] 02puppet/03paladox-patch-2 07paladox 035d203fe Update matomo151.yaml [20:42:16] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/bc97538ec38bc51c9f0056ebaab76310004431df [20:42:16] 02puppet/03paladox-patch-2 07paladox 03bc97538 Update matomo211.yaml [20:42:25] [02puppet] 07paladox left a file comment in pull request #5037 03bc97538: Fixed, thanks. 13https://github.com/miraheze/puppet/pull/5037#discussion_r3941994922 [20:42:41] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/d80a64dce28a72047cb7fde3a0ba175faecc30f5 [20:42:42] 02puppet/03paladox-patch-2 07paladox 03d80a64d Change type of enable_redis parameter to Boolean [20:42:52] [02puppet] 07Universal-Omega approved pull request #5037 13https://github.com/miraheze/puppet/pull/5037#pullrequestreview-5122904655 [20:43:13] [02puppet] 07paladox merged pull request #5037: Install matomo211 (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/5037 [20:43:13] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/6be987ef0d0e17942650b1ce4a740a3d78d545e7 [20:43:13] 02puppet/03main 07paladox 036be987e Install matomo211 (#5037) [20:43:14] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 03d80a64d 13https://api.github.com/repos/miraheze/puppet/commit/d80a64d [20:44:26] RECOVERY - matomo211 PowerDNS Recursor on matomo211 is OK: DNS OK: 0.025 seconds response time. matomo211.fsslc.wtnet returns 10.0.21.173 [20:44:26] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/3f29ec203342a24607ec977bcf3f25560820dd12 [20:44:26] 02puppet/03main 07paladox 033f29ec2 Whoops forgot to add matomo211 to site.pp [20:44:37] RECOVERY - matomo211 chrony on matomo211 is OK: chrony OK: System Time 0.000000002 seconds fast of NTP time, Stratum 3 [20:45:25] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/8700d9e83bd9b69582f7799b3b2e2e27ad82c1f8 [20:45:25] 02puppet/03main 07paladox 038700d9e Fix [20:47:07] PROBLEM - matomo211 Puppet on matomo211 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:47:24] RECOVERY - matomo151 Puppet on matomo151 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [20:49:03] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 10.57, 12.38, 13.32 [20:55:01] RECOVERY - matomo211 Puppet on matomo211 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:55:36] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/485a1f5acedc [20:55:36] 02puppet/03paladox-patch-2 07paladox 03485a1f5 matomo: Run 2 archivers and queuedtracking per server… [20:55:40] [02puppet] 07paladox opened pull request #5038: matomo: Run 2 archivers and queuedtracking per server (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/5038 [20:57:23] [02puppet] 07paladox merged pull request #5038: matomo: Run 2 archivers and queuedtracking per server (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/5038 [20:57:25] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/59a0078832d18c134bf0faf6c8f9270c66aff8d5 [20:57:27] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 03485a1f5 13https://api.github.com/repos/miraheze/puppet/commit/485a1f5 [20:57:29] 02puppet/03main 07paladox 0359a0078 matomo: Run 2 archivers and queuedtracking per server (#5038)… [21:00:23] [02TSPortal] 07renovate[bot] 04force-pushed 03renovate/prod-deps from 031d08108 to 03538338b 13https://api.github.com/repos/miraheze/TSPortal/commits/renovate/prod-deps [21:00:23] 02TSPortal/03renovate/prod-deps 07renovate[bot] 03538338b Update prod dependencies [21:00:39] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/27668dbbb431943a94bf67ed874523087003f8c5 [21:00:39] 02puppet/03main 07paladox 0327668db matomo: Enable 4th archiver [21:03:50] PROBLEM - matomo151 Check unit status of matomo-archiver-3 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-3 [21:03:51] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/3a37f8db3875 [21:03:51] 02puppet/03paladox-patch-2 07paladox 033a37f8d Add matomo211 to varnish [21:03:55] [02puppet] 07paladox opened pull request #5039: Add matomo211 to varnish (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/5039 [21:04:17] PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-1 [21:05:12] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/71e3f01617b2f24da16af3bcd32739ef55b54148 [21:05:14] 02puppet/03paladox-patch-2 07paladox 0371e3f01 Update print statement from 'Hello' to 'Goodbye' [21:06:07] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/701914366a195743f2371460e9dabc21a2db3d18 [21:06:07] 02puppet/03paladox-patch-2 07paladox 037019143 Update common.yaml [21:06:35] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/249b9c958c4b8c1f8e48e809cab8b4d7e840d6f6 [21:06:35] 02puppet/03paladox-patch-2 07paladox 03249b9c9 Update default.vcl [21:06:59] [02puppet] 07paladox merged pull request #5039: Add matomo211 to varnish (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/5039 [21:06:59] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/a14d91f53aa02d20c62744c10ce56aeb47e7f69a [21:06:59] 02puppet/03main 07paladox 03a14d91f Add matomo211 to varnish (#5039) [21:07:01] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 03249b9c9 13https://api.github.com/repos/miraheze/puppet/commit/249b9c9 [21:11:20] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. matomo [21:11:32] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/0723246f33e9cb5b197ec3e1753c4b7d1675c9f2 [21:11:32] 02puppet/03main 07paladox 030723246 Add matomopool configuration for matomo151 and matomo211 [21:11:39] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 1 backends are down. matomo [21:13:17] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 34 backends are healthy [21:14:27] PROBLEM - cp171 Haproxy TLS backend for mwtask161 on cp171 is CRITICAL: connect to address localhost and port 8163: Connection refused [21:14:27] PROBLEM - cp171 Nginx Backend for mw181 on cp171 is CRITICAL: connect to address localhost and port 8119: Connection refused [21:14:32] PROBLEM - cp171 Nginx Backend for mw163 on cp171 is CRITICAL: connect to address localhost and port 8123: Connection refused [21:14:32] PROBLEM - cp171 Nginx Backend for mw172 on cp171 is CRITICAL: connect to address localhost and port 8118: Connection refused [21:14:33] PROBLEM - cp171 Nginx Backend for mw152 on cp171 is CRITICAL: connect to address localhost and port 8114: Connection refused [21:14:35] PROBLEM - cp171 Haproxy TLS backend for mw161 on cp171 is CRITICAL: connect to address localhost and port 8115: Connection refused [21:14:36] PROBLEM - cp171 Haproxy TLS backend for swiftproxy161 on cp171 is CRITICAL: connect to address localhost and port 8206: Connection refused [21:14:36] PROBLEM - cp171 Haproxy TLS backend for swiftproxy171 on cp171 is CRITICAL: connect to address localhost and port 8207: Connection refused [21:14:37] PROBLEM - cp171 Nginx Backend for mw183 on cp171 is CRITICAL: connect to address localhost and port 8127: Connection refused [21:14:38] PROBLEM - cp171 Haproxy TLS backend for mw151 on cp171 is CRITICAL: connect to address localhost and port 8113: Connection refused [21:14:40] PROBLEM - cp171 Nginx Backend for reports171 on cp171 is CRITICAL: connect to address localhost and port 8205: Connection refused [21:14:44] PROBLEM - cp171 Nginx Backend for puppet181 on cp171 is CRITICAL: connect to address localhost and port 8204: Connection refused [21:14:44] PROBLEM - cp171 Nginx Backend for mw182 on cp171 is CRITICAL: connect to address localhost and port 8120: Connection refused [21:14:45] PROBLEM - cp171 Nginx Backend for mw193 on cp171 is CRITICAL: connect to address localhost and port 8131: Connection refused [21:14:48] PROBLEM - cp171 Haproxy TLS backend for mw173 on cp171 is CRITICAL: connect to address localhost and port 8125: Connection refused [21:14:49] PROBLEM - cp171 Haproxy TLS backend for speedscope211 on cp171 is CRITICAL: connect to address localhost and port 8208: Connection refused [21:14:51] PROBLEM - cp171 Haproxy TLS backend for test151 on cp171 is CRITICAL: connect to address localhost and port 8181: Connection refused [21:14:52] PROBLEM - cp171 Nginx Backend for test151 on cp171 is CRITICAL: connect to address localhost and port 8181: Connection refused [21:14:52] PROBLEM - cp171 Nginx Backend for mw201 on cp171 is CRITICAL: connect to address localhost and port 8132: Connection refused [21:14:53] PROBLEM - cp171 Haproxy TLS backend for mw163 on cp171 is CRITICAL: connect to address localhost and port 8123: Connection refused [21:14:53] PROBLEM - cp171 Nginx Backend for matomo151 on cp171 is CRITICAL: connect to address localhost and port 8203: Connection refused [21:14:53] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [21:14:55] PROBLEM - cp171 Haproxy TLS backend for mwtask181 on cp171 is CRITICAL: connect to address localhost and port 8160: Connection refused [21:14:57] PROBLEM - cp171 Haproxy TLS backend for mw181 on cp171 is CRITICAL: connect to address localhost and port 8119: Connection refused [21:14:58] PROBLEM - cp171 Nginx Backend for mw203 on cp171 is CRITICAL: connect to address localhost and port 8134: Connection refused [21:15:00] PROBLEM - cp171 Nginx Backend for mwtask151 on cp171 is CRITICAL: connect to address localhost and port 8162: Connection refused [21:15:00] PROBLEM - cp171 Haproxy TLS backend for mw153 on cp171 is CRITICAL: connect to address localhost and port 8121: Connection refused [21:15:07] PROBLEM - cp171 Haproxy TLS backend for puppet181 on cp171 is CRITICAL: connect to address localhost and port 8204: Connection refused [21:15:08] PROBLEM - cp171 Haproxy TLS backend for phorge171 on cp171 is CRITICAL: connect to address localhost and port 8202: Connection refused [21:15:10] PROBLEM - cp171 Haproxy TLS backend for mw193 on cp171 is CRITICAL: connect to address localhost and port 8131: Connection refused [21:15:10] PROBLEM - cp171 Haproxy TLS backend for mw202 on cp171 is CRITICAL: connect to address localhost and port 8133: Connection refused [21:15:14] PROBLEM - cp171 Haproxy TLS backend for mw171 on cp171 is CRITICAL: connect to address localhost and port 8117: Connection refused [21:15:16] PROBLEM - cp171 Nginx Backend for mw171 on cp171 is CRITICAL: connect to address localhost and port 8117: Connection refused [21:15:19] PROBLEM - cp171 Haproxy TLS backend for mw182 on cp171 is CRITICAL: connect to address localhost and port 8120: Connection refused [21:15:21] PROBLEM - cp171 Haproxy TLS backend for mw183 on cp171 is CRITICAL: connect to address localhost and port 8127: Connection refused [21:15:21] PROBLEM - cp171 Nginx Backend for mw151 on cp171 is CRITICAL: connect to address localhost and port 8113: Connection refused [21:15:21] PROBLEM - cp171 Nginx Backend for mw161 on cp171 is CRITICAL: connect to address localhost and port 8115: Connection refused [21:15:21] PROBLEM - cp171 Nginx Backend for mw192 on cp171 is CRITICAL: connect to address localhost and port 8130: Connection refused [21:15:22] PROBLEM - cp171 Nginx Backend for mw191 on cp171 is CRITICAL: connect to address localhost and port 8129: Connection refused [21:15:23] PROBLEM - cp171 Nginx Backend for mon181 on cp171 is CRITICAL: connect to address localhost and port 8201: Connection refused [21:15:25] PROBLEM - cp171 Nginx Backend for mwtask181 on cp171 is CRITICAL: connect to address localhost and port 8160: Connection refused [21:15:32] PROBLEM - cp171 Nginx Backend for phorge171 on cp171 is CRITICAL: connect to address localhost and port 8202: Connection refused [21:15:35]