[00:06:59] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 55608MiB (12% inode=99%); [00:08:35] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 28.56, 24.24, 21.72 [00:08:44] RECOVERY - cp38 Disk Space on cp38 is OK: DISK OK - free space: / 56067MiB (12% inode=99%); [00:16:20] miraheze/CreateWiki - Universal-Omega the build passed. [00:19:45] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.04, 6.69, 4.98 [00:21:45] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.68, 6.81, 5.25 [00:25:45] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 10.34, 7.44, 5.78 [00:27:45] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 2.43, 5.45, 5.25 [00:27:48] PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to matomo151.wikitide.net port 443 after 0 ms: Couldn't connect to server [00:40:11] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 18.12, 21.34, 23.61 [00:55:48] RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.218 second response time [01:01:55] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.85, 23.04, 21.79 [01:05:51] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.94, 23.71, 22.49 [01:11:47] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.22, 22.89, 22.31 [01:25:41] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.40, 21.79, 23.33 [01:35:41] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 14.01, 16.41, 20.01 [01:56:33] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.72, 21.54, 19.86 [02:57:47] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 9.62, 18.75, 23.08 [03:01:44] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 6.75, 12.13, 19.36 [04:15:29] miraheze/dns - MacFan4000 the build passed. [04:17:54] miraheze/dns - MacFan4000 the build passed. [05:43:47] miraheze/ManageWiki - Universal-Omega the build passed. [05:51:44] PROBLEM - cp38 Disk Space on cp38 is WARNING: DISK WARNING - free space: / 49921MiB (10% inode=99%); [06:15:09] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49880MiB (10% inode=99%); [06:15:51] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.27, 21.16, 16.62 [06:17:51] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.40, 21.60, 17.35 [06:21:51] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.33, 23.13, 18.93 [06:25:51] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.06, 22.95, 19.87 [06:27:51] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.23, 23.26, 20.35 [06:29:51] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.27, 22.71, 20.49 [06:33:51] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.14, 23.35, 21.18 [06:37:51] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.08, 23.52, 21.91 [06:41:51] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.01, 23.53, 22.22 [06:43:51] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.36, 23.03, 22.21 [06:53:51] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 13.28, 17.65, 20.12 [09:07:51] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.92, 19.40, 14.16 [09:09:51] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 14.92, 17.61, 14.15 [11:05:33] PROBLEM - ravendawn.wiki.br - Cloudflare on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [11:47:51] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.89, 16.19, 10.67 [11:49:51] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.89, 17.40, 11.81 [11:58:43] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.84, 20.37, 15.50 [12:00:40] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 15.45, 18.43, 15.35 [12:48:49] [02puppet] 07paladox created 03paladox-patch-9 (+1 new commit) 13https://github.com/miraheze/puppet/commit/ea6b06d8072f [12:48:49] 02puppet/03paladox-patch-9 07paladox 03ea6b06d mediawiki set fastcgi_send_timeout and fastcgi_read_timeout to 202 [12:48:56] [02puppet] 07paladox opened pull request #4334: mediawiki set fastcgi_send_timeout and fastcgi_read_timeout to 202 (03main...03paladox-patch-9) 13https://github.com/miraheze/puppet/pull/4334 [12:49:03] [02puppet] 07coderabbitai[bot] commented on pull request #4334: --- […] 13https://github.com/miraheze/puppet/pull/4334#issuecomment-2804943065 [13:34:30] [02mw-config] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/386a14e3275f [13:34:30] 02mw-config/03paladox-patch-1 07paladox 03386a14e Fix timeouts for jobrunner.wikitide.net [13:34:36] [02mw-config] 07paladox opened pull request #5908: Fix timeouts for jobrunner.wikitide.net (03main...03paladox-patch-1) 13https://github.com/miraheze/mw-config/pull/5908 [13:35:07] [02mw-config] 07github-actions[bot] pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/mw-config/commit/e97e8281c97a78bc0edf3a1ef14cfd0d7c28b16f [13:35:09] 02mw-config/03paladox-patch-1 07github-actions 03e97e828 CI: lint code to MediaWiki standards… [13:35:35] miraheze/mw-config - paladox the build passed. [13:36:45] [02mw-config] 07paladox closed pull request #5908: Fix timeouts for jobrunner.wikitide.net (03main...03paladox-patch-1) 13https://github.com/miraheze/mw-config/pull/5908 [15:15:28] PROBLEM - cp38 Puppet on cp38 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[haproxy] [15:16:29] PROBLEM - cp37 Puppet on cp37 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[haproxy] [15:17:28] RECOVERY - cp38 Puppet on cp38 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [15:18:29] RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:44:19] [02mw-config] 07Aeywoo opened pull request #5909: Updating LocalWiki.php settings for Kaiserreich Wiki per T13504 (07miraheze:03main...07Aeywoo:03patch-6) 13https://github.com/miraheze/mw-config/pull/5909 [15:45:15] miraheze/mw-config - Aeywoo the build passed. [15:46:56] miraheze/mw-config - Aeywoo the build passed. [16:13:39] [02ssl] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/186e46d11bc74c66f3d709edd48deb565587f061 [16:13:39] 02ssl/03main 07MacFan4000 03186e46d add wiki.alphaadvanced.online [16:13:51] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.52, 21.74, 17.11 [16:17:51] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 14.31, 20.02, 17.71 [16:18:29] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03types 13https://github.com/miraheze/ManageWiki/commit/799eae1dc29d9a182de130f228b806a8df897d03 [16:18:29] 02ManageWiki/03types 07CosmicAlpha 03799eae1 Minor cleanup [16:22:37] [02ssl] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/5b2e872042f9f1763c16ac978c6e3575ffab2d06 [16:22:37] 02ssl/03main 07MacFan4000 035b2e872 switch 9b9t.wiki to cloudflare. [16:22:38] [02dns] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/9969b85c90160eac32435a30349e4e0b60a443b7 [16:22:39] 02dns/03main 07MacFan4000 039969b85 Remove zone for 9b9t.wiki [16:23:21] miraheze/ManageWiki - Universal-Omega the build passed. [16:24:38] miraheze/dns - MacFan4000 the build passed. [16:28:44] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.99, 7.43, 6.30 [16:32:43] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 4.90, 6.32, 6.12 [16:42:40] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.06, 6.88, 6.45 [16:46:38] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 6.10, 6.61, 6.44 [16:58:05] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.71, 6.83, 6.64 [17:01:53] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.53, 7.82, 7.09 [17:03:46] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.95, 7.81, 7.17 [17:20:34] I think matomo will need more resources [17:31:00] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.55, 7.77, 7.41 [17:32:42] Matomo requires more sustenance. [17:33:00] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.74, 7.15, 7.21 [17:39:00] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 5.25, 6.13, 6.75 [17:39:10] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03types 13https://github.com/miraheze/ManageWiki/commit/0198502c8ceb302927f54fbbeafac2524be7450e [17:39:10] 02ManageWiki/03types 07CosmicAlpha 030198502 Cleanup [17:43:54] miraheze/ManageWiki - Universal-Omega the build passed. [18:00:00] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03types 13https://github.com/miraheze/ManageWiki/commit/afa7d6af1a66d04df5b19e36be366e70264e5e02 [18:00:00] 02ManageWiki/03types 07CosmicAlpha 03afa7d6a Minor cleanup [18:04:41] miraheze/ManageWiki - Universal-Omega the build passed. [18:15:00] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.38, 7.52, 6.89 [18:19:00] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 4.38, 6.24, 6.54 [18:51:13] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 29.08, 24.13, 18.74 [19:49:13] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 12.01, 15.92, 22.81 [19:55:13] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 16.75, 15.21, 20.07 [20:12:04] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.63, 21.67, 19.08 [20:34:59] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03types 13https://github.com/miraheze/ManageWiki/commit/b7bd9c901bbf2522961f8dab2936b3792ce95465 [20:34:59] 02ManageWiki/03types 07CosmicAlpha 03b7bd9c9 Major cleanup [20:36:17] [02ManageWiki] 07github-actions[bot] pushed 1 new commit to 03types 13https://github.com/miraheze/ManageWiki/commit/91ce7a75a8fe0b6f4efdf4193f9af879fc248ec7 [20:36:17] 02ManageWiki/03types 07github-actions 0391ce7a7 CI: lint code to MediaWiki standards… [20:39:52] miraheze/ManageWiki - Universal-Omega the build passed. [20:47:20] [Grafana] FIRING: The mediawiki job queue has more than 20000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1744746410000&orgId=1&to=1744750040034 [21:26:37] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03types 13https://github.com/miraheze/ManageWiki/commit/424ea5845aeb65451cc31d6af092f4fc1a82067d [21:26:37] 02ManageWiki/03types 07CosmicAlpha 03424ea58 Fix [21:27:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?from=1744746410000&orgId=1&to=1744752410000 [21:29:25] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03types 13https://github.com/miraheze/ManageWiki/commit/f38a872f6cc16401108ee580496607eb7ccc6f73 [21:29:25] 02ManageWiki/03types 07CosmicAlpha 03f38a872 - [21:31:04] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03types 13https://github.com/miraheze/ManageWiki/commit/202dd4536ea633ef489f203669b85796a1bd94ff [21:31:04] 02ManageWiki/03types 07CosmicAlpha 03202dd45 Fix [21:33:45] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03types 13https://github.com/miraheze/ManageWiki/commit/581fa309182fb400d8f4df9ba87301a35b48dd59 [21:33:45] 02ManageWiki/03types 07CosmicAlpha 03581fa30 Fix [21:38:35] miraheze/ManageWiki - Universal-Omega the build passed. [21:44:30] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03types 13https://github.com/miraheze/ManageWiki/commit/a56c54cd94428b50e9d6b582a594e935f473f049 [21:44:30] 02ManageWiki/03types 07CosmicAlpha 03a56c54c cleanup [21:45:13] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 8.29, 15.25, 22.94 [21:49:13] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 6.22, 10.50, 19.24 [21:49:19] miraheze/ManageWiki - Universal-Omega the build passed. [22:02:40] [02DiscordNotifications] 071108-Kiju opened pull request #94: Fix URL encoding for article titles while preserving query parameters (07miraheze:03main...071108-Kiju:031108-Kiju-patch-1) 13https://github.com/miraheze/DiscordNotifications/pull/94 [22:02:47] [02DiscordNotifications] 07coderabbitai[bot] commented on pull request #94: --- […] 13https://github.com/miraheze/DiscordNotifications/pull/94#issuecomment-2807628326 [22:08:13] [02DiscordNotifications] 07Universal-Omega merged 071108-Kiju's pull request #94: Fix URL encoding for article titles while preserving query parameters (07miraheze:03main...071108-Kiju:031108-Kiju-patch-1) 13https://github.com/miraheze/DiscordNotifications/pull/94 [22:08:13] [02DiscordNotifications] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/DiscordNotifications/commit/0e1833e852666dbc4d4a34694e02213326660b0e [22:08:13] 02DiscordNotifications/03main 07Kiju 030e1833e Fix URL encoding for article titles while preserving query parameters (#94)… [22:10:24] [02ManageWiki] 07Universal-Omega retitled PR #544: "Type hint extension and some cleanup" ➜ "Type hint extension and major cleanup" 13https://github.com/miraheze/ManageWiki/pull/544 [22:11:52] miraheze/DiscordNotifications - Universal-Omega the build has errored. [22:19:30] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03types 13https://github.com/miraheze/ManageWiki/commit/02e3a70ad68fe8e193ae824171367268522e7ea9 [22:19:31] 02ManageWiki/03types 07CosmicAlpha 0302e3a70 Use options-messages [22:22:34] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03types 13https://github.com/miraheze/ManageWiki/commit/99693d495e9f97236a10e05fb7a2b8f456e0a1aa [22:22:34] 02ManageWiki/03types 07CosmicAlpha 0399693d4 Comments [22:23:30] miraheze/DiscordNotifications - 1108-Kiju the build passed. [22:27:31] miraheze/ManageWiki - Universal-Omega the build passed. [22:28:29] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/520ecaa36ad9c4a52121cb4c85bb652a021194f6 [22:28:29] 02MirahezeMagic/03main 07CosmicAlpha 03520ecaa Use array for ManageWikiSettings->remove [22:33:58] miraheze/MirahezeMagic - Universal-Omega the build has errored. [22:38:31] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03types 13https://github.com/miraheze/ManageWiki/commit/76b5a661e89af46701f57511865311c1a0d332af [22:38:31] 02ManageWiki/03types 07CosmicAlpha 0376b5a66 Use RemoteWiki [22:43:14] miraheze/ManageWiki - Universal-Omega the build passed. [22:44:41] [02mw-config] 071108-Kiju opened pull request #5910: +securepoll-edit-poll (07miraheze:03main...071108-Kiju:03main) 13https://github.com/miraheze/mw-config/pull/5910 [22:52:44] [02mw-config] 071108-Kiju retitled PR #5910: "+securepoll-edit-poll" ➜ "Add securepoll-edit-poll perm to electionadmin on metawiki" 13https://github.com/miraheze/mw-config/pull/5910 [22:58:50] [02mw-config] 07OAuthority commented on pull request #5910: LGTM. 13https://github.com/miraheze/mw-config/pull/5910#issuecomment-2807717607 [22:58:53] [02mw-config] 07OAuthority merged 071108-Kiju's pull request #5910: Add securepoll-edit-poll perm to electionadmin on metawiki (07miraheze:03main...071108-Kiju:03main) 13https://github.com/miraheze/mw-config/pull/5910 [22:58:55] [02mw-config] 07OAuthority pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/9757e5c1c58eaa5ecd371bc2d3d7934143059004 [22:58:57] 02mw-config/03main 07Kiju 039757e5c Add securepoll-edit-poll perm to electionadmin on metawiki (#5910)… [22:59:43] miraheze/mw-config - OAuthority the build passed. [23:24:53] [02puppet] 07paladox created 03paladox-patch-10 (+1 new commit) 13https://github.com/miraheze/puppet/commit/72535f1f3171 [23:24:53] 02puppet/03paladox-patch-10 07paladox 0372535f1 cache::haproxy: unset forwardfor… [23:24:57] [02puppet] 07paladox opened pull request #4335: cache::haproxy: unset forwardfor (03main...03paladox-patch-10) 13https://github.com/miraheze/puppet/pull/4335 [23:25:03] [02puppet] 07coderabbitai[bot] commented on pull request #4335: --- […] 13https://github.com/miraheze/puppet/pull/4335#issuecomment-2807753729 [23:26:31] [02puppet] 07paladox merged pull request #4335: cache::haproxy: unset forwardfor (03main...03paladox-patch-10) 13https://github.com/miraheze/puppet/pull/4335 [23:26:31] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/36ed960aa4b1bbab27585d4882993844051f083a [23:26:31] 02puppet/03main 07paladox 0336ed960 cache::haproxy: unset forwardfor (#4335)… [23:26:33] [02puppet] 07paladox 04deleted 03paladox-patch-10 at 0372535f1 13https://api.github.com/repos/miraheze/puppet/commit/72535f1 [23:34:47] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 29.04, 23.44, 18.89 [23:36:02] !log [oa@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [23:36:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:36:35] !log [oa@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 33s [23:36:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log