[00:00:42] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 51679MiB (11% inode=99%); [00:02:19] RECOVERY - cp38 Disk Space on cp38 is OK: DISK OK - free space: / 53123MiB (11% inode=99%); [00:48:51] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03csrftokenset 13https://github.com/miraheze/CreateWiki/commit/33d8dda4aeb5e81adb7c9103e28d2a3633b07815 [00:48:51] 02CreateWiki/03csrftokenset 07CosmicAlpha 0333d8dda - [00:56:07] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/ManageWiki/commit/e903d646a50a600d04a51e614ff8a76f07540a02 [00:56:07] 02ManageWiki/03phan-strict 07CosmicAlpha 03e903d64 Try fix [00:59:24] miraheze/ManageWiki - Universal-Omega the build has errored. [01:07:39] miraheze/CreateWiki - Universal-Omega the build passed. [01:14:21] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03csrftokenset 13https://github.com/miraheze/CreateWiki/commit/d5c66069ec9b8b4c193435ad80c1189a48c2afa9 [01:14:22] 02CreateWiki/03csrftokenset 07CosmicAlpha 03d5c6606 Remove [01:15:18] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03csrftokenset 13https://github.com/miraheze/CreateWiki/commit/7684f900b2703d813bd1c9052af23c1309d77617 [01:15:18] 02CreateWiki/03csrftokenset 07CosmicAlpha 037684f90 Remove [01:22:09] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/ManageWiki/commit/fcca6eafc9caf5553f8f1872bc943e63778e8c23 [01:22:09] 02ManageWiki/03phan-strict 07CosmicAlpha 03fcca6ea Try [01:25:37] miraheze/ManageWiki - Universal-Omega the build has errored. [01:30:24] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/ManageWiki/commit/0d1ca8d7ada8b438d8b38d8d3102def6719aa7a7 [01:30:24] 02ManageWiki/03phan-strict 07CosmicAlpha 030d1ca8d Try [01:30:43] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/ManageWiki/commit/00efafcde39ee74fbb7fcb6903dccd87bb42c8a3 [01:30:44] 02ManageWiki/03phan-strict 07CosmicAlpha 0300efafc - [01:34:19] PROBLEM - cp38 Disk Space on cp38 is WARNING: DISK WARNING - free space: / 49845MiB (10% inode=99%); [01:34:22] miraheze/ManageWiki - Universal-Omega the build has errored. [01:34:37] miraheze/CreateWiki - Universal-Omega the build passed. [01:37:02] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/ManageWiki/commit/f3d4862c18e27294f38bd69fa90cb3f49b39e839 [01:37:02] 02ManageWiki/03phan-strict 07CosmicAlpha 03f3d4862 Fix [01:37:39] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/ManageWiki/commit/d4741b824554a75f79de5ba80e6b3e17103c6e9d [01:37:39] 02ManageWiki/03phan-strict 07CosmicAlpha 03d4741b8 Fix [01:39:34] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/ManageWiki/commit/aba8d6ec39c376543d8071e9bcb765c5d6bdeb13 [01:39:34] 02ManageWiki/03phan-strict 07CosmicAlpha 03aba8d6e Fix [01:41:41] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/ManageWiki/commit/6d09912c66a2276173c97e6f3c5756de42003815 [01:41:41] 02ManageWiki/03phan-strict 07CosmicAlpha 036d09912 Fix [01:43:01] [02CreateWiki] 07Universal-Omega merged pull request #744: SpecialRequestWikiTest: use CsrfTokenSet (03main...03csrftokenset) 13https://github.com/miraheze/CreateWiki/pull/744 [01:43:02] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/b5d7ecfebf9511b792c1948afd29f2ba53afa843 [01:43:02] 02CreateWiki/03main 07CosmicAlpha 03b5d7ecf SpecialRequestWikiTest: use CsrfTokenSet [01:43:03] [02CreateWiki] 07Universal-Omega 04deleted 03csrftokenset at 037684f90 13https://api.github.com/repos/miraheze/CreateWiki/commit/7684f90 [01:44:42] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49796MiB (10% inode=99%); [01:47:08] miraheze/ManageWiki - Universal-Omega the build passed. [01:54:03] [02ssl] 07Universal-Omega merged 07SomeMWDev's pull request #854: T13770: Redirect everythingupgradetreewiki (07miraheze:03main...07SomeMWDev:03T13770) 13https://github.com/miraheze/ssl/pull/854 [01:54:03] [02ssl] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/b5e68804ebf62cf0fbae21848f892dbea66855c4 [01:54:03] 02ssl/03main 07SomeRandomDeveloper 03b5e6880 T13770: Redirect everythingupgradetreewiki (#854)… [02:01:51] miraheze/CreateWiki - Universal-Omega the build passed. [02:18:10] [02RequestSSL] 07Universal-Omega closed pull request #40: T11699: Create a job to automate CNAME checks (07miraheze:03main...07redbluegreenhat:03T11699-introduce-domaincheckjob) 13https://github.com/miraheze/RequestSSL/pull/40 [02:47:07] [02RequestSSL] 07Universal-Omega requested changes on pull request #75 13https://github.com/miraheze/RequestSSL/pull/75#pullrequestreview-2869462637 [02:47:07] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: !== 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2108083248 [02:47:07] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: Both this and Logger should be added as services in ServiceWiring and injected 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117061024 [02:47:07] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: Same here, avoid optional parameters. Also, tabbing is off here. 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117063107 [02:47:09] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: I'd just return an array and use an empty array to indicate failure which reduces return type to just one. But up to you. 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117065537 [02:47:11] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: This means the job will repeatedly try. I'd avoid setLastError just rely on logger->debug, and return true here and other places. 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117062261 [02:47:13] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: Use in_array, this is very hard to read here with all the different codes when it could just check the codes in an array. 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117063458 [02:47:15] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: ```suggestion […] 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117072152 [02:47:17] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: Since you use ?? below that implies $request['body'] can be not set at all. That means it will give undefined index warnings here. Would consider using isset or similar here. 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117064858 [02:47:19] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: I'd just return an array and use an empty array to indicate failure which reduces return type to just one. But up to you. 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117065714 [02:47:21] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: Don't really need ->params, just pass as second option to ->msg but it's up to you. 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117074976 [02:47:23] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: avoid optional parameters, especially when private. Reduces the risk of unexpected behavior/forgetting to pass something if it has to be done explicitly. 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117062724 [02:47:25] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: !== 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117066604 [02:47:27] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: What if http is given to it, which since at this point we don't have a custom domain/ssl certificate it could make sense http could be given to it... 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117076268 [02:47:29] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: Condense to one line `[ $this->id ]` 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117077943 [02:47:31] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: Don't really need ->params, just pass as second option to ->msg but it's up to you. 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117074668 [02:47:33] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: Condense to one line `[ $this->id ]` 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117077022 [02:47:35] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: Condense to one line `[ $this->id ]` 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117076721 [02:47:37] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: Looking at it, statusresponse is indeed not guaranteed to be set. It's hard to follow the code here though, consider breaking things into empty new lines and maybe adding some comments making it easier to follow. But up to you. 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117066486 [02:47:39] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: just calling it $requestManager makes since since this is RequestSSL, also makes it easier for when we end up renaming the extension. Also make readonly. 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117083810 [02:47:41] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: Remove; this is set via property promotion 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117081403 [02:47:43] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: Is there any more specific error we can catch here? If not it's fine just wondering. 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117085805 [02:47:45] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: Condense to one line `[ $this->id ]` 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117078258 [02:47:47] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: Add `@inheritDoc` comment 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117088611 [02:47:49] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: Suggestion, not requirement: […] 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117090217 [02:47:51] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: Suggestion, not requirement: […] 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117089463 [02:47:53] [02RequestSSL] 07Universal-Omega left a file comment in pull request #75 034d205e0: Suggestion, not requirement: […] 13https://github.com/miraheze/RequestSSL/pull/75#discussion_r2117090774 [04:06:41] [02mw-config] 07OAuthority merged 07SomeMWDev's pull request #5978: T13740: Change footer icons for nicosnextbotswiki (07miraheze:03main...07SomeMWDev:03T13740) 13https://github.com/miraheze/mw-config/pull/5978 [04:06:41] [02mw-config] 07OAuthority pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/4808424b1630edb6e59a9d4c5ff4455ebd5e7378 [04:06:41] 02mw-config/03main 07SomeRandomDeveloper 034808424 T13740: Change footer icons for nicosnextbotswiki (#5978)… [04:06:57] [02mw-config] 07OAuthority merged 07SomeMWDev's pull request #5977: T13750: set $wgCategoryCollation for rapanuidictionaryprojectwiki (07miraheze:03main...07SomeMWDev:03T13750) 13https://github.com/miraheze/mw-config/pull/5977 [04:06:59] [02mw-config] 07OAuthority pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/a9c9f23bfbbec5babaf9c51da0ab0e8cb8fdd3d7 [04:07:01] 02mw-config/03main 07SomeRandomDeveloper 03a9c9f23 T13750: set $wgCategoryCollation for rapanuidictionaryprojectwiki (#5977)… [04:07:35] miraheze/mw-config - OAuthority the build passed. [04:07:46] miraheze/mw-config - OAuthority the build passed. [04:34:18] PROBLEM - decimatedrive.wiki - Cloudflare on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80FB5031BF140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1605:SSL alert number 40 [05:28:51] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.33, 5.77, 4.48 [05:30:48] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 3.44, 4.86, 4.31 [08:26:18] PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 13 backends are down. mw151 mw152 mw162 mw171 mw172 mw153 mw163 mw183 mw191 mw193 mw201 mw202 mw203 [08:28:18] RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy [09:09:10] PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 3 backends are down. mw172 mw182 mw163 [09:11:09] RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy [09:29:31] !log re-run smpearth import with additional images [09:29:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:41:50] !log deploy slightly aggressive captcha [09:41:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:03:13] PROBLEM - cp38 Disk Space on cp38 is CRITICAL: DISK CRITICAL - free space: / 27210MiB (5% inode=99%); [11:12:37] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old snagowiki --new snagowikiwiki --user=BlankEclair_(Miraheze) (END - exit=512) [11:12:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:13:13] blankeclair: test [11:14:57] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old snagowiki --new snagowikiwiki --user='BlankEclair_(Miraheze)' (END - exit=0) [11:15:31] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 5 backends are down. mw151 mw171 mw182 mw173 mw201 [11:15:40] oh fuck [11:16:25] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [11:16:29] PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 14 backends are down. mw151 mw152 mw162 mw172 mw181 mw182 mw153 mw163 mw173 mw192 mw193 mw201 mw202 mw203 [11:16:35] Sad [11:16:35] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 4 backends are down. mw151 mw172 mw153 mw163 [11:16:43] That's not supposed to happen [11:16:47] Do you need help BlankEclair ? [11:16:59] gonna check traffic in cf real quick [11:17:31] > [31/05/2025 21:16] Do you need help BlankEclair ? [11:17:34] yeah, i have no idea what i'm doing [11:17:54] huge spike in managed challenge [11:18:14] oooh [11:18:31] it's that recent rule you added [11:19:08] almost one million reqs in two bloody hours [11:19:11] That's not related ye [11:19:16] That's my rule to mitigate things [11:19:38] https://grafana.wikitide.net/d/GtxbP1Xnk/mediawiki?orgId=1&from=now-1h&to=now&timezone=browser&var-node=$__all&var-job=$__all is where we start [11:19:59] oh [11:20:26] A couple of mw's are at 100% [11:22:25] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4037 bytes in 0.081 second response time [11:23:47] Nothing is screaming at me [11:24:11] db171 looked a tiny bit angry [11:24:36] [[Special:ManageWiki/snagowikiwiki]] [11:24:44] @link [11:24:44] https://meta.miraheze.org/wiki/Special:ManageWiki/snagowikiwiki [11:25:26] ah yes xD [11:28:56] RhinosF1: cf detected a ddos btw [11:29:02] only 107 reqs blocked tho :p [11:29:19] BlankEclair: just now? [11:29:23] yep [11:29:36] 07:30 -> 08:00, utc [11:29:43] That's not just now [11:29:45] oh [11:29:50] I saw that one earlier [11:29:55] oh, sorry ^^; [11:30:12] I'm not seeing anything obvious as to why we are sad [11:30:38] idk, people say that to me all the time /j [11:31:17] BlankEclair: anything in graylog's errors? [11:31:25] lemmec heck [11:32:43] RhinosF1: graylog errors? [11:32:53] as in the mediawiki error logs stream? [11:32:55] BlankEclair: ye [11:33:10] Something like timeouts connecting to db / memcache [11:33:40] > Expected key global:centralautologin-returnurl:mhglobal: not found due to timeout. [11:35:31] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 31 backends are healthy [11:36:12] PROBLEM - cp38 HTTPS on cp38 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [11:40:17] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 31 backends are healthy [11:42:18] RECOVERY - cp38 HTTPS on cp38 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4037 bytes in 0.060 second response time [11:44:21] RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy [11:57:13] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 27193MiB (5% inode=99%); [12:29:44] BlankEclair: a lot of that? That's a memcache fault then [12:29:51] fun [12:38:18] PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 [12:39:15] do we _have_ to shit the bed [12:39:31] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw152 [12:39:41] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw152 [12:39:53] PROBLEM - cp38 HTTPS on cp38 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:40:20] BlankEclair: memcache playing up is a paladox issue [12:40:27] ah okay [12:40:40] i'm in the midst of investigating a wiki rn tho, so that's inconvenient [12:41:52] RECOVERY - cp38 HTTPS on cp38 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4037 bytes in 0.169 second response time [12:43:41] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 31 backends are healthy [12:44:18] RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy [12:48:18] PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 6 backends are down. mw171 mw181 mw182 mw153 mw192 mw201 [12:50:25] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [12:50:36] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw183 [12:51:31] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 31 backends are healthy [12:52:25] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4037 bytes in 0.064 second response time [12:52:35] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 31 backends are healthy [12:54:18] RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy [13:26:28] PROBLEM - cp37 Puppet on cp37 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 27 minutes ago with 0 failures [13:27:57] [02puppet] 07paladox created 03paladox-patch-10 (+1 new commit) 13https://github.com/miraheze/puppet/commit/6bd6696f1fe4 [13:27:57] 02puppet/03paladox-patch-10 07paladox 036bd6696 cache::haproxy: Put strings around some of the returned logging data… [13:28:01] [02puppet] 07paladox opened pull request #4358: cache::haproxy: Put strings around some of the returned logging data (03main...03paladox-patch-10) 13https://github.com/miraheze/puppet/pull/4358 [13:28:07] [02puppet] 07coderabbitai[bot] commented on pull request #4358: --- […] 13https://github.com/miraheze/puppet/pull/4358#issuecomment-2925184940 [13:32:56] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 31.09, 18.77, 10.36 [13:34:56] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 16.30, 18.73, 11.48 [13:51:49] [02puppet] 07paladox created 03paladox-patch-11 (+1 new commit) 13https://github.com/miraheze/puppet/commit/e326a406d9c7 [13:51:49] 02puppet/03paladox-patch-11 07paladox 03e326a40 cache::haproxy: modify log-format… [13:51:52] [02puppet] 07paladox opened pull request #4359: cache::haproxy: modify log-format (03main...03paladox-patch-11) 13https://github.com/miraheze/puppet/pull/4359 [13:51:59] [02puppet] 07coderabbitai[bot] commented on pull request #4359: --- […] 13https://github.com/miraheze/puppet/pull/4359#issuecomment-2925215133 [13:53:47] [02puppet] 07paladox merged pull request #4359: cache::haproxy: modify log-format (03main...03paladox-patch-11) 13https://github.com/miraheze/puppet/pull/4359 [13:53:47] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/005555bee6b1be5f3757a7a8cb0b065d0bd4103b [13:53:47] 02puppet/03main 07paladox 03005555b cache::haproxy: modify log-format (#4359)… [13:53:48] [02puppet] 07paladox 04deleted 03paladox-patch-11 at 03e326a40 13https://api.github.com/repos/miraheze/puppet/commit/e326a40 [13:54:18] PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 11 backends are down. mw151 mw152 mw171 mw172 mw182 mw163 mw173 mw191 mw193 mw202 mw203 [13:54:28] RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:55:31] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 3 backends are down. mw153 mw163 mw201 [13:55:45] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 3 backends are down. mw153 mw163 mw201 [13:57:31] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 31 backends are healthy [13:57:43] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 31 backends are healthy [14:02:35] [02puppet] 07paladox created 03paladox-patch-11 (+1 new commit) 13https://github.com/miraheze/puppet/commit/979f7fac3d62 [14:02:35] 02puppet/03paladox-patch-11 07paladox 03979f7fa varnish-requests-exporter: support haproxy alongside nginx [14:02:39] [02puppet] 07paladox opened pull request #4360: varnish-requests-exporter: support haproxy alongside nginx (03main...03paladox-patch-11) 13https://github.com/miraheze/puppet/pull/4360 [14:02:47] [02puppet] 07coderabbitai[bot] commented on pull request #4360: --- […] 13https://github.com/miraheze/puppet/pull/4360#issuecomment-2925224850 [14:03:15] [02puppet] 07paladox merged pull request #4360: varnish-requests-exporter: support haproxy alongside nginx (03main...03paladox-patch-11) 13https://github.com/miraheze/puppet/pull/4360 [14:03:16] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/138ca4c06ecb18d7f32f902115d36de8ea4e87e1 [14:03:16] 02puppet/03main 07paladox 03138ca4c varnish-requests-exporter: support haproxy alongside nginx (#4360) [14:03:16] [02puppet] 07paladox 04deleted 03paladox-patch-11 at 03979f7fa 13https://api.github.com/repos/miraheze/puppet/commit/979f7fa [14:06:18] RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy [14:23:02] [02dns] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/dns/commit/9e1367f2092a [14:23:02] 02dns/03paladox-patch-1 07paladox 039e1367f Switch polcompball.wikitide.org to use cf-lb [14:23:05] [02dns] 07paladox opened pull request #671: Switch polcompball.wikitide.org to use cf-lb (03main...03paladox-patch-1) 13https://github.com/miraheze/dns/pull/671 [14:24:03] [02dns] 07paladox merged pull request #671: Switch polcompball.wikitide.org to use cf-lb (03main...03paladox-patch-1) 13https://github.com/miraheze/dns/pull/671 [14:24:05] [02dns] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/8d75e3c65ca95e1d7c54596fdd9287b9d4731502 [14:24:07] [02dns] 07paladox 04deleted 03paladox-patch-1 at 039e1367f 13https://api.github.com/repos/miraheze/dns/commit/9e1367f [14:24:09] 02dns/03main 07paladox 038d75e3c Switch polcompball.wikitide.org to use cf-lb (#671) [14:24:56] miraheze/dns - paladox the build passed. [14:24:56] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 28.80, 18.08, 10.33 [14:25:13] miraheze/dns - paladox the build passed. [14:27:04] [02ssl] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/ssl/commit/ad3ea5f73040 [14:27:04] 02ssl/03paladox-patch-1 07paladox 03ad3ea5f Add polcompball temporarily until we can switch *.wikitide.org to cloudflare. [14:27:07] [02ssl] 07paladox opened pull request #855: Add polcompball temporarily until we can switch *.wikitide.org to clo… (03main...03paladox-patch-1) 13https://github.com/miraheze/ssl/pull/855 [14:27:14] [02ssl] 07paladox retitled PR #855: "Add polcompball temporarily until we can switch *.wikitide.org to clo…" ➜ "Add polcompball temporarily until we can switch *.wikitide.org to cloudflare" 13https://github.com/miraheze/ssl/pull/855 [14:27:27] [02ssl] 07paladox merged pull request #855: Add polcompball temporarily until we can switch *.wikitide.org to cloudflare (03main...03paladox-patch-1) 13https://github.com/miraheze/ssl/pull/855 [14:27:27] [02ssl] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/a752ba7032ce443840d7559959c56614d19f5bd4 [14:27:28] 02ssl/03main 07paladox 03a752ba7 Add polcompball temporarily until we can switch *.wikitide.org to cloudflare (#855) [14:27:28] [02ssl] 07paladox 04deleted 03paladox-patch-1 at 03ad3ea5f 13https://api.github.com/repos/miraheze/ssl/commit/ad3ea5f [14:30:48] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/rebuildall.php --wiki=vocaloidlyricswiki (END - exit=35072) [14:30:52] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/initSiteStats.php --wiki=vocaloidlyricswiki --update (END - exit=0) [14:30:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:32:31] [02ssl] 07coderabbitai[bot] commented on pull request #855: A new entry for the domain `polcompball.wikitide.org` was added to the `certs.yaml` configuration file. This entry specifies Cloudflare as the certificate authority and assigns the SSL certificate name `miraheze-origin-cert`. No other changes were […] 13https://github.com/miraheze/ssl/pull/855#issuecomment-2925263487 [14:32:56] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 13.79, 21.41, 15.89 [14:34:56] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 5.38, 16.14, 14.66 [15:21:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.45, 18.61, 11.50 [15:23:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.32, 20.55, 13.08 [15:37:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.12, 23.63, 19.62 [15:41:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.31, 23.98, 20.68 [15:45:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.62, 23.85, 21.40 [15:55:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.26, 23.78, 22.38 [15:57:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 12.41, 20.02, 21.22 [15:59:12] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 6.28, 15.22, 19.32 [16:42:32] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 7.64, 10.80, 7.00 [16:44:32] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 3.63, 8.21, 6.50 [16:50:18] great rebuildAll failed after running for 3 days [16:52:02] MacFan4000: OOM or? [16:52:52] no error, just says Killed [16:56:33] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=vocaloidlyricswiki 138300 (START) [16:56:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:31:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.42, 18.89, 13.44 [17:33:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.10, 20.36, 14.63 [17:37:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.80, 22.19, 16.71 [17:39:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.02, 22.73, 17.57 [17:41:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.65, 22.60, 18.15 [17:43:16] [02ssl] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/14c179145196a4d1258c8abfc19e3c898ff68e89 [17:43:17] 02ssl/03main 07MacFan4000 0314c1791 add 3 domains [17:53:12] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 14.25, 20.20, 19.82 [17:59:51] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/CreateWiki/commit/31aa346a4df658fad979bb6f8dc12f14f602aaaf [17:59:52] 02CreateWiki/03phan-strict 07CosmicAlpha 0331aa346 Update [18:00:23] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/CreateWiki/commit/3aa731a5a7e20e2beb60c6bdd8717ca10789e557 [18:00:23] 02CreateWiki/03phan-strict 07CosmicAlpha 033aa731a Update [18:04:37] miraheze/CreateWiki - Universal-Omega the build has errored. [18:13:03] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 9.83, 11.36, 8.75 [18:14:58] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 5.06, 9.05, 8.20 [18:40:16] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.97, 15.93, 11.27 [18:44:08] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.16, 20.43, 14.16 [18:44:50] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/CreateWiki/commit/e31d43cc67b85e49bceaf3fec9c8f00931e0c0c0 [18:44:50] 02CreateWiki/03phan-strict 07CosmicAlpha 03e31d43c Update [18:48:55] miraheze/CreateWiki - Universal-Omega the build has errored. [18:49:49] PROBLEM - cloud20 IPMI Sensors on cloud20 is WARNING: IPMI Status: Warning [1 system event log (SEL) entry present] [18:53:48] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 14.46, 21.00, 18.03 [18:55:44] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 10.10, 17.10, 16.94 [19:03:20] PROBLEM - graylog161 NTP time on graylog161 is WARNING: NTP WARNING: Offset -0.1032516658 secs [19:03:57] PROBLEM - mem191 NTP time on mem191 is WARNING: NTP WARNING: Offset -0.1035484672 secs [19:04:17] PROBLEM - swiftac171 NTP time on swiftac171 is WARNING: NTP WARNING: Offset -0.1013822854 secs [19:04:18] PROBLEM - mw173 NTP time on mw173 is WARNING: NTP WARNING: Offset -0.1013244689 secs [19:05:26] RECOVERY - graylog161 NTP time on graylog161 is OK: NTP OK: Offset -0.09027576447 secs [19:05:58] RECOVERY - mem191 NTP time on mem191 is OK: NTP OK: Offset -0.09904623032 secs [19:06:19] RECOVERY - swiftac171 NTP time on swiftac171 is OK: NTP OK: Offset -0.09691736102 secs [19:06:20] RECOVERY - mw173 NTP time on mw173 is OK: NTP OK: Offset -0.0818375349 secs [19:07:55] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 7 backends are down. mw152 mw161 mw153 mw192 mw193 mw201 mw202 [19:09:01] PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 12 backends are down. mw151 mw161 mw172 mw181 mw182 mw153 mw163 mw173 mw191 mw201 mw202 mw203 [19:09:03] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [19:09:32] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 2 backends are down. mw181 mw192 [19:11:01] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4037 bytes in 0.082 second response time [19:11:55] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 31 backends are healthy [19:15:17] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [19:18:27] PROBLEM - cp38 HTTPS on cp38 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [19:19:14] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4037 bytes in 0.059 second response time [19:21:33] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 31 backends are healthy [19:24:48] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/CreateWiki/commit/eedb2d02aa7c0db5cf7f44adeded5d3c6b3670b5 [19:24:48] 02CreateWiki/03phan-strict 07CosmicAlpha 03eedb2d0 Update [19:25:32] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw183 [19:25:55] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 13 backends are down. mw151 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw183 mw192 mw193 mw202 [19:26:53] PROBLEM - cp38 Varnish Backends on cp38 is WARNING: No backends detected. If this is an error, see readme.txt [19:27:27] !log restart varnish on cp* [19:27:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:32] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 31 backends are healthy [19:27:34] !log restart nginx/php-fpm on mw* [19:27:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:55] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 31 backends are healthy [19:28:01] RECOVERY - cp38 HTTPS on cp38 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4037 bytes in 0.065 second response time [19:28:52] RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy [19:28:52] miraheze/CreateWiki - Universal-Omega the build has errored. [19:30:55] Turning it off and on again is always the solution isn’t it [19:37:00] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/CreateWiki/commit/aae29edea1d9d4ea0154263b508ec32d0871f036 [19:37:00] 02CreateWiki/03phan-strict 07CosmicAlpha 03aae29ed Merge branch 'main' into phan-strict [19:38:56] [02CreateWiki] 07github-actions[bot] pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/CreateWiki/commit/c9d406d4cb0d0ec29f0934eecc24b24f03ae392e [19:38:56] 02CreateWiki/03phan-strict 07github-actions 03c9d406d CI: lint code to MediaWiki standards… [19:40:00] [02puppet] 07paladox created 03paladox-patch-11 (+1 new commit) 13https://github.com/miraheze/puppet/commit/a54c31fcc286 [19:40:00] 02puppet/03paladox-patch-11 07paladox 03a54c31f mediawiki: lower mediawiki::php::fpm::fpm_workers_multiplier to 1.5… [19:40:04] [02puppet] 07paladox opened pull request #4361: mediawiki: lower mediawiki::php::fpm::fpm_workers_multiplier to 1.5 (03main...03paladox-patch-11) 13https://github.com/miraheze/puppet/pull/4361 [19:40:10] [02puppet] 07coderabbitai[bot] commented on pull request #4361: --- […] 13https://github.com/miraheze/puppet/pull/4361#issuecomment-2925653242 [19:41:32] miraheze/CreateWiki - Universal-Omega the build has errored. [19:42:11] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/CreateWiki/commit/4b4d7dc7bd9b449e66cc97b8b19fb1466153956c [19:42:12] 02CreateWiki/03phan-strict 07CosmicAlpha 034b4d7dc Make non-null [19:42:34] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/CreateWiki/commit/f4fa96dbe60d6c7bcea2b9dc3d39b3ec51663351 [19:42:34] 02CreateWiki/03phan-strict 07CosmicAlpha 03f4fa96d - [19:43:08] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/CreateWiki/commit/ef9e1875ac92c9c5e8f145a90479af72f37e337d [19:43:08] 02CreateWiki/03phan-strict 07CosmicAlpha 03ef9e187 - [19:43:45] [02puppet] 07paladox merged pull request #4361: mediawiki: lower mediawiki::php::fpm::fpm_workers_multiplier to 1.5 (03main...03paladox-patch-11) 13https://github.com/miraheze/puppet/pull/4361 [19:43:45] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/2b3d325cd324620944868076f7a000485c9fa491 [19:43:45] 02puppet/03main 07paladox 032b3d325 mediawiki: lower mediawiki::php::fpm::fpm_workers_multiplier to 1.5 (#4361)… [19:43:46] [02puppet] 07paladox 04deleted 03paladox-patch-11 at 03a54c31f 13https://api.github.com/repos/miraheze/puppet/commit/a54c31f [19:45:27] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/CreateWiki/commit/7c86e2bc30b0b8b2f4c811953b98a42170ec8af0 [19:45:27] 02CreateWiki/03phan-strict 07CosmicAlpha 037c86e2b Fix [19:48:20] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/CreateWiki/commit/ec14b2c7623f46f9a56f344915abe212297f1984 [19:48:20] 02CreateWiki/03phan-strict 07CosmicAlpha 03ec14b2c Fix [19:52:15] miraheze/CreateWiki - Universal-Omega the build has errored. [19:56:04] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03phan-strict 13https://github.com/miraheze/CreateWiki/commit/767885c5333007025c410459c79cb3ce47726939 [19:56:04] 02CreateWiki/03phan-strict 07CosmicAlpha 03767885c Fix [20:14:32] miraheze/CreateWiki - Universal-Omega the build passed. [21:23:29] PROBLEM - cp36 Nginx Backend for swiftproxy161 on cp36 is CRITICAL: connect to address localhost and port 8206: Connection refused [21:23:38] PROBLEM - cp36 Nginx Backend for mw203 on cp36 is CRITICAL: connect to address localhost and port 8134: Connection refused [21:23:42] PROBLEM - cp36 Nginx Backend for test151 on cp36 is CRITICAL: connect to address localhost and port 8181: Connection refused [21:23:43] PROBLEM - cp36 Nginx Backend for mw192 on cp36 is CRITICAL: connect to address localhost and port 8130: Connection refused [21:23:46] PROBLEM - cp36 Nginx Backend for mw172 on cp36 is CRITICAL: connect to address localhost and port 8118: Connection refused [21:23:47] PROBLEM - cp36 Nginx Backend for mw162 on cp36 is CRITICAL: connect to address localhost and port 8116: Connection refused [21:23:54] PROBLEM - cp36 Nginx Backend for mw193 on cp36 is CRITICAL: connect to address localhost and port 8131: Connection refused [21:24:08] PROBLEM - cp36 Nginx Backend for mwtask171 on cp36 is CRITICAL: connect to address localhost and port 8161: Connection refused [21:24:09] PROBLEM - cp36 Nginx Backend for mw163 on cp36 is CRITICAL: connect to address localhost and port 8123: Connection refused [21:24:16] PROBLEM - cp36 Nginx Backend for mw152 on cp36 is CRITICAL: connect to address localhost and port 8114: Connection refused [21:24:18] PROBLEM - cp36 Nginx Backend for mw153 on cp36 is CRITICAL: connect to address localhost and port 8121: Connection refused [21:24:24] PROBLEM - cp36 Nginx Backend for mw191 on cp36 is CRITICAL: connect to address localhost and port 8129: Connection refused [21:24:25] PROBLEM - cp36 Nginx Backend for mon181 on cp36 is CRITICAL: connect to address localhost and port 8201: Connection refused [21:24:27] PROBLEM - cp36 Nginx Backend for mw182 on cp36 is CRITICAL: connect to address localhost and port 8120: Connection refused [21:24:28] PROBLEM - cp36 Nginx Backend for puppet181 on cp36 is CRITICAL: connect to address localhost and port 8204: Connection refused [21:24:30] PROBLEM - cp36 Nginx Backend for swiftproxy171 on cp36 is CRITICAL: connect to address localhost and port 8207: Connection refused [21:24:30] PROBLEM - cp36 NTP time on cp36 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o