[00:00:47] PROBLEM - swiftobject161 Disk Space on swiftobject161 is WARNING: DISK WARNING - free space: / 80285MiB (6% inode=84%); [00:02:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:17:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:17:38] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 45570MiB (10% inode=99%); [00:27:16] RECOVERY - tech.exputra.com - Cloudflare on sslhost is OK: OK - Certificate 'tech.exputra.com' will expire on Wed 26 Mar 2025 06:03:13 PM GMT +0000. [00:46:30] [02CreateWiki] 07AgentIsai pushed 1 new commit to 03ai-improvements 13https://github.com/miraheze/CreateWiki/commit/ce4b3975e418594e0b0250fd66eadfe7a925f3bd [00:46:30] 02CreateWiki/03ai-improvements 07Agent Isai 03ce4b397 Fix some CI errors [00:50:47] PROBLEM - swiftobject161 Disk Space on swiftobject161 is CRITICAL: DISK CRITICAL - free space: / 80113MiB (5% inode=84%); [00:53:11] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.50, 2.55, 1.63 [00:53:45] miraheze/CreateWiki - AgentIsai the build has errored. [00:55:11] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 4.03, 2.95, 1.88 [00:55:38] [02CreateWiki] 07AgentIsai pushed 1 new commit to 03ai-improvements 13https://github.com/miraheze/CreateWiki/commit/477e1e3752879da89fd3aee91da2d9d29864eabf [00:55:38] 02CreateWiki/03ai-improvements 07Agent Isai 03477e1e3 Add qqq docs [00:57:11] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 2.87, 2.98, 2.03 [01:03:59] miraheze/CreateWiki - AgentIsai the build has errored. [01:06:47] [02CreateWiki] 07AgentIsai pushed 1 new commit to 03ai-improvements 13https://github.com/miraheze/CreateWiki/commit/0e0703f34d4f5fce87f75efe272c69b399a55ab9 [01:06:47] 02CreateWiki/03ai-improvements 07Agent Isai 030e0703f Fix lengths [01:08:22] [02CreateWiki] 07github-actions[bot] pushed 1 new commit to 03ai-improvements 13https://github.com/miraheze/CreateWiki/commit/860d8668b9f43d6fb35a20f71313ab0d71d50d2b [01:08:22] 02CreateWiki/03ai-improvements 07github-actions 03860d866 CI: lint code to MediaWiki standards… [01:17:38] miraheze/CreateWiki - AgentIsai the build passed. [01:23:00] PROBLEM - jobchron171 NTP time on jobchron171 is WARNING: NTP WARNING: Offset 0.1129050553 secs [01:25:00] RECOVERY - jobchron171 NTP time on jobchron171 is OK: NTP OK: Offset 0.06964915991 secs [02:03:05] PROBLEM - project-patterns.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'project-patterns.com' expires in 14 day(s) (Thu 06 Feb 2025 01:32:17 AM GMT +0000). [02:03:17] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/9f4095b2727f6aba06eac565e51905ba709a967f [02:03:17] 02ssl/03master 07WikiTideSSLBot 039f4095b Bot: Update SSL cert for project-patterns.com [02:32:28] RECOVERY - project-patterns.com - LetsEncrypt on sslhost is OK: OK - Certificate 'project-patterns.com' will expire on Tue 22 Apr 2025 01:04:41 AM GMT +0000. [03:02:59] PROBLEM - vylet.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vylet.wiki' expires in 14 day(s) (Thu 06 Feb 2025 02:42:38 AM GMT +0000). [03:03:11] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/ff0956dde9ad9b17737beec23dcf22bb341b427e [03:03:11] 02ssl/03master 07WikiTideSSLBot 03ff0956d Bot: Update SSL cert for vylet.wiki [03:32:28] RECOVERY - vylet.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'vylet.wiki' will expire on Tue 22 Apr 2025 02:04:34 AM GMT +0000. [06:02:11] [02CreateWiki] 07AgentIsai commented on pull request #651: @coderabbitai review 13https://github.com/miraheze/CreateWiki/pull/651#issuecomment-2606367717 [06:02:18] [02CreateWiki] 07coderabbitai[bot] commented on pull request #651:

[…] 13https://github.com/miraheze/CreateWiki/pull/651#issuecomment-2606367917 [06:04:18] [02CreateWiki] 07coderabbitai[bot] left a review on pull request #651: **Actionable comments posted: 1** […] 13https://github.com/miraheze/CreateWiki/pull/651#pullrequestreview-2566261937 [06:04:18] [02CreateWiki] 07coderabbitai[bot] left a file comment in pull request #651 03860d866: _:warning: Potential issue_ […] 13https://github.com/miraheze/CreateWiki/pull/651#discussion_r1924742362 [06:10:45] [02CreateWiki] 07AgentIsai pushed 1 new commit to 03ai-improvements 13https://github.com/miraheze/CreateWiki/commit/b4f79ef307088bf231e29847cd9ce55081c54dd0 [06:10:45] 02CreateWiki/03ai-improvements 07Agent Isai 03b4f79ef Implement suggestion for XSS protection [06:12:36] [02CreateWiki] 07github-actions[bot] pushed 1 new commit to 03ai-improvements 13https://github.com/miraheze/CreateWiki/commit/f92173fda08201bbb28b578b2927d87fa8769a46 [06:12:36] 02CreateWiki/03ai-improvements 07github-actions 03f92173f CI: lint code to MediaWiki standards… [06:18:10] miraheze/CreateWiki - AgentIsai the build has errored. [06:19:16] [02CreateWiki] 07AgentIsai pushed 1 new commit to 03ai-improvements 13https://github.com/miraheze/CreateWiki/commit/05cabac22f6e20e410d86562cfacec842f465456 [06:19:16] 02CreateWiki/03ai-improvements 07Agent Isai 0305cabac Fix length (again) [06:29:53] miraheze/CreateWiki - AgentIsai the build passed. [08:07:06] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:07:38] Huh [08:07:58] You're not supposed to say that I icinga-miraheze [08:09:12] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [08:10:08] Ye why? [08:10:28] no way [08:10:31] server down buds! [08:14:28] PROBLEM - cp37 Nginx Backend for matomo151 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: Connection refusedconnect to host 10.0.17.138 port 5666: Connection refused [08:14:28] PROBLEM - cp37 Nginx Backend for mw161 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: Connection refusedconnect to host 10.0.17.138 port 5666: Connection refused [08:16:28] RECOVERY - cp37 Nginx Backend for matomo151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8203 [08:16:28] RECOVERY - cp37 Nginx Backend for mw161 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8115 [08:17:12] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 2% [08:17:30] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4107 bytes in 0.058 second response time [11:59:38] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 27179MiB (5% inode=99%); [12:07:51] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 15 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw173 mw183 mw184 [12:08:06] Hmm [12:08:28] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 7 backends are down. mw161 mw172 mw181 mw163 mw164 mw174 mw183 [12:08:51] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:09:40] is it down again [12:09:52] im pretty sure it is since its using always online right now [12:10:04] Ye [12:10:08] It broke [12:10:15] I'm at work though [12:10:23] uh oh [12:10:41] is it backups again.. [12:10:59] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4130 bytes in 8.572 second response time [12:11:05] No idea [12:12:06] does that ok stuff mean its back [12:12:15] or like still in the process [12:12:22] hey its back [12:12:26] i lost some of my writing though [12:13:29] In the process of coming back [12:14:05] It's about half OK [12:14:06] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [12:14:13] https://grafana.wikitide.net/d/GtxbP1Xnk/mediawiki?orgId=1&from=now-5m&to=now&timezone=browser&var-node=$__all&var-job=$__all&viewPanel=panel-57 [12:14:19] I’m getting that to [12:14:40] https://cdn.discordapp.com/attachments/808001911868489748/1331597902581071913/IMG_5920.png?ex=679232af&is=6790e12f&hm=7799ee71a4cb93aeaf4b81fd20442fec14059b4725d151bce5dd01a52d58f0f3& [12:15:01] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:15:16] Is infra alerted [12:15:31] may wanna try the edit recovery feature [12:15:37] ? [12:16:04] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.511 second response time [12:16:10] Saves your edit to your computer every few seconds in case something happens to the tab or whatever [12:16:18] how do i do that? [12:16:35] In special:preferences [12:17:07] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:17:11] @Infrastructure Specialists courtesy ping [12:17:28] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:17:31] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.009 second response time [12:17:59] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [12:18:55] Yeah Grafana is showing read on CPU load [12:19:00] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4130 bytes in 2.658 second response time [12:19:01] well i'll enable it when i can 😅 [12:19:08] I think it’s the backup script again RhinosF1: [12:19:29] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.557 second response time [12:19:37] PROBLEM - mw184 MediaWiki Rendering on mw184 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [12:19:40] PROBLEM - mw184 HTTPS on mw184 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:19:52] Hilariously db load is going down [12:20:57] …wait [12:21:10] if its backups why is the load on mw and not db [12:21:30] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.153 second response time [12:21:36] RECOVERY - mw184 HTTPS on mw184 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.222 second response time [12:21:36] RECOVERY - mw184 MediaWiki Rendering on mw184 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.563 second response time [12:21:46] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:22:40] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:23:00] I am so confused [12:23:13] problem [12:23:25] Could it be a DDoS? There’s a spike in TCP connections maybe [12:23:25]

PROBLEM - ice spice is not on miraheze [12:23:47] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.934 second response time [12:23:54] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:24:07] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.536 second response time [12:24:27] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:24:34] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.154 second response time [12:25:52] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.536 second response time [12:26:25] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 1.279 second response time [12:26:32] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [12:27:46] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:50] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:28:07] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:28:14] PROBLEM - mw174 MediaWiki Rendering on mw174 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [12:28:32] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.235 second response time [12:30:11] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4108 bytes in 6.103 second response time [12:30:12] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [12:30:12] RECOVERY - mw174 MediaWiki Rendering on mw174 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.456 second response time [12:31:26] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [12:31:53] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.176 second response time [12:31:54] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.450 second response time [12:31:55] PROBLEM - mw164 HTTPS on mw164 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:32:11] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.899 second response time [12:32:59] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [12:33:51] RECOVERY - mw164 HTTPS on mw164 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 0.151 second response time [12:34:06] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [12:34:57] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 27% [12:35:59] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:36:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:36:11] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:30] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4108 bytes in 0.107 second response time [12:37:40] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.509 second response time [12:37:41] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.009 second response time [12:38:01] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3945 bytes in 6.328 second response time [12:38:10] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.515 second response time [12:38:35] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:39:40] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.183 second response time [12:39:52] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [12:40:12] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.185 second response time [12:40:28] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [12:40:33] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.167 second response time [12:41:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:01:32] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/93f1ab5c437b8017004d1aaa3eaa4b2db0e3e416 [13:01:33] 02puppet/03master 07paladox 0393f1ab5 mcrouter: lower proxy to 2 [13:05:21] Thank you @paladox 🙏 [13:07:18] well that's not supposed to fix it. But the problem wasn't like the other days. [13:09:24] !log installed nethogs on mw152 and db171 [13:09:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:13:06] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/2510fec53843 [13:13:07] 02puppet/03paladox-patch-2 07paladox 032510fec varnish: Increase max_connections to 5k [13:13:13] [02puppet] 07paladox opened pull request #4143: varnish: Increase max_connections to 5k (03master...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4143 [13:13:20] [02puppet] 07coderabbitai[bot] commented on pull request #4143: --- […] 13https://github.com/miraheze/puppet/pull/4143#issuecomment-2607218166 [13:13:24] [02puppet] 07paladox merged pull request #4143: varnish: Increase max_connections to 5k (03master...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4143 [13:13:24] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/85af149a4aef35e25cdff4db4e8658935498d8db [13:13:25] 02puppet/03master 07paladox 0385af149 varnish: Increase max_connections to 5k (#4143) [13:13:27] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 032510fec 13https://github.com/miraheze/puppet/commit/2510fec [13:22:03] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/52d8858fcc48 [13:22:03] 02puppet/03paladox-patch-2 07paladox 0352d8858 mediawiki: update php-fpm config… [13:22:08] [02puppet] 07paladox opened pull request #4144: mediawiki: update php-fpm config (03master...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4144 [13:22:14] [02puppet] 07coderabbitai[bot] commented on pull request #4144: --- […] 13https://github.com/miraheze/puppet/pull/4144#issuecomment-2607238496 [13:22:38] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/2f49cc6b681fe0df65cde0063f89aa02d51f10d3 [13:22:38] 02puppet/03paladox-patch-2 07paladox 032f49cc6 Update mediawiki.yaml [13:25:50] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 13.00, 8.22, 6.10 [13:26:03] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/7a8eda4899ae39989427ac9f6fe11af467edae55 [13:26:03] 02puppet/03paladox-patch-2 07paladox 037a8eda4 Update php.pp [13:26:59] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/99aac32eba64beae5627f6cec65fd67435d03cff [13:26:59] 02puppet/03paladox-patch-2 07paladox 0399aac32 Update mediawiki_beta.yaml [13:27:39] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/5d06be13606efdc30db773df231eca896b99801e [13:27:39] 02puppet/03paladox-patch-2 07paladox 035d06be1 Update mediawiki_task.yaml [13:27:50] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.38, 7.71, 6.18 [13:31:50] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 6.28, 6.68, 6.09 [13:38:28] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/d881d23e79f9f65d5c3c682582b4c16bf1be4e1b [13:38:28] 02puppet/03paladox-patch-2 07paladox 03d881d23 Update php.pp [13:38:49] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/cfef9cdb9d72d8886d6f5c7c5fad609c6decee33 [13:38:50] 02puppet/03paladox-patch-2 07paladox 03cfef9cd Update mediawiki.yaml [13:39:40] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/b360784574d14d7fba575197b48362089c84a563 [13:39:40] 02puppet/03paladox-patch-2 07paladox 03b360784 Update mediawiki_beta.yaml [13:41:15] [02puppet] 07paladox merged pull request #4144: mediawiki: update php-fpm config (03master...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4144 [13:41:16] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/2b136d3a6b66800ee67fb57be42f7e17e3da18fe [13:41:17] 02puppet/03master 07paladox 032b136d3 mediawiki: update php-fpm config (#4144)… [13:41:19] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 03b360784 13https://github.com/miraheze/puppet/commit/b360784 [13:49:50] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/8d3cce581080 [13:49:51] 02puppet/03paladox-patch-2 07paladox 038d3cce5 mediawiki: install php-excimer [13:49:57] [02puppet] 07paladox opened pull request #4145: mediawiki: install php-excimer (03master...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4145 [13:50:08] [02puppet] 07coderabbitai[bot] commented on pull request #4145: --- […] 13https://github.com/miraheze/puppet/pull/4145#issuecomment-2607302093 [13:51:44] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/3349a2826d57ebc23c30c8c99b26bbb6464238cc [13:51:44] 02puppet/03paladox-patch-2 07paladox 033349a28 Update php.pp [13:52:57] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:53:01] PROBLEM - mw183 Puppet on mw183 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:53:02] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:53:04] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:53:11] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:53:17] PROBLEM - mwtask161 Puppet on mwtask161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:54:19] [02puppet] 07paladox merged pull request #4145: mediawiki: install php-excimer (03master...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4145 [13:54:19] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/57f684fa9a4eb055f8ae89ac423916322abf155c [13:54:21] 02puppet/03master 07paladox 0357f684f mediawiki: install php-excimer (#4145) [13:54:22] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 033349a28 13https://github.com/miraheze/puppet/commit/3349a28 [13:54:57] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:55:00] RECOVERY - mw183 Puppet on mw183 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:55:02] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:55:03] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:55:11] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:55:18] RECOVERY - mwtask161 Puppet on mwtask161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:10:30] [02mw-config] 07BlankEclair opened pull request #5785: T13108: Set $wgMinervaNightMode for roguetown2ewiki (07miraheze:03master...07BlankEclair:03T13108) 13https://github.com/miraheze/mw-config/pull/5785 [14:11:23] miraheze/mw-config - BlankEclair the build passed. [14:51:20] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.27, 6.33, 6.07 [14:52:01] [02CreateWiki] 07BlankEclair left a file comment in pull request #651 0305cabac: why are we bringing html into this 13https://github.com/miraheze/CreateWiki/pull/651#discussion_r1925455362 [14:52:01] [02CreateWiki] 07BlankEclair left a file comment in pull request #651 0305cabac: ```suggestion […] 13https://github.com/miraheze/CreateWiki/pull/651#discussion_r1925450573 [14:52:02] [02CreateWiki] 07BlankEclair left a file comment in pull request #651 0305cabac: Missing indentation 13https://github.com/miraheze/CreateWiki/pull/651#discussion_r1925452146 [14:53:16] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 5.08, 5.78, 5.90 [14:58:35] [02mw-config] 07PrabhasDaily opened pull request #5786: Updated footer icons for jbcstudios (07miraheze:03master...07PrabhasDaily:03patch-2) 13https://github.com/miraheze/mw-config/pull/5786 [15:09:09] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/82fac4782bd7b1603a9b93d0cff6ea96d1383d9d [15:09:09] 02puppet/03master 07paladox 0382fac47 mediawiki_task: increase request timeout by 1 [15:17:05] miraheze/mw-config - PrabhasDaily the build passed. [15:17:17] [02mw-config] 07OAuthority merged 07BlankEclair's pull request #5785: T13108: Set $wgMinervaNightMode for roguetown2ewiki (07miraheze:03master...07BlankEclair:03T13108) 13https://github.com/miraheze/mw-config/pull/5785 [15:17:19] [02mw-config] 07OAuthority pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/88d1a9fdda0cb9d1220a7a398f202748b004e97a [15:17:21] 02mw-config/03master 07Claire 0388d1a9f T13108: Set $wgMinervaNightMode for roguetown2ewiki (#5785)… [15:17:33] [02mw-config] 07OAuthority merged 07PrabhasDaily's pull request #5786: Updated footer icons for jbcstudios (07miraheze:03master...07PrabhasDaily:03patch-2) 13https://github.com/miraheze/mw-config/pull/5786 [15:17:33] [02mw-config] 07OAuthority pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/b9def6d7715cce5d7301054e37d60c1ddc0209ea [15:17:34] 02mw-config/03master 07PrabhasXD 03b9def6d Updated footer icons for jbcstudios (#5786)… [15:18:13] miraheze/mw-config - OAuthority the build passed. [15:18:31] miraheze/mw-config - OAuthority the build passed. [15:27:05] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.48, 7.25, 6.48 [15:29:00] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 6.45, 6.73, 6.37 [15:48:04] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 90%, RTA = 142.92 ms [15:50:07] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 143.00 ms [16:26:43] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 9.70, 7.99, 6.51 [16:28:39] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.08, 7.19, 6.39 [16:36:22] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 6.12, 6.61, 6.49 [16:56:23] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/25ffeec0c32f8bbed69d5ab85955d0695c38270f [16:56:24] 02ssl/03master 07MacFan4000 0325ffeec add wiki.animalroyale.com [17:15:09] [02ssl] 07MacFan4000 created 03MacFan4000-patch-1 (+1 new commit) 13https://github.com/miraheze/ssl/commit/f3870d132f8f [17:15:09] 02ssl/03MacFan4000-patch-1 07MacFan4000 03f3870d1 Update certs.yaml [17:15:24] [02ssl] 07MacFan4000 opened pull request #819: remove expired/non-pointed domains (03master...03MacFan4000-patch-1) 13https://github.com/miraheze/ssl/pull/819 [17:16:33] [02ssl] 07coderabbitai[bot] commented on pull request #819: The pull request involves the removal of six domain entries from the `certs.yaml` configuration file. These entries include various domains such as `patternarchiveonline`, `www.pomologia.org`, `ru-teirailway.f5.si`, `starveil.wiki`, […] 13https://github.com/miraheze/ssl/pull/819#issuecomment-2607819076 [17:17:49] [02ssl] 07MacFan4000 pushed 1 new commit to 03MacFan4000-patch-1 13https://github.com/miraheze/ssl/commit/763f251f18e9e29f38c34fa56d96f4e3ec0e96ab [17:17:49] 02ssl/03MacFan4000-patch-1 07MacFan4000 03763f251 Update redirects.yaml [17:18:24] [02ssl] 07MacFan4000 pushed 1 new commit to 03MacFan4000-patch-1 13https://github.com/miraheze/ssl/commit/f8e61b1c68c577d83c55510939ee0a5cccb7f552 [17:18:25] 02ssl/03MacFan4000-patch-1 07MacFan4000 03f8e61b1 Delete certificates/starveil.wiki.crt [17:18:49] [02ssl] 07MacFan4000 pushed 1 new commit to 03MacFan4000-patch-1 13https://github.com/miraheze/ssl/commit/c2243071e421a9ba06d93a5d94a838f493ac24ba [17:18:49] 02ssl/03MacFan4000-patch-1 07MacFan4000 03c224307 Delete certificates/patternarchive.online.crt [17:19:04] [02ssl] 07MacFan4000 pushed 1 new commit to 03MacFan4000-patch-1 13https://github.com/miraheze/ssl/commit/4d4c47a96672d26bc42877a85a0bc278a437591e [17:19:05] 02ssl/03MacFan4000-patch-1 07MacFan4000 034d4c47a Delete certificates/lostidols.wiki.crt [17:19:17] [02ssl] 07MacFan4000 pushed 1 new commit to 03MacFan4000-patch-1 13https://github.com/miraheze/ssl/commit/c87e4c7af4d3ff961bcf47f6bcc6fb9e805bc0d8 [17:19:18] 02ssl/03MacFan4000-patch-1 07MacFan4000 03c87e4c7 Delete certificates/www.gengbaike.top.crt [17:22:15] [02ssl] 07MacFan4000 merged pull request #819: remove expired/non-pointed domains (03master...03MacFan4000-patch-1) 13https://github.com/miraheze/ssl/pull/819 [17:22:15] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/b669f094f0d4320b720fd21737aff9a60dff2c48 [17:22:16] 02ssl/03master 07MacFan4000 03b669f09 remove expired/non-pointed domains (#819) [17:22:18] [02ssl] 07MacFan4000 04deleted 03MacFan4000-patch-1 at 03c87e4c7 13https://github.com/miraheze/ssl/commit/c87e4c7 [17:25:30] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 12.49, 7.82, 6.45 [17:26:43] [02dns] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/8e9167d55eacab1110b8563d44530d9bce1f2932 [17:26:44] 02dns/03master 07MacFan4000 038e9167d Delete zones/starveil.wiki [17:27:07] [02dns] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/a2ba9f4259f64769b5cfaf6138096d21768fe5be [17:27:07] 02dns/03master 07MacFan4000 03a2ba9f4 Delete zones/satepedia.com [17:27:29] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.20, 7.24, 6.40 [17:27:31] [02dns] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/05ab8864e5ec726b4a03a80ddeee678f84b4ee61 [17:27:31] 02dns/03master 07MacFan4000 0305ab886 Delete zones/patternarchive.online [17:27:40] miraheze/dns - MacFan4000 the build passed. [17:27:43] [02dns] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/38b240a26e16b9871846b527ac4b3e962586368a [17:27:43] 02dns/03master 07MacFan4000 0338b240a Delete zones/lostidols.wiki [17:28:02] miraheze/dns - MacFan4000 the build passed. [17:28:03] [02dns] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/9b3ecdf9207d1944141b648dc449f6d8f5719d99 [17:28:03] 02dns/03master 07MacFan4000 039b3ecdf Delete zones/pomologia.org [17:28:27] miraheze/dns - MacFan4000 the build passed. [17:28:37] miraheze/dns - MacFan4000 the build passed. [17:28:58] miraheze/dns - MacFan4000 the build passed. [17:29:28] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 5.10, 6.56, 6.25 [17:45:45] [02CreateWiki] 07AgentIsai left a review on pull request #651 13https://github.com/miraheze/CreateWiki/pull/651#pullrequestreview-2567888316 [17:45:45] [02CreateWiki] 07AgentIsai left a file comment in pull request #651 0305cabac: `confidence` returns an interger from 0 to 100 so I added that percentage sign on purpose for readability. 13https://github.com/miraheze/CreateWiki/pull/651#discussion_r1925728008 [17:46:19] [02CreateWiki] 07AgentIsai left a file comment in pull request #651 0305cabac: Extra security against inputs in RequestWiki. Do you think this is not necessary? 13https://github.com/miraheze/CreateWiki/pull/651#discussion_r1925728702 [17:46:19] [02CreateWiki] 07AgentIsai left a review on pull request #651 13https://github.com/miraheze/CreateWiki/pull/651#pullrequestreview-2567889435 [17:47:50] [02CreateWiki] 07AgentIsai pushed 1 new commit to 03ai-improvements 13https://github.com/miraheze/CreateWiki/commit/be04bb96d6aae4d1dc40eea98dfac66747aca3b0 [17:47:50] 02CreateWiki/03ai-improvements 07Agent Isai 03be04bb9 Fix indentations [17:58:49] miraheze/CreateWiki - AgentIsai the build passed. [18:34:31] [02mw-config] 07Reception123 created 03Reception123-patch-1 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/7d4ff19bd224 [18:34:31] 02mw-config/03Reception123-patch-1 07Reception123 037d4ff19 add NumberHeadings per T12989 [18:34:35] [02mw-config] 07Reception123 opened pull request #5787: add NumberHeadings per T12989 (03master...03Reception123-patch-1) 13https://github.com/miraheze/mw-config/pull/5787 [18:35:08] [02mw-config] 07github-actions[bot] pushed 1 new commit to 03Reception123-patch-1 13https://github.com/miraheze/mw-config/commit/9327032cc01eaa8e47018127588b8aac1abe8633 [18:35:10] 02mw-config/03Reception123-patch-1 07github-actions 039327032 CI: lint code to MediaWiki standards… [18:35:41] miraheze/mw-config - Reception123 the build passed. [18:38:01] [02mw-config] 07Reception123 merged pull request #5787: add NumberHeadings per T12989 (03master...03Reception123-patch-1) 13https://github.com/miraheze/mw-config/pull/5787 [18:38:01] [02mw-config] 07Reception123 pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/ac3bb6d89fff497399d65fe93dd71300d1edd7c2 [18:38:01] 02mw-config/03master 07Reception123 03ac3bb6d add NumberHeadings per T12989 (#5787)… [18:38:03] [02mw-config] 07Reception123 04deleted 03Reception123-patch-1 at 039327032 13https://github.com/miraheze/mw-config/commit/9327032 [18:38:23] !log [reception@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [18:38:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:38:46] !log [reception@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 23s [18:38:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:39:00] miraheze/mw-config - Reception123 the build passed. [18:39:02] !log [reception@mwtask181] starting deploy of {'world': True, 'l10n': True, 'extension_list': True, 'force': True, 'versions': '1.42'} to all [18:39:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:48:28] !log [reception@mwtask181] finished deploy of {'world': True, 'l10n': True, 'extension_list': True, 'force': True, 'versions': '1.42'} to all - SUCCESS in 565s [18:48:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:28] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 9.78, 7.96, 6.35 [18:59:28] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.47, 7.59, 6.43 [19:03:28] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 6.71, 6.74, 6.30 [19:27:13] PROBLEM - swiftobject171 Disk Space on swiftobject171 is CRITICAL: DISK CRITICAL - free space: / 80122MiB (5% inode=84%); [21:49:03] [02CreateWiki] 07BlankEclair left a file comment in pull request #651 03be04bb9: You can use `str_replace( '\'', '\\\'', $text )` (and escape backslashes, newlines, potentially control characters, etc) […] 13https://github.com/miraheze/CreateWiki/pull/651#discussion_r1926021222 [21:55:08] Morning BlankEclair [21:55:13] mornyan~ [21:57:28] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.44, 6.76, 5.67 [21:59:21] * RhinosF1 is about to go to sleep. This is called a sensible time. [21:59:28] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 5.35, 6.25, 5.62 [21:59:42] [02CreateWiki] 07BlankEclair left a file comment in pull request #651 03be04bb9: This should be casted to an int, otherwise the functions that take this in will die if they're type hinted 13https://github.com/miraheze/CreateWiki/pull/651#discussion_r1926035430 [22:06:11] https://github.com/Servermanagementpanel/Website/blob/main/src/FormHandler.php [22:06:11] [GitHub] [Servermanagementpanel/Website] src/FormHandler.php @ main [22:06:14] ah yes, code quality [23:01:14] Mood [23:02:14] It doesn’t look thaaaaat bad [23:02:15] Minus [23:02:17] Ya know [23:02:28] The $_SESSION and others [23:02:29] yeah, not too bad [23:02:39] but not great tbh [23:02:50] sometimes, queerphobia is funny [23:03:19] read a conversation like 1.5 years ago where somebody tried to tell me that bisexuals aren't a thing and that it's a conversion stage to 'picking a side' [23:03:32] i messaged them 1.5y later asking them when the conversion is [23:04:20] Oh yeah did you know all cis people in the US are now legally non binary [23:04:31] oh yeahhh [23:04:34] because at conception xD [23:05:11] Hilarious [23:05:18] Off to a great start