[00:00:07] RECOVERY - mwtask181 Check unit status of update-static-tech-docs on mwtask181 is OK: OK: Status of the systemd unit update-static-tech-docs [00:04:07] PROBLEM - mwtask181 Check unit status of update-static-tech-docs on mwtask181 is CRITICAL: CRITICAL: Status of the systemd unit update-static-tech-docs [00:30:51] RECOVERY - speedscope211 Check unit status of speedscope_daily_aggregation on speedscope211 is OK: OK: Status of the systemd unit speedscope_daily_aggregation [00:46:51] PROBLEM - speedscope211 Check unit status of speedscope_daily_aggregation on speedscope211 is CRITICAL: CRITICAL: Status of the systemd unit speedscope_daily_aggregation [00:51:44] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 86434MiB (19.1% inode=100%); [01:11:42] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 86136MiB (19.0% inode=100%); [02:57:58] PROBLEM - db161 mariadb_memory on db161 is WARNING: WARN Memory 90% used. Largest process: mariadbd (1639) = 88.5% [05:13:20] [02mw-config] 07whostacking opened pull request #6552: Add custom Discord embed to specific utgwiki pages (07miraheze:03main...07whostacking:03utg-disc-embed) 13https://github.com/miraheze/mw-config/pull/6552 [05:15:04] [02mw-config] 07Universal-Omega approved pull request #6552 13https://github.com/miraheze/mw-config/pull/6552#pullrequestreview-5254731373 [05:15:12] [02mw-config] 07Universal-Omega merged 07whostacking's pull request #6552: Add custom Discord embed to specific utgwiki pages (07miraheze:03main...07whostacking:03utg-disc-embed) 13https://github.com/miraheze/mw-config/pull/6552 [05:15:12] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/7f7b947edc75088a395638ca8cf9df1db6d1eaa0 [05:15:12] 02mw-config/03main 07whostacking 037f7b947 Add custom Discord embed to specific utgwiki pages (#6552)… [05:16:23] miraheze/mw-config - Universal-Omega the build passed. [05:16:25] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [05:16:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:16:55] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 30s [05:17:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:32:53] [02mw-config] 07whostacking opened pull request #6553: Minor fix to #6552 (07miraheze:03main...07whostacking:03utg-disc-embed) 13https://github.com/miraheze/mw-config/pull/6553 [05:34:14] [02mw-config] 07Universal-Omega merged 07whostacking's pull request #6553: Minor fix to #6552 (07miraheze:03main...07whostacking:03utg-disc-embed) 13https://github.com/miraheze/mw-config/pull/6553 [05:34:14] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/f3be8217f5c2d67c5f4744f1fd9a235b40d4b7a4 [05:34:14] 02mw-config/03main 07whostacking 03f3be821 Minor fix to #6552 (#6553) [05:34:20] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [05:34:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:34:35] miraheze/mw-config - whostacking the build passed. [05:34:46] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 27s [05:34:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:35:31] miraheze/mw-config - Universal-Omega the build passed. [06:19:55] [02TSPortal] 07renovate[bot] 04force-pushed 03renovate/prod-deps from 03ca7dd3f to 038d80659 13https://api.github.com/repos/miraheze/TSPortal/commits/renovate/prod-deps [06:19:57] 02TSPortal/03renovate/prod-deps 07renovate[bot] 038d80659 Update prod dependencies [06:24:05] [02TSPortal] 07Universal-Omega merged 07renovate[bot]'s pull request #123: Update prod dependencies (03main...03renovate/prod-deps) 13https://github.com/miraheze/TSPortal/pull/123 [06:24:05] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/TSPortal/commit/cfda34a3ef2328a9845e908efb5b708ff9570f26 [06:24:05] 02TSPortal/03main 07renovate[bot] 03cfda34a Update prod dependencies (#123)… [06:24:07] [02TSPortal] 07Universal-Omega 04deleted 03renovate/prod-deps at 038d80659 13https://api.github.com/repos/miraheze/TSPortal/commit/8d80659 [09:30:24] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/69d2028bd6f2a2f9641c2cba00f9a12f7bc2092f [09:30:24] 02ssl/03main 07WikiTideBot 0369d2028 Bot: Auto-update domain lists [09:47:38] [02mw-config] 07oksouhhhhhhhhh opened pull request #6554: Update LocalWiki.php (07miraheze:03main...07oksouhhhhhhhhh:03patch-4) 13https://github.com/miraheze/mw-config/pull/6554 [09:48:49] miraheze/mw-config - oksouhhhhhhhhh the build passed. [09:50:58] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 8.36, 6.43, 3.64 [09:52:58] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 7.85, 6.91, 4.16 [09:54:55] [02mw-config] 07oksouhhhhhhhhh opened pull request #6555: Update LocalSettings.php (07miraheze:03main...07oksouhhhhhhhhh:03patch-5) 13https://github.com/miraheze/mw-config/pull/6555 [09:55:52] miraheze/mw-config - oksouhhhhhhhhh the build passed. [10:00:58] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 2.46, 5.77, 4.93 [12:00:22] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/2605ed0768e6c01c4f4d83d27de06c11445d415b [12:00:23] 02ssl/03main 07WikiTideBot 032605ed0 Bot: Auto-update domain lists [15:07:07] PROBLEM - cloud17 APT on cloud17 is CRITICAL: APT CRITICAL: 191 packages available for upgrade (1 critical updates). [15:17:36] PROBLEM - cloud15 APT on cloud15 is CRITICAL: APT CRITICAL: 197 packages available for upgrade (1 critical updates). [15:21:28] PROBLEM - cloud21 APT on cloud21 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (1 critical updates). [15:24:54] PROBLEM - cloud19 APT on cloud19 is CRITICAL: APT CRITICAL: 197 packages available for upgrade (1 critical updates). [15:31:27] PROBLEM - cloud18 APT on cloud18 is CRITICAL: APT CRITICAL: 191 packages available for upgrade (1 critical updates). [15:34:30] PROBLEM - cloud20 APT on cloud20 is CRITICAL: APT CRITICAL: 197 packages available for upgrade (1 critical updates). [15:36:23] PROBLEM - cloud16 APT on cloud16 is CRITICAL: APT CRITICAL: 191 packages available for upgrade (1 critical updates). [16:31:58] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 20.72, 19.33, 14.81 [16:33:58] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 19.31, 18.85, 15.16 [17:21:40] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [17:21:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:22:32] miraheze/mw-config - Universal-Omega the build passed. [17:23:30] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [17:23:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:24:00] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 30s [17:24:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:41:02] miraheze/mw-config - Universal-Omega the build passed. [19:16:28] miraheze/ManageWiki - Universal-Omega the build passed. [20:34:06] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:34:34] PROBLEM - mw173 SSH on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:35:28] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [20:36:48] PROBLEM - mw173 Check unit status of php8.4-fpm_check_restart on mw173 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:37:24] PROBLEM - mw173 APT on mw173 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:37:42] PROBLEM - mw173 conntrack_table_size on mw173 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:37:44] PROBLEM - mw173 Current Load on mw173 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:37:47] PROBLEM - mw173 PowerDNS Recursor on mw173 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:37:50] PROBLEM - mw173 Puppet on mw173 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:37:53] PROBLEM - mw173 Disk Space on mw173 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:38:05] PROBLEM - mw173 php-fpm on mw173 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:38:19] PROBLEM - mw173 nftables_active on mw173 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:38:36] PROBLEM - mw173 chrony on mw173 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:42:13] RECOVERY - mw173 Check unit status of php8.4-fpm_check_restart on mw173 is OK: OK: Status of the systemd unit php8.4-fpm_check_restart [20:42:21] RECOVERY - mw173 APT on mw173 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [20:42:29] RECOVERY - mw173 conntrack_table_size on mw173 is OK: OK: nf_conntrack is 1 % full [20:42:44] RECOVERY - mw173 Puppet on mw173 is OK: OK: Puppet is currently enabled, last run 28 minutes ago with 0 failures [20:42:46] RECOVERY - mw173 PowerDNS Recursor on mw173 is OK: DNS OK: 0.053 seconds response time. mw173.fsslc.wtnet returns 10.0.17.153 [20:42:51] RECOVERY - mw173 Disk Space on mw173 is OK: DISK OK - free space: / 23591MiB (45.5% inode=84%); [20:42:52] RECOVERY - mw173 SSH on mw173 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [20:43:04] RECOVERY - mw173 php-fpm on mw173 is OK: PROCS OK: 25 processes with command name 'php-fpm8.4' [20:43:08] RECOVERY - mw173 nftables_active on mw173 is OK: OK nftables input default policy is set [20:43:23] RECOVERY - mw173 chrony on mw173 is OK: chrony OK: System Time 0.000039237 seconds fast of NTP time, Stratum 4 [20:43:39] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.077 second response time [20:44:15] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.193 second response time [20:54:10] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 6.74, 12.75, 22.83 [20:58:02] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 9.57, 10.60, 19.64 [21:42:20] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 45247MiB (10.0% inode=100%); [21:51:40] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 45228MiB (10.0% inode=100%); [21:58:20] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 86140MiB (19.0% inode=100%); [22:10:10] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 44285MiB (9.7% inode=100%); [22:12:10] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 91111MiB (20.1% inode=100%);