[00:01:27] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/b4bd365b8d2d50543da5848ffe692dda7576eff3 [00:01:27] 02statichelp/03main 07WikiTideBot 03b4bd365 Bot: Auto-update Tech namespace pages 2026-09-25 00:01:25 [00:02:57] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 45266MiB (10.0% inode=100%); [00:03:23] PROBLEM - speedscope211 Check unit status of speedscope_hourly_aggregation on speedscope211 is CRITICAL: CRITICAL: Status of the systemd unit speedscope_hourly_aggregation [00:22:02] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 44091MiB (9.7% inode=100%); [00:29:24] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 8.08, 7.10, 11.81 [00:30:02] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 86814MiB (19.1% inode=100%); [00:33:24] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 4.53, 5.63, 10.14 [01:07:59] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 86081MiB (19.0% inode=100%); [01:12:57] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 85873MiB (18.9% inode=100%); [01:25:29] PROBLEM - mwtask181 Check unit status of mediawiki_job_generate-sitemap-index on mwtask181 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki_job_generate-sitemap-index [01:45:29] PROBLEM - speedscope211 Puppet on speedscope211 is WARNING: WARNING: Puppet last ran 1 hour ago [06:30:19] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/6bb234e7cefdf3b4f7c8817eab6d8f4ea2c6240a [06:30:19] 02ssl/03main 07WikiTideBot 036bb234e Bot: Auto-update domain lists [07:24:03] [02mw-config] 07Conscience93 opened pull request #6566: T16062: Add CSP for zhninjagowiki (07miraheze:03main...07Conscience93:03cnvideo-zhninjagowiki) 13https://github.com/miraheze/mw-config/pull/6566 [07:51:20] [02mw-config] 07Universal-Omega merged 07Conscience93's pull request #6566: T16062: Add CSP for zhninjagowiki (07miraheze:03main...07Conscience93:03cnvideo-zhninjagowiki) 13https://github.com/miraheze/mw-config/pull/6566 [07:51:20] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/422733264a49bd34ddaaff17efb2446ba33f52c2 [07:51:20] 02mw-config/03main 07Conscience93 034227332 T16062: Add CSP for zhninjagowiki (#6566)… [07:51:37] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [07:51:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:52:05] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 28s [07:52:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:52:20] miraheze/mw-config - Universal-Omega the build passed. [08:44:45] PROBLEM - db161 mariadb_memory on db161 is WARNING: WARN Memory 94% used. Largest process: mariadbd (1246794) = 92.7% [10:19:24] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 72.95, 34.15, 15.59 [10:21:37] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [10:21:55] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:22:05] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 1 backends are down. mw183 [10:22:17] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [10:22:19] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [10:22:23] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:26] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:22:33] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.012 second response time [10:22:35] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:22:42] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:22:46] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [10:22:47] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:51] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [10:23:06] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 6 backends are down. mw161 mw171 mw181 mw153 mw201 mw202 [10:23:07] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 6 backends are down. mw161 mw171 mw181 mw153 mw201 mw202 [10:23:13] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:33] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:56] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [10:24:23] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1519 bytes in 0.085 second response time [10:24:36] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:24:39] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 7.537 second response time [10:24:40] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.012 second response time [10:24:46] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 26% [10:24:48] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.023 second response time [10:24:55] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.241 second response time [10:25:07] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 34 backends are healthy [10:25:18] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:25:34] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.342 second response time [10:25:39] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 6.811 second response time [10:25:45] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 3.254 second response time [10:25:54] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 2.392 second response time [10:26:12] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.191 second response time [10:26:20] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.260 second response time [10:26:24] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.179 second response time [10:26:33] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.074 second response time [10:26:35] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.071 second response time [10:27:06] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 34 backends are healthy [10:27:15] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.082 second response time [10:27:17] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.219 second response time [10:28:05] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 34 backends are healthy [10:55:24] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 4.57, 5.37, 11.51 [10:59:24] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 4.68, 4.71, 9.81 [11:33:06] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 2 backends are down. mw181 mw183 [11:33:07] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw201 [11:35:07] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 34 backends are healthy [11:37:06] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 34 backends are healthy [12:08:06] PROBLEM - mem201 APT on mem201 is CRITICAL: APT CRITICAL: 112 packages available for upgrade (2 critical updates). [12:08:23] PROBLEM - cp201 APT on cp201 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [12:08:53] PROBLEM - mattermost2 APT on mattermost2 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [12:09:06] PROBLEM - cloud17 APT on cloud17 is CRITICAL: APT CRITICAL: 192 packages available for upgrade (2 critical updates). [12:09:39] PROBLEM - matomo151 APT on matomo151 is CRITICAL: APT CRITICAL: 115 packages available for upgrade (2 critical updates). [12:09:40] PROBLEM - changeprop202 APT on changeprop202 is CRITICAL: APT CRITICAL: 124 packages available for upgrade (2 critical updates). [12:10:04] PROBLEM - swiftproxy161 APT on swiftproxy161 is CRITICAL: APT CRITICAL: 126 packages available for upgrade (2 critical updates). [12:10:19] PROBLEM - db172 APT on db172 is CRITICAL: APT CRITICAL: 120 packages available for upgrade (2 critical updates). [12:11:05] PROBLEM - rdb191 APT on rdb191 is CRITICAL: APT CRITICAL: 115 packages available for upgrade (2 critical updates). [12:11:06] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (2 critical updates). [12:11:14] PROBLEM - mw202 APT on mw202 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (2 critical updates). [12:11:25] PROBLEM - bast161 APT on bast161 is CRITICAL: APT CRITICAL: 119 packages available for upgrade (2 critical updates). [12:11:38] PROBLEM - mw203 APT on mw203 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (2 critical updates). [12:12:01] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (2 critical updates). [12:12:17] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 188 packages available for upgrade (2 critical updates). [12:12:39] PROBLEM - puppet181 APT on puppet181 is CRITICAL: APT CRITICAL: 97 packages available for upgrade (2 critical updates). [12:12:46] PROBLEM - swiftproxy171 APT on swiftproxy171 is CRITICAL: APT CRITICAL: 126 packages available for upgrade (2 critical updates). [12:12:53] PROBLEM - mem161 APT on mem161 is CRITICAL: APT CRITICAL: 113 packages available for upgrade (2 critical updates). [12:13:13] PROBLEM - cp171 APT on cp171 is CRITICAL: APT CRITICAL: 118 packages available for upgrade (2 critical updates). [12:13:27] PROBLEM - llm191 APT on llm191 is CRITICAL: APT CRITICAL: 127 packages available for upgrade (2 critical updates). [12:13:31] PROBLEM - swiftobject191 APT on swiftobject191 is CRITICAL: APT CRITICAL: 131 packages available for upgrade (2 critical updates). [12:13:37] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 193 packages available for upgrade (2 critical updates). [12:14:11] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: APT CRITICAL: 192 packages available for upgrade (2 critical updates). [12:14:21] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: APT CRITICAL: 192 packages available for upgrade (2 critical updates). [12:14:50] PROBLEM - swiftobject201 APT on swiftobject201 is CRITICAL: APT CRITICAL: 131 packages available for upgrade (2 critical updates). [12:15:01] PROBLEM - mw173 APT on mw173 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (2 critical updates). [12:15:22] PROBLEM - phorge171 APT on phorge171 is CRITICAL: APT CRITICAL: 118 packages available for upgrade (2 critical updates). [12:15:42] PROBLEM - swiftobject181 APT on swiftobject181 is CRITICAL: APT CRITICAL: 131 packages available for upgrade (2 critical updates). [12:16:16] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: APT CRITICAL: 127 packages available for upgrade (2 critical updates). [12:17:06] PROBLEM - db161 APT on db161 is CRITICAL: APT CRITICAL: 119 packages available for upgrade (2 critical updates). [12:17:24] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 12.65, 9.99, 7.13 [12:17:49] PROBLEM - os201 APT on os201 is CRITICAL: APT CRITICAL: 121 packages available for upgrade (2 critical updates). [12:18:01] PROBLEM - os151 APT on os151 is CRITICAL: APT CRITICAL: 122 packages available for upgrade (2 critical updates). [12:18:16] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 115 packages available for upgrade (2 critical updates). [12:18:31] [02TSPortal] 07renovate[bot] created 03renovate/dev-deps (+1 new commit) 13https://github.com/miraheze/TSPortal/commit/e0a01969b718 [12:18:31] 02TSPortal/03renovate/dev-deps 07renovate[bot] 03e0a0196 Update dependency phpunit/phpunit to v13.3.5 [12:18:45] PROBLEM - mem151 APT on mem151 is CRITICAL: APT CRITICAL: 113 packages available for upgrade (2 critical updates). [12:19:25] PROBLEM - mw193 APT on mw193 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (2 critical updates). [12:19:46] PROBLEM - cloud21 APT on cloud21 is CRITICAL: APT CRITICAL: 190 packages available for upgrade (2 critical updates). [12:19:52] PROBLEM - cloud15 APT on cloud15 is CRITICAL: APT CRITICAL: 198 packages available for upgrade (2 critical updates). [12:21:06] PROBLEM - db171 APT on db171 is CRITICAL: APT CRITICAL: 120 packages available for upgrade (2 critical updates). [12:22:07] PROBLEM - mw191 APT on mw191 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (2 critical updates). [12:22:32] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (2 critical updates). [12:22:40] PROBLEM - db182 APT on db182 is CRITICAL: APT CRITICAL: 121 packages available for upgrade (2 critical updates). [12:23:24] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 6.99, 10.49, 8.68 [12:23:28] PROBLEM - kafka182 APT on kafka182 is CRITICAL: APT CRITICAL: 127 packages available for upgrade (2 critical updates). [12:23:42] Nuh uh db161 you cannot go hiding behind all the apt warnings with your high load 🥀 [12:23:46] PROBLEM - db201 APT on db201 is CRITICAL: APT CRITICAL: 126 packages available for upgrade (2 critical updates). [12:23:59] PROBLEM - mw153 APT on mw153 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (2 critical updates). [12:25:24] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 5.28, 8.73, 8.25 [12:25:56] PROBLEM - cloud19 APT on cloud19 is CRITICAL: APT CRITICAL: 198 packages available for upgrade (2 critical updates). [12:26:10] PROBLEM - bots171 APT on bots171 is CRITICAL: APT CRITICAL: 129 packages available for upgrade (2 critical updates). [12:26:36] PROBLEM - cp161 APT on cp161 is CRITICAL: APT CRITICAL: 116 packages available for upgrade (2 critical updates). [12:27:10] PROBLEM - cp191 APT on cp191 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [12:27:26] PROBLEM - mw192 APT on mw192 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (2 critical updates). [12:27:26] PROBLEM - swiftobject171 APT on swiftobject171 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (2 critical updates). [12:28:04] PROBLEM - os162 APT on os162 is CRITICAL: APT CRITICAL: 122 packages available for upgrade (2 critical updates). [12:28:15] PROBLEM - mem191 APT on mem191 is CRITICAL: APT CRITICAL: 112 packages available for upgrade (2 critical updates). [12:28:55] PROBLEM - mw163 APT on mw163 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (2 critical updates). [12:29:12] PROBLEM - graylog161 APT on graylog161 is CRITICAL: APT CRITICAL: 124 packages available for upgrade (2 critical updates). [12:30:20] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/de1b5713b834726b8adef4346718301005f8ce75 [12:30:20] 02ssl/03main 07WikiTideBot 03de1b571 Bot: Auto-update domain lists [12:30:36] PROBLEM - os202 APT on os202 is CRITICAL: APT CRITICAL: 127 packages available for upgrade (2 critical updates). [12:30:48] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (2 critical updates). [12:30:56] PROBLEM - db181 APT on db181 is CRITICAL: APT CRITICAL: 120 packages available for upgrade (2 critical updates). [12:31:57] PROBLEM - cloud18 APT on cloud18 is CRITICAL: APT CRITICAL: 192 packages available for upgrade (2 critical updates). [12:31:59] PROBLEM - os191 APT on os191 is CRITICAL: APT CRITICAL: 121 packages available for upgrade (2 critical updates). [12:32:07] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (2 critical updates). [12:32:20] PROBLEM - reports171 APT on reports171 is CRITICAL: APT CRITICAL: 118 packages available for upgrade (2 critical updates). [12:32:23] PROBLEM - db151 APT on db151 is CRITICAL: APT CRITICAL: 120 packages available for upgrade (2 critical updates). [12:32:49] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 72 packages available for upgrade (2 critical updates). [12:33:02] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 194 packages available for upgrade (2 critical updates). [12:33:27] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 197 packages available for upgrade (2 critical updates). [12:33:32] PROBLEM - swiftobject151 APT on swiftobject151 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (2 critical updates). [12:33:48] PROBLEM - eventgate182 APT on eventgate182 is CRITICAL: APT CRITICAL: 123 packages available for upgrade (2 critical updates). [12:33:57] PROBLEM - swiftobject161 APT on swiftobject161 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (2 critical updates). [12:34:17] PROBLEM - matomo211 APT on matomo211 is CRITICAL: APT CRITICAL: 72 packages available for upgrade (2 critical updates). [12:34:20] PROBLEM - os161 APT on os161 is CRITICAL: APT CRITICAL: 122 packages available for upgrade (2 critical updates). [12:34:40] PROBLEM - db192 APT on db192 is CRITICAL: APT CRITICAL: 126 packages available for upgrade (2 critical updates). [12:34:42] PROBLEM - cloud20 APT on cloud20 is CRITICAL: APT CRITICAL: 198 packages available for upgrade (2 critical updates). [12:35:28] PROBLEM - bast181 APT on bast181 is CRITICAL: APT CRITICAL: 120 packages available for upgrade (2 critical updates). [12:35:32] PROBLEM - mw201 APT on mw201 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (2 critical updates). [12:35:42] PROBLEM - cloud16 APT on cloud16 is CRITICAL: APT CRITICAL: 192 packages available for upgrade (2 critical updates). [12:36:04] PROBLEM - ldap171 APT on ldap171 is CRITICAL: APT CRITICAL: 115 packages available for upgrade (2 critical updates). [12:36:10] PROBLEM - swiftobject211 APT on swiftobject211 is CRITICAL: APT CRITICAL: 120 packages available for upgrade (2 critical updates). [12:36:28] PROBLEM - mon181 APT on mon181 is CRITICAL: APT CRITICAL: 111 packages available for upgrade (2 critical updates). [12:36:29] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (2 critical updates). [12:37:05] PROBLEM - mw183 APT on mw183 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (2 critical updates). [12:37:18] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (2 critical updates). [12:38:51] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [12:43:29] PROBLEM - speedscope211 Puppet on speedscope211 is CRITICAL: CRITICAL: Puppet last ran 12 hours ago [13:00:26] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/a9235f4358800a6c612dea23f3d1d232cd8f52c2 [13:00:26] 02ssl/03main 07WikiTideBot 03a9235f4 Bot: Auto-update domain lists [13:27:24] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 39.46, 20.46, 11.14 [13:39:24] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 6.69, 11.06, 11.45 [13:45:24] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 4.52, 7.23, 9.65 [15:00:26] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/d91b3256dcec7151d3ecb7ea189f4309a215d9d6 [15:00:26] 02ssl/03main 07WikiTideBot 03d91b325 Bot: Auto-update domain lists [15:22:12] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php CreateWiki:ManageInactiveWikis --wiki=loginwiki (END - exit=0) [15:22:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:54:22] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 40% [16:56:19] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 35% [18:09:27] !log [universalomega@mwtask181] starting deploy of {'versions': '1.46', 'upgrade_extensions': 'SubpageFun'} to all [18:09:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:10:35] !log [universalomega@mwtask181] finished deploy of {'versions': '1.46', 'upgrade_extensions': 'SubpageFun'} to all - SUCCESS in 68s [18:10:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:07] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/9a5422f5f799d6fd2fe8fe8dbb18a756c9e69f74 [18:23:07] 02puppet/03main 07CosmicAlpha 039a5422f Update Discord mapping for Jenny [18:23:32] @jenny_on_wiki FYI ^ [18:24:33] miraheze/puppet - Universal-Omega the build passed. [18:30:24] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/62a307f63186462005293e5f2189df950bd54d16 [18:30:24] 02ssl/03main 07WikiTideBot 0362a307f Bot: Auto-update domain lists [18:49:59] !log [universalomega@puppet181] Upgraded packages on bast161: jq and libjq1 [18:50:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:50:16] !log [universalomega@puppet181] Upgraded packages on cp191: jq and libjq1 [18:50:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:50:25] RECOVERY - cp191 APT on cp191 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [18:50:34] !log [universalomega@puppet181] Upgraded packages on cloud15: jq and libjq1 [18:50:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:50:51] !log [universalomega@puppet181] Upgraded packages on bots171: jq and libjq1 [18:50:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:50:58] RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 196 packages available for upgrade (0 critical updates). [18:51:12] !log [universalomega@puppet181] Upgraded packages on cp201: jq and libjq1 [18:51:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:51:28] !log [universalomega@puppet181] Upgraded packages on cloud21: jq and libjq1 [18:51:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:51:41] RECOVERY - cloud21 APT on cloud21 is OK: APT OK: 188 packages available for upgrade (0 critical updates). [18:51:44] RECOVERY - bast161 APT on bast161 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [18:51:45] !log [universalomega@puppet181] Upgraded packages on cp161: jq and libjq1 [18:51:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:52:05] !log [universalomega@puppet181] Upgraded packages on cloud16: jq and libjq1 [18:52:05] RECOVERY - bots171 APT on bots171 is OK: APT OK: 127 packages available for upgrade (0 critical updates). [18:52:06] RECOVERY - cp161 APT on cp161 is OK: APT OK: 114 packages available for upgrade (0 critical updates). [18:52:08] RECOVERY - cp201 APT on cp201 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [18:52:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:52:21] !log [universalomega@puppet181] Upgraded packages on db172: jq and libjq1 [18:52:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:52:40] !log [universalomega@puppet181] Upgraded packages on cp171: jq and libjq1 [18:52:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:52:59] !log [universalomega@puppet181] Upgraded packages on cloud19: jq and libjq1 [18:53:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:53:05] RECOVERY - cp171 APT on cp171 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [18:53:06] RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 190 packages available for upgrade (0 critical updates). [18:53:16] !log [universalomega@puppet181] Upgraded packages on db151: jq and libjq1 [18:53:35] !log [universalomega@puppet181] Upgraded packages on changeprop202: jq and libjq1 [18:53:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:53:55] !log [universalomega@puppet181] Upgraded packages on cloud20: jq and libjq1 [18:54:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:15] !log [universalomega@puppet181] Upgraded packages on cloud18: jq and libjq1 [18:54:16] RECOVERY - cloud20 APT on cloud20 is OK: APT OK: 196 packages available for upgrade (0 critical updates). [18:54:17] RECOVERY - db172 APT on db172 is OK: APT OK: 118 packages available for upgrade (0 critical updates). [18:54:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:29] RECOVERY - changeprop202 APT on changeprop202 is OK: APT OK: 122 packages available for upgrade (0 critical updates). [18:54:35] !log [universalomega@puppet181] Upgraded packages on cloud17: jq and libjq1 [18:54:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:45] RECOVERY - cloud19 APT on cloud19 is OK: APT OK: 196 packages available for upgrade (0 critical updates). [18:54:48] RECOVERY - db151 APT on db151 is OK: APT OK: 118 packages available for upgrade (0 critical updates). [18:54:55] !log [universalomega@puppet181] Upgraded packages on db182: jq and libjq1 [18:54:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:55:13] !log [universalomega@puppet181] Upgraded packages on db161: jq and libjq1 [18:55:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:55:25] RECOVERY - db161 APT on db161 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [18:55:36] !log [universalomega@puppet181] Upgraded packages on db192: jq and libjq1 [18:55:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:55:54] !log [universalomega@puppet181] Upgraded packages on db171: jq and libjq1 [18:55:58] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 190 packages available for upgrade (0 critical updates). [18:55:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:56:02] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 190 packages available for upgrade (0 critical updates). [18:56:15] !log [universalomega@puppet181] Upgraded packages on bast181: jq and libjq1 [18:56:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:56:27] RECOVERY - db182 APT on db182 is OK: APT OK: 119 packages available for upgrade (0 critical updates). [18:56:29] RECOVERY - db192 APT on db192 is OK: APT OK: 124 packages available for upgrade (0 critical updates). [18:56:30] RECOVERY - bast181 APT on bast181 is OK: APT OK: 118 packages available for upgrade (0 critical updates). [18:56:34] !log [universalomega@puppet181] Upgraded packages on graylog161: jq and libjq1 [18:56:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:56:57] !log [universalomega@puppet181] Upgraded packages on db201: jq and libjq1 [18:57:00] RECOVERY - db171 APT on db171 is OK: APT OK: 118 packages available for upgrade (0 critical updates). [18:57:00] RECOVERY - db201 APT on db201 is OK: APT OK: 124 packages available for upgrade (0 critical updates). [18:57:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:16] !log [universalomega@puppet181] Upgraded packages on eventgate182: jq and libjq1 [18:57:16] RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 122 packages available for upgrade (0 critical updates). [18:57:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:38] !log [universalomega@puppet181] Upgraded packages on db181: jq and libjq1 [18:57:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:49] RECOVERY - eventgate182 APT on eventgate182 is OK: APT OK: 121 packages available for upgrade (0 critical updates). [18:57:50] !log [universalomega@puppet181] Upgraded packages on mattermost2: jq and libjq1 [18:57:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:58:08] !log [universalomega@puppet181] Upgraded packages on matomo151: jq and libjq1 [18:58:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:58:22] !log [universalomega@mwtask181] starting deploy of {'versions': '1.46', 'upgrade_extensions': 'RandomSelection'} to all [18:58:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:58:30] !log [universalomega@puppet181] Upgraded packages on kafka182: jq and libjq1 [18:58:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:58:47] !log [universalomega@puppet181] Upgraded packages on matomo211: jq and libjq1 [18:58:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:58:52] RECOVERY - mattermost2 APT on mattermost2 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [18:59:05] !log [universalomega@puppet181] Upgraded packages on mem161: jq and libjq1 [18:59:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:59:17] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 113 packages available for upgrade (0 critical updates). [18:59:17] RECOVERY - db181 APT on db181 is OK: APT OK: 118 packages available for upgrade (0 critical updates). [18:59:23] !log [universalomega@puppet181] Upgraded packages on mem151: jq and libjq1 [18:59:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:59:35] !log [universalomega@mwtask181] finished deploy of {'versions': '1.46', 'upgrade_extensions': 'RandomSelection'} to all - SUCCESS in 72s [18:59:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:59:45] !log [universalomega@puppet181] Upgraded packages on mw151: jq and libjq1 [18:59:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:00:03] !log [universalomega@puppet181] Upgraded packages on mon181: jq and libjq1 [19:00:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:00:13] RECOVERY - mem161 APT on mem161 is OK: APT OK: 111 packages available for upgrade (0 critical updates). [19:00:17] RECOVERY - matomo211 APT on matomo211 is OK: APT OK: 70 packages available for upgrade (0 critical updates). [19:00:17] RECOVERY - mem151 APT on mem151 is OK: APT OK: 111 packages available for upgrade (0 critical updates). [19:00:22] !log [universalomega@puppet181] Upgraded packages on mem201: jq and libjq1 [19:00:23] RECOVERY - kafka182 APT on kafka182 is OK: APT OK: 125 packages available for upgrade (0 critical updates). [19:00:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:00:44] !log [universalomega@puppet181] Upgraded packages on mw152: jq and libjq1 [19:00:48] RECOVERY - mw152 APT on mw152 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:00:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:01:00] RECOVERY - mw151 APT on mw151 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:01:01] !log [universalomega@puppet181] Upgraded packages on llm191: jq and libjq1 [19:01:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:01:23] RECOVERY - llm191 APT on llm191 is OK: APT OK: 125 packages available for upgrade (0 critical updates). [19:01:23] !log [universalomega@puppet181] Upgraded packages on mw161: jq and libjq1 [19:01:28] RECOVERY - mon181 APT on mon181 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [19:01:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:01:46] !log [universalomega@puppet181] Upgraded packages on mw153: jq and libjq1 [19:01:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:01:59] RECOVERY - mw161 APT on mw161 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:02:09] !log [universalomega@puppet181] Upgraded packages on mw171: jq and libjq1 [19:02:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:02:17] RECOVERY - mem201 APT on mem201 is OK: APT OK: 110 packages available for upgrade (0 critical updates). [19:02:32] !log [universalomega@puppet181] Upgraded packages on mw163: jq and libjq1 [19:02:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:02:55] !log [universalomega@puppet181] Upgraded packages on mw173: jq and libjq1 [19:02:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:03:10] RECOVERY - mw153 APT on mw153 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:03:11] RECOVERY - mw173 APT on mw173 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:03:14] RECOVERY - mw163 APT on mw163 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:03:17] !log [universalomega@puppet181] Upgraded packages on mw162: jq and libjq1 [19:03:20] RECOVERY - mw171 APT on mw171 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:03:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:03:42] !log [universalomega@puppet181] Upgraded packages on ldap171: jq and libjq1 [19:03:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:04:04] !log [universalomega@puppet181] Upgraded packages on mw172: jq and libjq1 [19:04:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:04:27] !log [universalomega@puppet181] Upgraded packages on mw181: jq and libjq1 [19:04:28] RECOVERY - mw162 APT on mw162 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:04:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:04:50] !log [universalomega@puppet181] Upgraded packages on mw182: jq and libjq1 [19:04:55] RECOVERY - mw182 APT on mw182 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:04:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:05:03] RECOVERY - ldap171 APT on ldap171 is OK: APT OK: 113 packages available for upgrade (0 critical updates). [19:05:12] !log [universalomega@puppet181] Upgraded packages on mw191: jq and libjq1 [19:05:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:05:36] RECOVERY - mw191 APT on mw191 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:05:37] !log [universalomega@puppet181] Upgraded packages on mw183: jq and libjq1 [19:05:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:05:54] RECOVERY - mw172 APT on mw172 is OK: APT OK: 186 packages available for upgrade (0 critical updates). [19:05:57] !log [universalomega@puppet181] Upgraded packages on mem191: jq and libjq1 [19:06:00] RECOVERY - mw181 APT on mw181 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:06:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:06:20] !log [universalomega@puppet181] Upgraded packages on mw192: jq and libjq1 [19:06:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:06:42] !log [universalomega@puppet181] Upgraded packages on mw193: jq and libjq1 [19:06:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:04] !log [universalomega@puppet181] Upgraded packages on mw201: jq and libjq1 [19:07:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:15] RECOVERY - mw183 APT on mw183 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:07:16] RECOVERY - mw193 APT on mw193 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:07:23] !log [universalomega@puppet181] Upgraded packages on ns1: jq and libjq1 [19:07:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:46] !log [universalomega@puppet181] Upgraded packages on mwtask151: jq and libjq1 [19:07:50] RECOVERY - mem191 APT on mem191 is OK: APT OK: 110 packages available for upgrade (0 critical updates). [19:07:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:06] RECOVERY - mw192 APT on mw192 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:08:08] !log [universalomega@puppet181] Upgraded packages on mwtask171: jq and libjq1 [19:08:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:25] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 191 packages available for upgrade (0 critical updates). [19:08:29] !log [universalomega@puppet181] Upgraded packages on mwtask161: jq and libjq1 [19:08:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:43] !log [universalomega@puppet181] Upgraded packages on ns2: jq and libjq1 [19:08:46] RECOVERY - mw201 APT on mw201 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:08:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:09:16] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 190 packages available for upgrade (0 critical updates). [19:09:17] RECOVERY - ns1 APT on ns1 is OK: APT OK: 113 packages available for upgrade (0 critical updates). [19:09:22] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 190 packages available for upgrade (0 critical updates). [19:09:52] RECOVERY - ns2 APT on ns2 is OK: APT OK: 70 packages available for upgrade (0 critical updates). [19:10:54] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 203 packages available for upgrade (18 critical updates). [19:10:59] RECOVERY - mw202 APT on mw202 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:11:09] !log [universalomega@puppet181] Upgraded packages on mw202: jq, libjq1, php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:11:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:12:08] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 204 packages available for upgrade (18 critical updates). [19:12:21] RECOVERY - mw203 APT on mw203 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:12:38] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 203 packages available for upgrade (18 critical updates). [19:12:44] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 209 packages available for upgrade (18 critical updates). [19:13:30] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: APT CRITICAL: 208 packages available for upgrade (18 critical updates). [19:15:42] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: APT CRITICAL: 208 packages available for upgrade (18 critical updates). [19:15:45] PROBLEM - mw173 APT on mw173 is CRITICAL: APT CRITICAL: 203 packages available for upgrade (18 critical updates). [19:16:51] !log [universalomega@puppet181] Upgraded packages on mw203: jq, libjq1, php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:16:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:17:08] !log [universalomega@puppet181] Upgraded packages on prometheus151: jq and libjq1 [19:17:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:04] !log [universalomega@puppet181] Upgraded packages on phorge171: jq, libjq1, php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-curl, php8.4-cli, and php8.4-common [19:18:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:10] !log [universalomega@puppet181] Upgraded packages on speedscope211: command, [Errno, handling, (most, "",, lines>..., "",, "",, and [Errno [19:18:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:19] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [19:18:35] !log [universalomega@puppet181] Upgraded packages on mwtask181: jq and libjq1 [19:18:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:47] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [19:18:53] !log [universalomega@puppet181] Upgraded packages on reports171: jq and libjq1 [19:18:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:19:12] !log [universalomega@puppet181] Upgraded packages on os162: jq and libjq1 [19:19:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:19:21] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 192 packages available for upgrade (0 critical updates). [19:19:29] RECOVERY - os162 APT on os162 is OK: APT OK: 120 packages available for upgrade (0 critical updates). [19:19:29] !log [universalomega@puppet181] Upgraded packages on swiftobject151: jq and libjq1 [19:19:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:19:52] !log [universalomega@puppet181] Upgraded packages on rdb191: jq and libjq1 [19:19:54] PROBLEM - mw193 APT on mw193 is CRITICAL: APT CRITICAL: 203 packages available for upgrade (18 critical updates). [19:19:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:20:19] !log [universalomega@puppet181] Upgraded packages on os202: jq and libjq1 [19:20:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:20:37] !log [universalomega@puppet181] Upgraded packages on puppet181: jq and libjq1 [19:20:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:20:46] RECOVERY - reports171 APT on reports171 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [19:20:55] !log [universalomega@puppet181] Upgraded packages on os151: jq and libjq1 [19:20:59] RECOVERY - rdb191 APT on rdb191 is OK: APT OK: 113 packages available for upgrade (0 critical updates). [19:21:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:21:12] RECOVERY - os151 APT on os151 is OK: APT OK: 120 packages available for upgrade (0 critical updates). [19:21:13] !log [universalomega@puppet181] Upgraded packages on os161: jq and libjq1 [19:21:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:21:21] RECOVERY - os202 APT on os202 is OK: APT OK: 125 packages available for upgrade (0 critical updates). [19:21:29] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 128 packages available for upgrade (0 critical updates). [19:21:32] !log [universalomega@puppet181] Upgraded packages on swiftac171: jq and libjq1 [19:21:34] RECOVERY - os161 APT on os161 is OK: APT OK: 120 packages available for upgrade (0 critical updates). [19:21:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:21:46] PROBLEM - mw153 APT on mw153 is CRITICAL: APT CRITICAL: 203 packages available for upgrade (18 critical updates). [19:21:51] !log [universalomega@puppet181] Upgraded packages on swiftobject161: jq and libjq1 [19:21:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:22:08] PROBLEM - mw191 APT on mw191 is CRITICAL: APT CRITICAL: 203 packages available for upgrade (18 critical updates). [19:22:10] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 203 packages available for upgrade (18 critical updates). [19:22:11] !log [universalomega@puppet181] Upgraded packages on os201: jq and libjq1 [19:22:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:22:20] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 128 packages available for upgrade (0 critical updates). [19:22:31] !log [universalomega@puppet181] Upgraded packages on os191: jq and libjq1 [19:22:35] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 95 packages available for upgrade (0 critical updates). [19:22:36] RECOVERY - os201 APT on os201 is OK: APT OK: 119 packages available for upgrade (0 critical updates). [19:22:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:22:44] RECOVERY - os191 APT on os191 is OK: APT OK: 119 packages available for upgrade (0 critical updates). [19:22:45] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 125 packages available for upgrade (0 critical updates). [19:22:50] !log [universalomega@puppet181] Upgraded packages on swiftobject171: jq and libjq1 [19:22:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:23:09] !log [universalomega@puppet181] Upgraded packages on swiftproxy161: jq and libjq1 [19:23:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:23:28] !log [universalomega@puppet181] Upgraded packages on swiftproxy171: jq and libjq1 [19:23:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:23:38] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 124 packages available for upgrade (0 critical updates). [19:23:44] !log [universalomega@puppet181] Upgraded packages on swiftobject211: jq and libjq1 [19:23:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:24:06] RECOVERY - swiftobject211 APT on swiftobject211 is OK: APT OK: 118 packages available for upgrade (0 critical updates). [19:24:10] !log [universalomega@puppet181] Upgraded packages on test151: jq and libjq1 [19:24:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:24:29] !log [universalomega@puppet181] Upgraded packages on swiftobject181: jq and libjq1 [19:24:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:24:44] RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 124 packages available for upgrade (0 critical updates). [19:24:46] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 128 packages available for upgrade (0 critical updates). [19:24:47] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [19:24:50] !log [universalomega@puppet181] Upgraded packages on swiftobject201: jq and libjq1 [19:24:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:11] !log [universalomega@puppet181] Upgraded packages on swiftobject191: jq and libjq1 [19:25:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:33] RECOVERY - swiftobject201 APT on swiftobject201 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [19:25:45] RECOVERY - swiftobject191 APT on swiftobject191 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [19:25:49] RECOVERY - test151 APT on test151 is OK: APT OK: 195 packages available for upgrade (0 critical updates). [19:28:09] PROBLEM - mw192 APT on mw192 is CRITICAL: APT CRITICAL: 203 packages available for upgrade (18 critical updates). [19:29:17] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 203 packages available for upgrade (18 critical updates). [19:29:49] PROBLEM - mw163 APT on mw163 is CRITICAL: APT CRITICAL: 203 packages available for upgrade (18 critical updates). [19:32:55] PROBLEM - reports171 APT on reports171 is CRITICAL: APT CRITICAL: 131 packages available for upgrade (15 critical updates). [19:33:26] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 203 packages available for upgrade (18 critical updates). [19:34:02] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 210 packages available for upgrade (18 critical updates). [19:34:04] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 214 packages available for upgrade (19 critical updates). [19:34:09] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 203 packages available for upgrade (18 critical updates). [19:34:38] PROBLEM - matomo211 APT on matomo211 is CRITICAL: APT CRITICAL: 84 packages available for upgrade (14 critical updates). [19:34:46] PROBLEM - mw183 APT on mw183 is CRITICAL: APT CRITICAL: 203 packages available for upgrade (18 critical updates). [19:34:46] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 49% [19:35:23] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 203 packages available for upgrade (18 critical updates). [19:35:27] PROBLEM - mw201 APT on mw201 is CRITICAL: APT CRITICAL: 203 packages available for upgrade (18 critical updates). [19:35:36] PROBLEM - mon181 APT on mon181 is CRITICAL: APT CRITICAL: 126 packages available for upgrade (17 critical updates). [19:35:43] PROBLEM - matomo151 APT on matomo151 is CRITICAL: APT CRITICAL: 128 packages available for upgrade (15 critical updates). [19:36:46] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 22% [19:38:45] !log [universalomega@puppet181] Upgraded packages on matomo151: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-curl, php8.4-cli, and php8.4-common [19:38:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:39:45] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 113 packages available for upgrade (0 critical updates). [19:39:48] !log [universalomega@puppet181] Upgraded packages on matomo211: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-curl, php8.4-cli, and php8.4-common [19:39:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:42] RECOVERY - matomo211 APT on matomo211 is OK: APT OK: 70 packages available for upgrade (0 critical updates). [19:41:25] RECOVERY - mw151 APT on mw151 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:41:53] !log [universalomega@puppet181] Upgraded packages on mw151: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:41:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:44:31] RECOVERY - mw153 APT on mw153 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:45:08] !log [universalomega@puppet181] Upgraded packages on mw153: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:45:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:46:24] RECOVERY - mw161 APT on mw161 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:50:52] !log [universalomega@puppet181] Upgraded packages on mw161: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:50:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:51:40] RECOVERY - mon181 APT on mon181 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [19:51:53] !log [universalomega@puppet181] Upgraded packages on mon181: php8.4-opcache, libapache2-mod-php8.4, php8.4-xml, php8.4-soap, php8.4-readline, php8.4-pgsql, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-curl, php8.4-cli, php8.4-common, and php8.4 [19:51:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:17] RECOVERY - mw162 APT on mw162 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:54:34] !log [universalomega@puppet181] Upgraded packages on mw162: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:54:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:56] !log [universalomega@puppet181] Upgraded packages on mw163: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:56:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:25] RECOVERY - mw163 APT on mw163 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:57:28] !log [universalomega@puppet181] Upgraded packages on mw171: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:57:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:48] RECOVERY - mw171 APT on mw171 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [19:59:46] RECOVERY - mw152 APT on mw152 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [20:02:00] !log [universalomega@puppet181] Upgraded packages on mw152: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [20:02:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:24] RECOVERY - mw173 APT on mw173 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [20:07:35] !log [universalomega@puppet181] Upgraded packages on mw173: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [20:07:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:09:20] RECOVERY - mw181 APT on mw181 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [20:09:27] !log [universalomega@puppet181] Upgraded packages on mw181: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [20:09:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:18] RECOVERY - mw172 APT on mw172 is OK: APT OK: 186 packages available for upgrade (0 critical updates). [20:12:08] !log [universalomega@puppet181] Upgraded packages on mw172: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [20:12:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:14:43] RECOVERY - mw192 APT on mw192 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [20:16:59] !log [universalomega@puppet181] Upgraded packages on mw192: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [20:17:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:19:27] RECOVERY - mw183 APT on mw183 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [20:20:30] !log [universalomega@puppet181] Upgraded packages on mw183: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [20:21:05] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 83.60, 37.15, 17.58 [20:21:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:21:58] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [20:22:11] !log [universalomega@puppet181] Upgraded packages on mw182: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [20:22:12] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.012 second response time [20:22:17] RECOVERY - mw182 APT on mw182 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [20:22:23] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:33] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:42] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [20:22:42] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.010 second response time [20:22:54] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:23:06] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 1 backends are down. mw181 [20:23:07] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw202 [20:23:10] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:23:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:23:14] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [20:23:32] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [20:23:39] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [20:23:52] RECOVERY - mw191 APT on mw191 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [20:23:54] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [20:24:05] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 3 backends are down. mw171 mw153 mw201 [20:24:09] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [20:24:14] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 2 backends are down. mw171 mw202 [20:24:20] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.015 second response time [20:24:27] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [20:24:57] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.014 second response time [20:24:58] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.930 second response time [20:24:58] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.012 second response time [20:25:11] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.463 second response time [20:25:24] !log [void@mwtask181] starting deploy of {'config': True, 'force': True} to all [20:25:26] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.012 second response time [20:25:36] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 5.307 second response time [20:25:43] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 6.229 second response time [20:25:51] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.649 second response time [20:25:53] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 47% [20:26:05] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 34 backends are healthy [20:26:09] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.082 second response time [20:26:17] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.257 second response time [20:26:28] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.720 second response time [20:26:28] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1519 bytes in 4.584 second response time [20:26:37] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 3.830 second response time [20:26:39] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.220 second response time [20:26:43] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.509 second response time [20:26:53] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.267 second response time [20:26:57] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.316 second response time [20:27:06] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 34 backends are healthy [20:27:07] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 34 backends are healthy [20:27:19] !log [void@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 114s [20:27:20] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.232 second response time [20:27:23] !log [universalomega@puppet181] Upgraded packages on mw191: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [20:27:23] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 5.171 second response time [20:27:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:27:28] !log [universalomega@puppet181] Upgraded packages on speedscope211: command, [Errno, handling, (most, "",, lines>..., "",, "",, and [Errno [20:27:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:27:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:27:43] !log [void@mwtask181] depool db161 [20:27:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:28:14] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 34 backends are healthy [20:28:16] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.278 second response time [20:28:34] !log reboot db161 from host [20:28:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:28:45] RECOVERY - db161 mariadb_memory on db161 is OK: OK Memory 20% used [20:28:57] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 0.99, 0.28, 0.10 [20:29:03] !log [universalomega@puppet181] Upgraded packages on mw193: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [20:29:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:29:18] RECOVERY - mw193 APT on mw193 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [20:29:57] !log [void@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [20:30:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:30:17] !log [void@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 20s [20:30:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:30:45] RECOVERY - mw201 APT on mw201 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [20:31:40] !log [void@mwtask181] repool db161 [20:31:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:32:46] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 79.77, 31.33, 11.62 [20:33:43] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 76% [20:34:32] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [20:35:06] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 3 backends are down. mw161 mw201 mw202 [20:35:07] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 2 backends are down. mw161 mw201 [20:35:32] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [20:35:33] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [20:35:38] !log [universalomega@puppet181] Upgraded packages on mw201: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [20:35:40] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [20:35:43] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [20:35:54] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [20:35:55] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [20:36:05] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 4 backends are down. mw181 mw182 mw153 mw183 [20:36:13] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [20:36:14] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [20:36:17] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.012 second response time [20:36:53] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 190 packages available for upgrade (0 critical updates). [20:36:53] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [20:36:56] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:37:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:37:16] !log [void@mwtask181] starting deploy of {'config': True, 'force': True} to all [20:37:30] !log [universalomega@puppet181] Upgraded packages on mwtask161: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [20:37:35] !log [void@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 19s [20:37:36] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.010 second response time [20:37:36] !log [void@mwtask181] depool db161 again [20:37:38] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 44% [20:37:39] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 9.383 second response time [20:37:42] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 2.934 second response time [20:37:42] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 5.039 second response time [20:37:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:37:50] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 1.221 second response time [20:37:51] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.907 second response time [20:37:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:37:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:38:05] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 34 backends are healthy [20:38:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:38:12] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.287 second response time [20:38:13] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.094 second response time [20:38:15] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.217 second response time [20:38:37] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.075 second response time [20:38:49] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.074 second response time [20:38:55] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.211 second response time [20:39:06] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 34 backends are healthy [20:39:07] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 34 backends are healthy [20:39:20] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 191 packages available for upgrade (0 critical updates). [20:39:25] !log [universalomega@puppet181] Upgraded packages on mwtask171: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, php8.4-cli, and php8.4-common [20:39:26] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.208 second response time [20:39:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:39:32] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.318 second response time [20:40:11] [02TSPortal] 07renovate[bot] merged 03renovate/dev-deps into 03main 13https://github.com/miraheze/TSPortal/compare/93005808a6b6...e0a01969b718 [20:40:12] [02TSPortal] 07renovate[bot] 04deleted 03renovate/dev-deps at 03e0a0196 13https://api.github.com/repos/miraheze/TSPortal/commit/e0a0196 [20:40:33] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 190 packages available for upgrade (0 critical updates). [20:40:43] !log [universalomega@puppet181] Upgraded packages on mwtask151: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [20:40:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:41:33] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 29% [20:42:47] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 192 packages available for upgrade (0 critical updates). [20:43:11] !log [universalomega@puppet181] Upgraded packages on mwtask181: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, php8.4-cli, and php8.4-common [20:43:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:44:05] RECOVERY - reports171 APT on reports171 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [20:44:11] !log [universalomega@puppet181] Upgraded packages on reports171: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-curl, php8.4-cli, and php8.4-common [20:44:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:46:15] !log [void@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [20:46:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:46:36] !log [void@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 21s [20:46:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:47:01] !log [void@mwtask181] repool db161 [20:47:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:47:39] RECOVERY - test151 APT on test151 is OK: APT OK: 195 packages available for upgrade (0 critical updates). [20:47:41] !log [universalomega@puppet181] Upgraded packages on test151: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dev, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, php8.4-cli, and php8.4-common [20:47:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:59:24] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 4.12, 4.90, 11.18 [21:01:44] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 2.94, 4.18, 10.03 [21:19:36] PROBLEM - mw193 APT on mw193 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (4 critical updates). [21:21:07] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (4 critical updates). [21:23:09] PROBLEM - mw153 APT on mw153 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (4 critical updates). [21:23:24] PROBLEM - mw191 APT on mw191 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (4 critical updates). [21:27:17] PROBLEM - mw192 APT on mw192 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (4 critical updates). [21:30:09] PROBLEM - mw163 APT on mw163 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (4 critical updates). [21:30:10] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (4 critical updates). [21:33:33] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (4 critical updates). [21:33:59] PROBLEM - mw201 APT on mw201 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (4 critical updates). [21:34:34] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 199 packages available for upgrade (4 critical updates). [21:34:37] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 196 packages available for upgrade (4 critical updates). [21:36:54] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (4 critical updates). [21:37:17] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (4 critical updates). [21:37:20] PROBLEM - mw183 APT on mw183 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (4 critical updates). [21:40:36] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (4 critical updates). [21:41:03] PROBLEM - mw202 APT on mw202 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (4 critical updates). [21:41:40] PROBLEM - mw203 APT on mw203 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (4 critical updates). [21:42:05] PROBLEM - mw173 APT on mw173 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (4 critical updates). [21:42:06] !log [universalomega@puppet181] Upgraded packages on mw171: ghostscript, libgs10, libgs10-common, and libgs-common [21:42:06] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (4 critical updates). [21:42:08] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: APT CRITICAL: 194 packages available for upgrade (4 critical updates). [21:42:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:42:27] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 190 packages available for upgrade (4 critical updates). [21:42:28] !log [universalomega@puppet181] Upgraded packages on mw152: ghostscript, libgs10, libgs10-common, and libgs-common [21:42:33] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 195 packages available for upgrade (4 critical updates). [21:42:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:42:49] !log [universalomega@puppet181] Upgraded packages on mw153: ghostscript, libgs10, libgs10-common, and libgs-common [21:42:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:42:58] RECOVERY - mw152 APT on mw152 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [21:43:11] !log [universalomega@puppet181] Upgraded packages on mw161: ghostscript, libgs10, libgs10-common, and libgs-common [21:43:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:43:17] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: APT CRITICAL: 194 packages available for upgrade (4 critical updates). [21:43:33] !log [universalomega@puppet181] Upgraded packages on mw151: ghostscript, libgs10, libgs10-common, and libgs-common [21:43:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:43:40] RECOVERY - mw153 APT on mw153 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [21:43:55] !log [universalomega@puppet181] Upgraded packages on mw163: ghostscript, libgs10, libgs10-common, and libgs-common [21:43:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:44:01] RECOVERY - mw171 APT on mw171 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [21:44:07] !log [petramagna@test151] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=jawikibeta (END - exit=256) [21:44:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:44:13] RECOVERY - mw161 APT on mw161 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [21:44:13] RECOVERY - mw151 APT on mw151 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [21:44:16] !log [universalomega@puppet181] Upgraded packages on mw191: ghostscript, libgs10, libgs10-common, and libgs-common [21:44:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:44:32] RECOVERY - mw163 APT on mw163 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [21:44:39] !log [universalomega@puppet181] Upgraded packages on mwtask151: ghostscript, libgs10, libgs10-common, and libgs-common [21:44:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:45:01] !log [universalomega@puppet181] Upgraded packages on mwtask161: ghostscript, libgs10, libgs10-common, and libgs-common [21:45:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:45:21] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 190 packages available for upgrade (0 critical updates). [21:45:24] !log [universalomega@puppet181] Upgraded packages on mw172: ghostscript, libgs10, libgs10-common, and libgs-common [21:45:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:45:46] !log [universalomega@puppet181] Upgraded packages on mw182: ghostscript, libgs10, libgs10-common, and libgs-common [21:45:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:45:54] RECOVERY - mw191 APT on mw191 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [21:46:08] !log [universalomega@puppet181] Upgraded packages on mw173: ghostscript, libgs10, libgs10-common, and libgs-common [21:46:13] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 190 packages available for upgrade (0 critical updates). [21:46:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:46:30] RECOVERY - mw172 APT on mw172 is OK: APT OK: 186 packages available for upgrade (0 critical updates). [21:46:31] !log [universalomega@puppet181] Upgraded packages on mwtask171: ghostscript, libgs10, libgs10-common, and libgs-common [21:46:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:46:44] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 191 packages available for upgrade (0 critical updates). [21:46:57] !log [universalomega@puppet181] Upgraded packages on test151: ghostscript, libgs10, libgs10-common, and libgs-common [21:47:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:19] !log [universalomega@puppet181] Upgraded packages on mw181: ghostscript, libgs10, libgs10-common, and libgs-common [21:47:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:43] !log [universalomega@puppet181] Upgraded packages on mw201: ghostscript, libgs10, libgs10-common, and libgs-common [21:47:43] RECOVERY - mw182 APT on mw182 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [21:47:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:48:05] !log [universalomega@puppet181] Upgraded packages on mw162: ghostscript, libgs10, libgs10-common, and libgs-common [21:48:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:48:14] RECOVERY - mw201 APT on mw201 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [21:48:16] RECOVERY - mw173 APT on mw173 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [21:48:16] RECOVERY - mw181 APT on mw181 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [21:48:27] !log [universalomega@puppet181] Upgraded packages on mw192: ghostscript, libgs10, libgs10-common, and libgs-common [21:48:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:48:47] RECOVERY - test151 APT on test151 is OK: APT OK: 195 packages available for upgrade (0 critical updates). [21:48:50] !log [universalomega@puppet181] Upgraded packages on mw193: ghostscript, libgs10, libgs10-common, and libgs-common [21:48:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:00] RECOVERY - mw162 APT on mw162 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [21:49:03] RECOVERY - mw193 APT on mw193 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [21:49:13] !log [universalomega@puppet181] Upgraded packages on mw203: ghostscript, libgs10, libgs10-common, and libgs-common [21:49:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:27] RECOVERY - mw192 APT on mw192 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [21:49:36] !log [universalomega@puppet181] Upgraded packages on mw202: ghostscript, libgs10, libgs10-common, and libgs-common [21:49:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:42] RECOVERY - mw202 APT on mw202 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [21:49:47] RECOVERY - mw203 APT on mw203 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [21:50:00] !log [universalomega@puppet181] Upgraded packages on mw183: ghostscript, libgs10, libgs10-common, and libgs-common [21:50:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:50:08] RECOVERY - mw183 APT on mw183 is OK: APT OK: 185 packages available for upgrade (0 critical updates). [21:50:24] !log [universalomega@puppet181] Upgraded packages on mwtask181: ghostscript, libgs10, libgs10-common, and libgs-common [21:50:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:50:31] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 192 packages available for upgrade (0 critical updates). [22:37:00] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 44% [22:38:57] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 23% [23:50:57] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 45276MiB (10.0% inode=100%);