[00:01:20] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/ae16ecf66688f8c22149fe0424bee81ccb7fe3f5 [00:01:20] 02statichelp/03main 07WikiTideBot 03ae16ecf Bot: Auto-update Tech namespace pages 2025-10-23 00:01:17 [00:08:22] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.62, 16.92, 14.79 [00:08:30] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.88, 17.40, 14.67 [00:09:16] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.56, 18.17, 15.55 [00:09:19] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 21.98, 17.28, 14.78 [00:09:19] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.72, 17.70, 14.34 [00:11:37] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.85, 17.18, 13.45 [00:11:45] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 22.72, 18.50, 14.89 [00:11:51] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 22.57, 19.13, 15.16 [00:12:01] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.96, 16.85, 12.88 [00:13:12] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 19.07, 18.52, 15.47 [00:13:25] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.76, 19.15, 15.18 [00:13:53] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 72450MiB (15% inode=99%); [00:14:01] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 19.51, 18.19, 13.89 [00:15:30] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 73761MiB (16% inode=99%); [00:15:37] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 20.17, 19.38, 15.24 [00:16:15] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 72955MiB (16% inode=99%); [00:17:16] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.85, 20.08, 17.72 [00:17:19] PROBLEM - mw191 Current Load on mw191 is CRITICAL: LOAD CRITICAL - total load average: 24.02, 21.96, 17.98 [00:17:25] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 15.70, 18.21, 15.79 [00:19:19] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 23.14, 22.33, 18.60 [00:20:17] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.01, 20.10, 16.84 [00:20:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.10, 19.51, 16.14 [00:21:16] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.99, 21.51, 18.84 [00:22:30] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.09, 20.04, 18.34 [00:24:05] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 19.32, 20.29, 17.67 [00:24:09] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 18.87, 20.32, 17.29 [00:24:44] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.17, 20.07, 17.32 [00:25:03] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 21.74, 19.74, 17.13 [00:26:30] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.32, 21.08, 19.13 [00:26:41] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 17.66, 19.05, 17.27 [00:27:03] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 18.60, 19.12, 17.22 [00:27:25] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 23.23, 21.29, 18.25 [00:28:00] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 20.96, 20.97, 18.56 [00:29:25] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 17.88, 19.60, 17.98 [00:29:38] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 18.18, 19.76, 18.98 [00:29:45] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 17.46, 20.23, 19.19 [00:30:00] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 19.09, 20.17, 18.56 [00:33:03] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 23.67, 21.12, 18.65 [00:33:25] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 20.46, 20.06, 18.54 [00:34:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1761176030000&orgId=1&to=1761179660038 [00:35:25] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 20.23, 20.22, 18.79 [00:35:45] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 20.93, 20.94, 19.81 [00:39:03] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 15.81, 19.13, 18.72 [00:40:22] PROBLEM - mw203 Current Load on mw203 is CRITICAL: LOAD CRITICAL - total load average: 24.41, 23.64, 22.26 [00:41:37] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.14, 19.60, 17.94 [00:41:45] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 18.90, 20.38, 19.97 [00:42:22] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 23.70, 23.69, 22.46 [00:45:03] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 20.99, 19.63, 18.94 [00:45:25] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.45, 19.81, 18.73 [00:45:37] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 20.17, 20.11, 18.55 [00:47:03] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 20.06, 19.90, 19.14 [00:47:25] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 19.34, 19.74, 18.86 [00:49:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1761176030000&orgId=1&to=1761180560000 [00:50:30] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 8.13, 16.63, 19.40 [00:51:16] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 4.82, 13.78, 18.53 [00:51:19] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 6.11, 14.38, 18.68 [00:52:22] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 6.42, 13.79, 18.89 [02:09:33] [02mw-config] 07pskyechology requested changes on pull request #6148 13https://github.com/miraheze/mw-config/pull/6148#pullrequestreview-3368033130 [02:09:34] [02mw-config] 07pskyechology left a file comment in pull request #6148 03fd6c7e6: Should be just `'cloudcrafterwiki'` 13https://github.com/miraheze/mw-config/pull/6148#discussion_r2453743755 [02:34:51] [02mw-config] 07pskyechology left a file comment in pull request #6148 03fd6c7e6: Check indentation 13https://github.com/miraheze/mw-config/pull/6148#discussion_r2453767229 [02:34:51] [02mw-config] 07pskyechology requested changes on pull request #6148 13https://github.com/miraheze/mw-config/pull/6148#pullrequestreview-3368059372 [02:34:51] [02mw-config] 07pskyechology left a file comment in pull request #6148 03fd6c7e6: Check indentation 13https://github.com/miraheze/mw-config/pull/6148#discussion_r2453768345 [02:42:07] [02mw-config] 07pskyechology requested changes on pull request #6149: Check indentation 13https://github.com/miraheze/mw-config/pull/6149#pullrequestreview-3368072647 [02:48:47] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 42% [02:50:47] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 33% [03:02:07] miraheze/mw-config - originalcloudcrafter the build passed. [03:05:24] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/8c95b688e017a673ee598374e5bf05ee52e65dca [03:05:25] 02ssl/03main 07WikiTideBot 038c95b68 Bot: Auto-update domain lists [03:29:16] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.50, 19.64, 15.72 [03:29:57] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 20.86, 19.54, 15.87 [03:32:30] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.16, 20.74, 16.40 [03:33:46] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 16.34, 19.05, 16.58 [03:34:22] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 21.70, 20.31, 17.06 [03:39:30] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 21.92, 20.89, 18.12 [03:40:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1761187220000&orgId=1&to=1761190850035 [03:42:30] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.74, 20.31, 18.49 [03:42:32] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 23.56, 21.69, 18.37 [03:45:46] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 20.96, 19.31, 16.73 [03:47:43] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 18.76, 18.96, 16.91 [03:50:15] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 12.01, 18.68, 18.65 [03:50:22] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 11.04, 18.76, 19.22 [03:50:50] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1761187820000&orgId=1&to=1761191420000 [03:51:16] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 8.46, 16.57, 18.65 [03:51:19] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 9.95, 16.86, 18.26 [04:09:02] PROBLEM - mw202 Current Load on mw202 is CRITICAL: LOAD CRITICAL - total load average: 24.12, 17.41, 14.02 [04:09:16] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.76, 16.39, 14.25 [04:11:00] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 21.44, 18.60, 14.85 [04:12:22] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.94, 18.48, 15.44 [04:13:25] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.48, 17.75, 13.72 [04:14:30] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.23, 19.98, 16.45 [04:14:54] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 19.84, 19.22, 15.94 [04:15:03] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 21.25, 17.47, 13.90 [04:15:25] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 18.08, 17.96, 14.31 [04:15:38] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 20.48, 18.31, 15.06 [04:16:22] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 19.99, 20.19, 16.89 [04:17:19] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 22.16, 20.32, 16.59 [04:19:03] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 17.25, 18.18, 15.10 [04:20:30] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.31, 20.03, 17.75 [04:21:16] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.61, 19.01, 17.73 [04:21:38] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 16.17, 18.98, 16.62 [04:25:19] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 19.41, 20.12, 18.11 [04:27:38] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 20.67, 20.03, 17.77 [04:28:22] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 23.27, 21.26, 18.99 [04:29:16] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.56, 20.34, 18.63 [04:29:38] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 19.87, 19.85, 17.98 [04:30:30] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.41, 20.98, 18.88 [04:33:19] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 23.41, 21.60, 19.44 [04:35:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1761190490000&orgId=1&to=1761194120041 [04:35:38] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 21.53, 21.09, 19.11 [04:36:00] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 22.07, 20.06, 17.54 [04:37:38] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 19.78, 20.22, 19.01 [04:40:00] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 20.13, 19.73, 17.95 [04:40:16] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 22.58, 20.93, 18.80 [04:41:19] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 17.86, 19.95, 19.69 [04:42:22] PROBLEM - mw203 Current Load on mw203 is CRITICAL: LOAD CRITICAL - total load average: 24.19, 23.38, 21.56 [04:44:11] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 18.00, 19.95, 18.92 [04:44:22] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 20.78, 22.63, 21.52 [04:49:19] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 22.39, 21.01, 20.12 [04:51:19] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 14.01, 18.92, 19.52 [04:52:22] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 7.61, 15.26, 19.04 [04:52:30] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 7.06, 15.33, 18.68 [04:53:16] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 8.87, 15.97, 18.94 [04:55:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1761191480000&orgId=1&to=1761195080000 [05:07:45] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 22.86, 16.04, 13.93 [05:11:16] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.52, 18.32, 15.70 [05:11:19] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 22.72, 17.44, 14.97 [05:11:38] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 20.53, 17.58, 15.11 [05:11:45] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 18.99, 18.80, 15.65 [05:13:16] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 23.53, 20.16, 17.07 [05:14:01] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.68, 19.08, 15.18 [05:15:38] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 19.15, 19.31, 16.44 [05:15:56] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.55, 19.59, 15.82 [05:16:30] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.34, 20.18, 17.13 [05:17:52] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.98, 19.04, 16.07 [05:19:14] PROBLEM - mw203 Current Load on mw203 is CRITICAL: LOAD CRITICAL - total load average: 24.16, 22.51, 19.10 [05:20:20] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 21.23, 20.67, 17.70 [05:21:09] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 19.40, 21.66, 19.23 [05:21:16] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.45, 20.39, 18.41 [05:21:38] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 16.38, 19.03, 17.46 [05:22:30] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.54, 19.73, 18.06 [05:25:45] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 21.50, 19.91, 17.64 [05:27:19] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 16.58, 19.33, 18.47 [05:27:45] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 17.20, 18.97, 17.58 [05:32:59] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 14.14, 19.07, 19.51 [08:12:22] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 23.35, 18.40, 13.01 [08:14:22] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 14.67, 16.84, 13.10 [08:29:16] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.11, 19.40, 15.84 [08:29:17] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 20.93, 19.80, 17.00 [08:31:16] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 18.33, 19.23, 17.13 [08:31:16] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.18, 21.12, 16.90 [08:33:16] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.85, 19.17, 16.74 [08:35:11] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.74, 21.54, 18.58 [08:44:19] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 22.61, 20.20, 17.88 [08:46:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1761205550000&orgId=1&to=1761209180036 [08:50:02] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 9.38, 16.98, 17.51 [08:50:57] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 7.56, 15.83, 18.21 [08:51:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1761205790000&orgId=1&to=1761209390000 [09:09:19] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 22.48, 16.64, 13.94 [09:10:39] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.46, 17.47, 14.44 [09:11:19] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 16.58, 17.12, 14.49 [09:12:37] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 14.15, 16.27, 14.39 [09:46:18] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=wackipediawiki (END - exit=0) [09:46:20] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/initSiteStats.php --wiki=wackipediawiki --update (END - exit=0) [09:46:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:46:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:15:20] [02CreateWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/7ba781f9198a23215531c02dc83890e98a651ab1 [12:15:20] 02CreateWiki/03main 07translatewiki.net 037ba781f Localisation updates from https://translatewiki.net. [12:37:17] miraheze/CreateWiki - translatewiki the build passed. [13:22:51] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 18.28, 11.86, 6.84 [13:24:52] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 4.93, 8.95, 6.39 [15:01:53] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49833MiB (10% inode=99%); [15:18:15] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49795MiB (10% inode=99%); [15:45:30] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49846MiB (10% inode=99%); [18:14:47] PROBLEM - mem161 APT on mem161 is CRITICAL: APT CRITICAL: 120 packages available for upgrade (3 critical updates). [18:14:56] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: APT CRITICAL: 127 packages available for upgrade (3 critical updates). [18:15:07] PROBLEM - mw173 APT on mw173 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:15:19] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: APT CRITICAL: 136 packages available for upgrade (3 critical updates). [18:16:09] PROBLEM - phorge171 APT on phorge171 is CRITICAL: APT CRITICAL: 77 packages available for upgrade (3 critical updates). [18:16:11] PROBLEM - swiftobject181 APT on swiftobject181 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (3 critical updates). [18:17:01] PROBLEM - db161 APT on db161 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (3 critical updates). [18:17:02] PROBLEM - cloud16 APT on cloud16 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (3 critical updates). [18:17:36] PROBLEM - mem151 APT on mem151 is CRITICAL: APT CRITICAL: 120 packages available for upgrade (3 critical updates). [18:17:55] PROBLEM - cloud15 APT on cloud15 is CRITICAL: APT CRITICAL: 190 packages available for upgrade (3 critical updates). [18:18:21] PROBLEM - os151 APT on os151 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (3 critical updates). [18:18:27] PROBLEM - os201 APT on os201 is CRITICAL: APT CRITICAL: 84 packages available for upgrade (3 critical updates). [18:18:46] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 118 packages available for upgrade (3 critical updates). [18:18:49] PROBLEM - mw193 APT on mw193 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:21:11] PROBLEM - kafka181 APT on kafka181 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [18:21:35] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:21:51] PROBLEM - mw153 APT on mw153 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:21:53] PROBLEM - db171 APT on db171 is CRITICAL: APT CRITICAL: 122 packages available for upgrade (3 critical updates). [18:23:17] PROBLEM - mw191 APT on mw191 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:23:39] PROBLEM - cloud19 APT on cloud19 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (3 critical updates). [18:23:55] PROBLEM - cloud17 APT on cloud17 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (3 critical updates). [18:24:04] PROBLEM - db182 APT on db182 is CRITICAL: APT CRITICAL: 85 packages available for upgrade (3 critical updates). [18:25:39] PROBLEM - cloud18 APT on cloud18 is CRITICAL: APT CRITICAL: 187 packages available for upgrade (3 critical updates). [18:25:56] PROBLEM - bots171 APT on bots171 is CRITICAL: APT CRITICAL: 118 packages available for upgrade (3 critical updates). [18:26:08] PROBLEM - rdb151 APT on rdb151 is CRITICAL: APT CRITICAL: 120 packages available for upgrade (3 critical updates). [18:26:26] PROBLEM - cp191 APT on cp191 is CRITICAL: APT CRITICAL: 111 packages available for upgrade (3 critical updates). [18:27:14] PROBLEM - mem191 APT on mem191 is CRITICAL: APT CRITICAL: 112 packages available for upgrade (3 critical updates). [18:27:30] PROBLEM - graylog161 APT on graylog161 is CRITICAL: APT CRITICAL: 118 packages available for upgrade (3 critical updates). [18:27:40] PROBLEM - cp161 APT on cp161 is CRITICAL: APT CRITICAL: 121 packages available for upgrade (3 critical updates). [18:28:15] PROBLEM - os162 APT on os162 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (3 critical updates). [18:28:15] PROBLEM - swiftobject171 APT on swiftobject171 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (3 critical updates). [18:28:17] PROBLEM - mw192 APT on mw192 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:29:10] PROBLEM - changeprop201 APT on changeprop201 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [18:29:42] PROBLEM - mw163 APT on mw163 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:29:50] updoots [18:29:52] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 134 packages available for upgrade (3 critical updates). [18:29:52] PROBLEM - db181 APT on db181 is CRITICAL: APT CRITICAL: 122 packages available for upgrade (3 critical updates). [18:30:31] PROBLEM - os202 APT on os202 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (4 critical updates). [18:31:47] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 118 packages available for upgrade (4 critical updates). [18:31:58] PROBLEM - db151 APT on db151 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (3 critical updates). [18:31:59] PROBLEM - os191 APT on os191 is CRITICAL: APT CRITICAL: 112 packages available for upgrade (3 critical updates). [18:32:24] PROBLEM - mattermost1 APT on mattermost1 is CRITICAL: APT CRITICAL: 116 packages available for upgrade (4 critical updates). [18:33:21] PROBLEM - swiftobject161 APT on swiftobject161 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (3 critical updates). [18:33:21] PROBLEM - reports171 APT on reports171 is CRITICAL: APT CRITICAL: 116 packages available for upgrade (3 critical updates). [18:33:22] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:33:51] PROBLEM - cloud20 APT on cloud20 is CRITICAL: APT CRITICAL: 191 packages available for upgrade (3 critical updates). [18:34:13] PROBLEM - swiftobject151 APT on swiftobject151 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (3 critical updates). [18:34:27] PROBLEM - mon181 APT on mon181 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (3 critical updates). [18:34:27] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 120 packages available for upgrade (3 critical updates). [18:34:32] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 136 packages available for upgrade (3 critical updates). [18:34:33] PROBLEM - mw201 APT on mw201 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:34:41] PROBLEM - os161 APT on os161 is CRITICAL: APT CRITICAL: 112 packages available for upgrade (3 critical updates). [18:35:44] PROBLEM - bast181 APT on bast181 is CRITICAL: APT CRITICAL: 112 packages available for upgrade (3 critical updates). [18:36:08] PROBLEM - ldap171 APT on ldap171 is CRITICAL: APT CRITICAL: 112 packages available for upgrade (3 critical updates). [18:36:23] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:37:07] PROBLEM - mw183 APT on mw183 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:37:40] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:38:27] PROBLEM - mem201 APT on mem201 is CRITICAL: APT CRITICAL: 120 packages available for upgrade (3 critical updates). [18:38:30] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: APT CRITICAL: 116 packages available for upgrade (3 critical updates). [18:39:21] PROBLEM - swiftproxy161 APT on swiftproxy161 is CRITICAL: APT CRITICAL: 129 packages available for upgrade (3 critical updates). [18:39:30] PROBLEM - matomo151 APT on matomo151 is CRITICAL: APT CRITICAL: 75 packages available for upgrade (3 critical updates). [18:40:04] PROBLEM - cp201 APT on cp201 is CRITICAL: APT CRITICAL: 83 packages available for upgrade (4 critical updates). [18:40:30] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:40:48] PROBLEM - mw202 APT on mw202 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:40:56] PROBLEM - eventgate181 APT on eventgate181 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:40:59] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:41:16] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:41:45] PROBLEM - swiftproxy171 APT on swiftproxy171 is CRITICAL: APT CRITICAL: 129 packages available for upgrade (3 critical updates). [18:41:50] PROBLEM - db172 APT on db172 is CRITICAL: APT CRITICAL: 95 packages available for upgrade (3 critical updates). [18:42:05] PROBLEM - llm191 APT on llm191 is CRITICAL: APT CRITICAL: 49 packages available for upgrade (3 critical updates). [18:42:11] PROBLEM - cp171 APT on cp171 is CRITICAL: APT CRITICAL: 111 packages available for upgrade (3 critical updates). [18:42:21] PROBLEM - bast161 APT on bast161 is CRITICAL: APT CRITICAL: 112 packages available for upgrade (3 critical updates). [18:42:31] PROBLEM - mw203 APT on mw203 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (3 critical updates). [18:43:09] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 136 packages available for upgrade (3 critical updates). [18:43:19] PROBLEM - swiftobject191 APT on swiftobject191 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (3 critical updates). [18:43:28] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: APT CRITICAL: 136 packages available for upgrade (3 critical updates). [18:44:02] PROBLEM - swiftobject201 APT on swiftobject201 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (3 critical updates). [18:44:14] PROBLEM - puppet181 APT on puppet181 is CRITICAL: APT CRITICAL: 79 packages available for upgrade (3 critical updates). [18:46:00] PROBLEM - db182 Disk Space on db182 is WARNING: DISK WARNING - free space: / 53452MiB (9% inode=99%); [19:26:54] room full of problema [19:27:06] WOW IS THAT PUPPET FROM... FROM FNAF?!? [19:34:00] PROBLEM - db182 Disk Space on db182 is CRITICAL: DISK CRITICAL - free space: / 33633MiB (5% inode=99%); [19:36:00] PROBLEM - db182 Disk Space on db182 is WARNING: DISK WARNING - free space: / 48299MiB (8% inode=99%); [19:59:51] [02puppet] 07pskyechology opened pull request #4586: Add fluidsynth to mediawiki packages list (07miraheze:03main...07pskyechology:03T14371) 13https://github.com/miraheze/puppet/pull/4586 [20:00:39] [02puppet] 07coderabbitai[bot] commented on pull request #4586:
[…] 13https://github.com/miraheze/puppet/pull/4586#issuecomment-3438900886 [20:01:38] !log [void@puppet181] Upgraded packages on changeprop201: bind9-host, bind9-dnsutils, and bind9-libs [20:01:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:01:54] !log [void@puppet181] Upgraded packages on cloud17: bind9-host, bind9-dnsutils, and bind9-libs [20:01:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:10] !log [void@puppet181] Upgraded packages on bots171: bind9-host, bind9-dnsutils, and bind9-libs [20:02:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:25] !log [void@puppet181] Upgraded packages on cp191: bind9-host, bind9-dnsutils, and bind9-libs [20:02:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:32] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 184 packages available for upgrade (0 critical updates). [20:02:41] !log [void@puppet181] Upgraded packages on cloud18: bind9-host, bind9-dnsutils, and bind9-libs [20:02:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:54] !log [void@puppet181] Upgraded packages on db172: bind9-host, bind9-dnsutils, and bind9-libs [20:02:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:09] RECOVERY - changeprop201 APT on changeprop201 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:03:10] !log [void@puppet181] Upgraded packages on db151: bind9-host, bind9-dnsutils, and bind9-libs [20:03:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:17] RECOVERY - db172 APT on db172 is OK: APT OK: 92 packages available for upgrade (0 critical updates). [20:03:23] RECOVERY - cp191 APT on cp191 is OK: APT OK: 108 packages available for upgrade (0 critical updates). [20:03:26] !log [void@puppet181] Upgraded packages on cloud16: bind9-host, bind9-dnsutils, and bind9-libs [20:03:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:36] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 184 packages available for upgrade (0 critical updates). [20:03:41] !log [void@puppet181] Upgraded packages on cp171: bind9-host, bind9-dnsutils, and bind9-libs [20:03:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:03:53] RECOVERY - bots171 APT on bots171 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [20:03:53] RECOVERY - db151 APT on db151 is OK: APT OK: 127 packages available for upgrade (0 critical updates). [20:03:56] !log [void@puppet181] Upgraded packages on cloud15: bind9-host, bind9-dnsutils, and bind9-libs [20:04:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:06] RECOVERY - cp171 APT on cp171 is OK: APT OK: 108 packages available for upgrade (0 critical updates). [20:04:11] !log [void@puppet181] Upgraded packages on eventgate181: bind9-host, bind9-dnsutils, bind9-libs, and dnsutils [20:04:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:25] !log [void@puppet181] Upgraded packages on llm191: bind9-host, bind9-dnsutils, and bind9-libs [20:04:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:38] RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 187 packages available for upgrade (0 critical updates). [20:04:39] !log [void@puppet181] Upgraded packages on kafka181: bind9-host, bind9-dnsutils, and bind9-libs [20:04:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:51] RECOVERY - eventgate181 APT on eventgate181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:04:53] RECOVERY - llm191 APT on llm191 is OK: APT OK: 46 packages available for upgrade (0 critical updates). [20:04:54] !log [void@puppet181] Upgraded packages on cp201: bind9-host, bind9-dnsutils, bind9-libs, and dnsutils [20:04:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:59] RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 184 packages available for upgrade (0 critical updates). [20:05:09] !log [void@puppet181] Upgraded packages on db161: bind9-host, bind9-dnsutils, and bind9-libs [20:05:11] RECOVERY - kafka181 APT on kafka181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:05:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:23] !log [void@puppet181] Upgraded packages on cloud19: bind9-host, bind9-dnsutils, and bind9-libs [20:05:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:37] !log [void@puppet181] Upgraded packages on cp161: bind9-host, bind9-dnsutils, and bind9-libs [20:05:38] RECOVERY - cloud19 APT on cloud19 is OK: APT OK: 3 packages available for upgrade (0 critical updates). [20:05:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:53] !log [void@puppet181] Upgraded packages on db171: bind9-host, bind9-dnsutils, and bind9-libs [20:05:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:00] RECOVERY - cp201 APT on cp201 is OK: APT OK: 79 packages available for upgrade (0 critical updates). [20:06:02] RECOVERY - db171 APT on db171 is OK: APT OK: 119 packages available for upgrade (0 critical updates). [20:06:09] !log [void@puppet181] Upgraded packages on cloud20: bind9-host, bind9-dnsutils, and bind9-libs [20:06:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:26] !log [void@puppet181] Upgraded packages on db181: bind9-host, bind9-dnsutils, and bind9-libs [20:06:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:43] !log [void@puppet181] Upgraded packages on graylog161: bind9-host, bind9-dnsutils, and bind9-libs [20:06:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:52] RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [20:06:58] RECOVERY - db161 APT on db161 is OK: APT OK: 127 packages available for upgrade (0 critical updates). [20:06:58] !log [void@puppet181] Upgraded packages on ldap171: bind9-host, bind9-dnsutils, and bind9-libs [20:07:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:07:16] !log [void@puppet181] Upgraded packages on db182: bind9-host, bind9-dnsutils, and bind9-libs [20:07:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:07:27] !log [void@puppet181] Upgraded packages on mattermost1: bind9-host, bind9-dnsutils, bind9-libs, and dnsutils [20:07:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:07:36] RECOVERY - cp161 APT on cp161 is OK: APT OK: 118 packages available for upgrade (0 critical updates). [20:07:45] !log [void@puppet181] Upgraded packages on bast181: bind9-host, bind9-dnsutils, and bind9-libs [20:07:46] RECOVERY - ldap171 APT on ldap171 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [20:07:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:07:51] RECOVERY - cloud20 APT on cloud20 is OK: APT OK: 188 packages available for upgrade (0 critical updates). [20:07:56] RECOVERY - db181 APT on db181 is OK: APT OK: 119 packages available for upgrade (0 critical updates). [20:07:59] !log [void@puppet181] Upgraded packages on matomo151: bind9-host, bind9-dnsutils, and bind9-libs [20:08:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:08:18] !log [void@puppet181] Upgraded packages on bast161: bind9-host, bind9-dnsutils, and bind9-libs [20:08:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:08:23] RECOVERY - mattermost1 APT on mattermost1 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [20:08:33] !log [void@puppet181] Upgraded packages on mem151: bind9-host, bind9-dnsutils, and bind9-libs [20:08:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:08:43] RECOVERY - db182 APT on db182 is OK: APT OK: 82 packages available for upgrade (0 critical updates). [20:08:48] !log [void@puppet181] Upgraded packages on mem161: bind9-host, bind9-dnsutils, and bind9-libs [20:08:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:09:03] !log [void@puppet181] Upgraded packages on mem201: bind9-host, bind9-dnsutils, and bind9-libs [20:09:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:09:13] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 72 packages available for upgrade (0 critical updates). [20:09:20] !log [void@puppet181] Upgraded packages on mem191: bind9-host, bind9-dnsutils, and bind9-libs [20:09:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:09:28] RECOVERY - bast161 APT on bast161 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [20:09:30] RECOVERY - mem151 APT on mem151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [20:09:33] RECOVERY - bast181 APT on bast181 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [20:09:35] !log [void@puppet181] Upgraded packages on mon181: bind9-host, bind9-dnsutils, and bind9-libs [20:09:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:09:52] !log [void@puppet181] Upgraded packages on mw152: bind9-host, bind9-dnsutils, and bind9-libs [20:09:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:10:09] !log [void@puppet181] Upgraded packages on mw161: bind9-host, bind9-dnsutils, and bind9-libs [20:10:09] RECOVERY - mem191 APT on mem191 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [20:10:10] RECOVERY - mw152 APT on mw152 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:10:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:10:23] RECOVERY - mem201 APT on mem201 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [20:10:25] !log [void@puppet181] Upgraded packages on mw151: bind9-host, bind9-dnsutils, and bind9-libs [20:10:26] RECOVERY - mon181 APT on mon181 is OK: APT OK: 7 packages available for upgrade (0 critical updates). [20:10:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:10:41] RECOVERY - mem161 APT on mem161 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [20:10:42] !log [void@puppet181] Upgraded packages on mw172: bind9-host, bind9-dnsutils, and bind9-libs [20:10:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:10:58] RECOVERY - mw172 APT on mw172 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:11:00] !log [void@puppet181] Upgraded packages on mw163: bind9-host, bind9-dnsutils, and bind9-libs [20:11:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:11] RECOVERY - mw161 APT on mw161 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:11:18] !log [void@puppet181] Upgraded packages on mw173: bind9-host, bind9-dnsutils, and bind9-libs [20:11:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:34] !log [void@puppet181] Upgraded packages on mw162: bind9-host, bind9-dnsutils, and bind9-libs [20:11:36] RECOVERY - mw163 APT on mw163 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:11:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:48] RECOVERY - mw151 APT on mw151 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [20:11:52] !log [void@puppet181] Upgraded packages on mw153: bind9-host, bind9-dnsutils, and bind9-libs [20:11:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:56] RECOVERY - mw162 APT on mw162 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:12:10] !log [void@puppet181] Upgraded packages on mw171: bind9-host, bind9-dnsutils, and bind9-libs [20:12:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:12:18] RECOVERY - mw153 APT on mw153 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:12:27] !log [void@puppet181] Upgraded packages on mw192: bind9-host, bind9-dnsutils, and bind9-libs [20:12:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:12:40] RECOVERY - mw171 APT on mw171 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:12:46] !log [void@puppet181] Upgraded packages on mw183: bind9-host, bind9-dnsutils, and bind9-libs [20:12:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:13:04] !log [void@puppet181] Upgraded packages on mw191: bind9-host, bind9-dnsutils, and bind9-libs [20:13:04] RECOVERY - mw173 APT on mw173 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:13:05] RECOVERY - mw183 APT on mw183 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:13:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:13:23] !log [void@puppet181] Upgraded packages on mw181: bind9-host, bind9-dnsutils, and bind9-libs [20:13:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:13:40] !log [void@puppet181] Upgraded packages on mw193: bind9-host, bind9-dnsutils, and bind9-libs [20:13:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:13:58] RECOVERY - mw181 APT on mw181 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:13:58] !log [void@puppet181] Upgraded packages on mw182: bind9-host, bind9-dnsutils, and bind9-libs [20:14:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:14:15] !log [void@puppet181] Upgraded packages on mw201: bind9-host, bind9-dnsutils, and bind9-libs [20:14:16] RECOVERY - mw192 APT on mw192 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:14:17] RECOVERY - mw182 APT on mw182 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:14:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:14:28] RECOVERY - mw191 APT on mw191 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:14:33] !log [void@puppet181] Upgraded packages on mw202: bind9-host, bind9-dnsutils, and bind9-libs [20:14:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:14:46] RECOVERY - mw202 APT on mw202 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:14:50] !log [void@puppet181] Upgraded packages on mwtask161: bind9-host, bind9-dnsutils, bind9-libs, and python3-internetarchive [20:14:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:15:09] !log [void@puppet181] Upgraded packages on mw203: bind9-host, bind9-dnsutils, and bind9-libs [20:15:10] RECOVERY - mw203 APT on mw203 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:15:16] RECOVERY - mw193 APT on mw193 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:15:18] RECOVERY - mw201 APT on mw201 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [20:15:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:15:29] !log [void@puppet181] Upgraded packages on mwtask151: bind9-host, bind9-dnsutils, bind9-libs, and python3-internetarchive [20:15:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:15:47] !log [void@puppet181] Upgraded packages on os151: bind9-host, bind9-dnsutils, and bind9-libs [20:15:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:16:06] !log [void@puppet181] Upgraded packages on mwtask171: bind9-host, bind9-dnsutils, bind9-libs, and python3-internetarchive [20:16:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:16:20] RECOVERY - os151 APT on os151 is OK: APT OK: 111 packages available for upgrade (0 critical updates). [20:16:22] !log [void@puppet181] Upgraded packages on os162: bind9-host, bind9-dnsutils, and bind9-libs [20:16:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:16:38] !log [void@puppet181] Upgraded packages on os161: bind9-host, bind9-dnsutils, and bind9-libs [20:16:41] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 133 packages available for upgrade (0 critical updates). [20:16:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:16:56] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 133 packages available for upgrade (0 critical updates). [20:16:58] RECOVERY - os161 APT on os161 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [20:17:04] !log [void@puppet181] Upgraded packages on ns1: bind9-host, bind9-dnsutils, and bind9-libs [20:17:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:17:22] !log [void@puppet181] Upgraded packages on mwtask181: bind9-host, bind9-dnsutils, bind9-libs, and python3-internetarchive [20:17:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:17:28] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 133 packages available for upgrade (0 critical updates). [20:17:36] !log [void@puppet181] Upgraded packages on phorge171: bind9-host, bind9-dnsutils, and bind9-libs [20:17:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:17:52] !log [void@puppet181] Upgraded packages on prometheus151: bind9-host, bind9-dnsutils, and bind9-libs [20:17:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:18:05] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 74 packages available for upgrade (0 critical updates). [20:18:09] !log [void@puppet181] Upgraded packages on os191: bind9-host, bind9-dnsutils, and bind9-libs [20:18:11] RECOVERY - os162 APT on os162 is OK: APT OK: 111 packages available for upgrade (0 critical updates). [20:18:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:18:29] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 133 packages available for upgrade (0 critical updates). [20:18:29] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 113 packages available for upgrade (0 critical updates). [20:18:30] !log [void@puppet181] Upgraded packages on os202: bind9-host, bind9-dnsutils, bind9-libs, and dnsutils [20:18:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:18:40] RECOVERY - ns1 APT on ns1 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [20:18:45] !log [void@puppet181] Upgraded packages on os201: bind9-host, bind9-dnsutils, and bind9-libs [20:18:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:18:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1761247100000&orgId=1&to=1761250733352 [20:19:00] !log [void@puppet181] Upgraded packages on reports171: bind9-host, bind9-dnsutils, and bind9-libs [20:19:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:19:15] !log [void@puppet181] Upgraded packages on rdb151: bind9-host, bind9-dnsutils, and bind9-libs [20:19:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:19:30] !log [void@puppet181] Upgraded packages on puppet181: bind9-host, bind9-dnsutils, and bind9-libs [20:19:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:19:48] !log [void@puppet181] Upgraded packages on swiftobject161: bind9-host, bind9-dnsutils, and bind9-libs [20:19:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:19:53] RECOVERY - os191 APT on os191 is OK: APT OK: 109 packages available for upgrade (0 critical updates). [20:19:55] RECOVERY - rdb151 APT on rdb151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [20:19:55] RECOVERY - reports171 APT on reports171 is OK: APT OK: 113 packages available for upgrade (0 critical updates). [20:19:58] RECOVERY - os202 APT on os202 is OK: APT OK: 72 packages available for upgrade (0 critical updates). [20:20:05] !log [void@puppet181] Upgraded packages on swiftobject151: bind9-host, bind9-dnsutils, and bind9-libs [20:20:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:20:10] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [20:20:23] RECOVERY - os201 APT on os201 is OK: APT OK: 81 packages available for upgrade (0 critical updates). [20:20:23] !log [void@puppet181] Upgraded packages on swiftac171: bind9-host, bind9-dnsutils, and bind9-libs [20:20:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:20:30] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 124 packages available for upgrade (0 critical updates). [20:20:41] !log [void@puppet181] Upgraded packages on swiftobject171: bind9-host, bind9-dnsutils, and bind9-libs [20:20:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:20:57] !log [void@puppet181] Upgraded packages on swiftobject181: bind9-host, bind9-dnsutils, and bind9-libs [20:21:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:21:14] !log [void@puppet181] Upgraded packages on swiftobject191: bind9-host, bind9-dnsutils, and bind9-libs [20:21:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:21:28] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [20:21:31] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 76 packages available for upgrade (0 critical updates). [20:21:31] !log [void@puppet181] Upgraded packages on test151: bind9-host, bind9-dnsutils, and bind9-libs [20:21:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:21:53] !log [void@puppet181] Upgraded packages on ns2: bind9-host, bind9-dnsutils, bind9-libs, and dnsutils [20:21:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:22:10] !log [void@puppet181] Upgraded packages on swiftproxy171: bind9-host, bind9-dnsutils, and bind9-libs [20:22:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:22:28] !log [void@puppet181] Upgraded packages on swiftobject201: bind9-host, bind9-dnsutils, and bind9-libs [20:22:30] RECOVERY - swiftobject191 APT on swiftobject191 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [20:22:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:22:34] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [20:22:46] !log [void@puppet181] Upgraded packages on swiftproxy161: bind9-host, bind9-dnsutils, and bind9-libs [20:22:51] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [20:22:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:23:07] RECOVERY - test151 APT on test151 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [20:23:16] RECOVERY - swiftobject201 APT on swiftobject201 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [20:23:18] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 126 packages available for upgrade (0 critical updates). [20:23:19] RECOVERY - ns2 APT on ns2 is OK: APT OK: 114 packages available for upgrade (0 critical updates). [20:23:40] RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 126 packages available for upgrade (0 critical updates). [20:28:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1761247700000&orgId=1&to=1761251300000 [20:32:38] [1/3] @everyone [20:32:39] [2/3] I'll help the first 10 people interested on how to  start earning $100k or more within a week, but you will reimburse me 10% of your profits when you receive it. Note: only interested people should send a friend request or send me a dm! ask me (HOW) via Telegram username @Official_JamesHarrington [20:32:39] [3/3] Or The telegram link in my bio [20:36:00] PROBLEM - db182 Disk Space on db182 is CRITICAL: DISK CRITICAL - free space: / 33472MiB (5% inode=99%); [20:42:00] RECOVERY - db182 Disk Space on db182 is OK: DISK OK - free space: / 212636MiB (35% inode=99%); [21:10:14] [02mw-config] 07SomeMWDev created 03T14133 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/32439eb4da64 [21:10:14] 02mw-config/03T14133 07SomeRandomDeveloper 0332439eb T14133: Fix SQL paths for StandardDialogs [21:10:41] !log [somerandomdeveloper@test151] starting deploy of {'config': True} to test151 [21:10:42] !log [somerandomdeveloper@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [21:10:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:10:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:12:26] [02mw-config] 07SomeMWDev opened pull request #6152: T14133: Fix SQL paths for StandardDialogs (03main...03T14133) 13https://github.com/miraheze/mw-config/pull/6152 [21:13:42] miraheze/mw-config - SomeMWDev the build passed. [21:14:01] [02mw-config] 07SomeMWDev merged pull request #6152: T14133: Fix SQL paths for StandardDialogs (03main...03T14133) 13https://github.com/miraheze/mw-config/pull/6152 [21:14:03] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/847f8c751f96d8ce7b4e7267c1a6673f236e00cd [21:14:04] !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [21:14:05] !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [21:14:05] [02mw-config] 07SomeMWDev 04deleted 03T14133 at 0332439eb 13https://api.github.com/repos/miraheze/mw-config/commit/32439eb [21:14:07] 02mw-config/03main 07SomeRandomDeveloper 03847f8c7 T14133: Fix SQL paths for StandardDialogs (#6152) [21:14:07] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [21:14:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:14:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:14:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:14:31] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 24s [21:14:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:15:07] miraheze/mw-config - SomeMWDev the build passed. [22:00:22] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'CSS'} to test151 [22:00:23] !log [somerandomdeveloper@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [22:00:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:00:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:04:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1761253460000&orgId=1&to=1761257093357 [22:06:13] !log [somerandomdeveloper@test151] starting deploy of {'force_upgrade': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'CSS'} to test151 [22:06:15] !log [somerandomdeveloper@test151] finished deploy of {'force_upgrade': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'CSS'} to test151 - SUCCESS in 1s [22:06:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:06:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:06:52] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'CSS'} to all [22:06:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:07:16] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'CSS'} to all - SUCCESS in 24s [22:07:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:09:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1761253520000&orgId=1&to=1761257120000 [23:09:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1761257360000&orgId=1&to=1761260993357 [23:14:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1761257420000&orgId=1&to=1761261020000 [23:31:55] [02mw-config] 07SomeMWDev pushed 2 new commits to 03T14400 13https://github.com/miraheze/mw-config/compare/fa773e4bb797...e03e64765911 [23:31:55] 02mw-config/03T14400 07SomeRandomDeveloper 032cc6327 Revert "Add TimedMediaHandler SQL to wgCreateWikiSQLFiles"… [23:31:55] 02mw-config/03T14400 07SomeRandomDeveloper 03e03e647 Remove TMH again temporarily [23:32:59] [02mw-config] 07SomeMWDev retitled PR #6143: "Restore default extension changes" ➜ "Partially restore default extension changes" 13https://github.com/miraheze/mw-config/pull/6143 [23:33:07] miraheze/mw-config - SomeMWDev the build passed. [23:33:56] [02mw-config] 07SomeMWDev readied pull request #6143: Partially restore default extension changes (03main...03T14400) 13https://github.com/miraheze/mw-config/pull/6143 [23:46:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1761259580000&orgId=1&to=1761263213356 [23:51:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1761259640000&orgId=1&to=1761263240000