[00:39:12] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php rebuildtextindex --wiki=pocketdragonswiki (START) [00:39:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:39:21] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php rebuildtextindex --wiki=pocketdragonswiki (END - exit=0) [00:39:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:42:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:45:35] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php cleanupTitles --wiki=pocketdragonswiki (END - exit=0) [00:45:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:57:16] !log [void@mwtask181] applied DataDump/sql/patches/patch-dumps_status.sql to lgballtwiki [00:57:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:03:22] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.08, 21.75, 17.45 [01:07:15] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.15, 22.71, 18.74 [01:09:12] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.20, 19.94, 18.22 [01:14:29] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [01:47:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:49:52] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:00:14] RECOVERY - db151 Backups SQL on db151 is OK: FILE_AGE OK: /var/log/sql-backup.log is 12 seconds old and 0 bytes [03:19:53] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:25:19] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.78, 20.10, 15.01 [03:29:12] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.31, 22.74, 17.28 [03:33:05] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.67, 23.93, 18.85 [03:36:32] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 29.15, 22.45, 15.52 [03:38:32] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.42, 23.49, 16.80 [03:41:39] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.75, 22.80, 17.81 [03:42:32] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 13.46, 18.00, 16.08 [03:45:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.07, 22.87, 19.09 [03:47:22] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.58, 24.76, 20.22 [03:50:40] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.23, 20.35, 16.54 [03:52:03] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.63, 22.56, 18.59 [03:55:08] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.76, 23.46, 18.97 [03:57:24] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 29.12, 23.39, 19.26 [03:57:55] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 27.15, 22.60, 19.66 [03:58:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:59:04] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.86, 22.24, 19.64 [03:59:58] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 29.39, 23.57, 18.00 [04:00:54] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.71, 23.64, 19.37 [04:01:02] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.29, 24.27, 20.70 [04:01:49] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.05, 23.75, 20.91 [04:03:00] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.13, 23.35, 20.85 [04:03:46] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 27.99, 25.42, 21.87 [04:06:55] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 26.64, 24.20, 21.62 [04:08:53] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.08, 22.81, 21.43 [04:08:56] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 15.88, 21.94, 21.55 [04:10:51] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.87, 24.86, 22.34 [04:10:51] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 26.12, 23.27, 22.04 [04:11:37] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.59, 23.28, 21.51 [04:13:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.46, 24.00, 21.94 [04:24:32] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.95, 22.81, 23.63 [04:24:36] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 16.86, 21.64, 22.93 [04:24:54] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 13.20, 20.03, 22.73 [04:26:32] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 28.99, 25.19, 24.41 [04:27:10] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.44, 23.93, 23.32 [04:29:39] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 11.72, 20.15, 23.18 [04:30:30] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 8.17, 15.61, 20.10 [04:30:32] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 7.90, 19.05, 22.42 [04:30:54] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 7.32, 15.17, 20.17 [04:32:23] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 5.75, 15.13, 21.88 [04:33:03] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 5.33, 12.93, 18.88 [04:33:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:33:39] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 5.65, 12.19, 19.20 [04:34:12] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 6.84, 13.98, 21.91 [04:34:22] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 7.12, 12.41, 20.06 [04:34:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 6.60, 14.43, 22.77 [04:34:32] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 5.71, 11.89, 18.71 [04:36:12] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 6.08, 11.21, 19.92 [04:38:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 8.47, 10.82, 19.37 [06:22:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.55, 18.56, 12.89 [06:24:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.69, 20.28, 14.27 [06:28:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.23, 21.40, 16.02 [06:30:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.81, 20.24, 16.24 [06:34:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.59, 24.82, 18.96 [06:38:54] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.17, 19.25, 14.77 [06:40:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.78, 19.65, 15.50 [06:46:31] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.62, 21.89, 17.49 [06:48:25] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.88, 19.22, 17.01 [06:53:22] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.17, 20.97, 16.99 [06:55:21] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.83, 20.78, 17.44 [06:56:12] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.72, 25.61, 20.72 [06:57:21] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.74, 21.35, 17.98 [06:58:31] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 29.06, 23.40, 17.76 [06:59:40] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.36, 22.67, 17.70 [07:00:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:01:11] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.67, 22.35, 16.64 [07:01:19] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.80, 23.88, 19.88 [07:02:53] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 26.45, 21.85, 18.36 [07:03:03] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.01, 21.54, 17.07 [07:03:18] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.24, 24.56, 20.58 [07:05:23] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.95, 21.51, 17.76 [07:06:43] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.95, 23.01, 19.65 [07:07:03] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.26, 23.57, 18.77 [07:07:20] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 16.65, 19.32, 17.38 [07:07:40] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.40, 23.66, 20.44 [07:07:59] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 15.83, 22.59, 20.74 [07:08:39] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 28.25, 24.96, 20.75 [07:11:03] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.84, 22.30, 19.49 [07:11:11] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 29.68, 24.92, 20.11 [07:11:40] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.99, 24.43, 21.39 [07:13:03] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.89, 23.89, 20.44 [07:15:03] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.29, 22.75, 20.46 [07:15:34] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.87, 23.82, 21.76 [07:19:03] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.31, 24.23, 21.42 [07:33:39] PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.013 second response time [07:34:15] PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 502 Bad Gateway [07:35:03] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.36, 23.68, 23.82 [07:35:07] PROBLEM - phorge171 php-fpm on phorge171 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [07:38:15] RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17717 bytes in 0.044 second response time [07:39:07] RECOVERY - phorge171 php-fpm on phorge171 is OK: PROCS OK: 9 processes with command name 'php-fpm8.2' [07:39:39] RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 19090 bytes in 0.061 second response time [07:39:40] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.68, 21.07, 23.58 [07:40:54] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 17.16, 21.24, 23.69 [07:44:24] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 13.57, 18.33, 23.02 [07:45:03] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 10.85, 14.39, 19.24 [07:46:32] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 11.41, 17.37, 22.71 [07:46:54] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 12.10, 14.90, 20.01 [07:47:00] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 12.93, 18.44, 23.15 [07:47:40] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 8.56, 13.70, 19.16 [07:48:12] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 9.71, 15.98, 22.52 [07:48:18] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 9.51, 14.17, 20.34 [07:50:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:50:32] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 9.62, 14.26, 20.30 [07:50:59] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 10.85, 14.22, 20.36 [07:52:12] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 9.20, 12.47, 19.62 [07:56:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.19, 18.82, 23.99 [08:10:32] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.54, 19.59, 18.02 [08:14:32] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 19.20, 18.97, 18.07 [08:14:49] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.66, 20.14, 19.05 [08:16:42] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.17, 19.67, 19.01 [08:18:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.56, 21.88, 21.67 [08:20:31] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.88, 22.50, 20.31 [08:22:25] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.28, 23.68, 21.03 [08:22:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.53, 23.55, 22.54 [08:25:03] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.12, 19.63, 15.96 [08:25:14] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.17, 21.59, 19.84 [08:25:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:27:03] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 13.83, 17.87, 15.79 [08:27:10] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.93, 23.37, 20.65 [08:28:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.42, 24.37, 23.08 [08:32:40] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.63, 21.69, 18.60 [08:33:39] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 27.12, 20.96, 17.44 [08:35:39] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.87, 21.40, 18.03 [08:37:39] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 19.51, 20.33, 18.03 [08:41:40] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.60, 23.01, 19.77 [08:41:40] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.07, 22.83, 19.73 [08:43:03] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.06, 20.38, 17.90 [08:43:40] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.20, 22.62, 20.04 [08:45:40] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.67, 24.71, 21.10 [08:46:54] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.92, 21.26, 18.09 [08:47:03] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.57, 19.98, 18.43 [08:47:39] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.01, 21.35, 19.86 [08:47:40] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.50, 23.13, 20.93 [08:48:54] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.33, 20.86, 18.33 [08:49:40] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.35, 19.94, 20.01 [08:50:32] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.84, 22.78, 22.59 [08:50:54] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.29, 19.51, 18.14 [08:51:03] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.78, 22.17, 19.68 [08:52:31] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.14, 24.62, 23.29 [08:53:39] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 15.55, 22.42, 21.29 [08:53:40] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 29.25, 24.84, 22.00 [08:55:03] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.95, 23.25, 20.63 [08:55:39] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.01, 22.97, 21.57 [08:55:40] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.38, 23.32, 21.82 [08:57:03] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 17.97, 21.60, 20.38 [08:57:39] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.64, 22.07, 21.41 [08:58:29] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.96, 23.11, 23.28 [08:59:03] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 13.05, 18.55, 19.41 [08:59:39] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.26, 24.04, 22.23 [09:01:28] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.09, 21.74, 19.24 [09:01:39] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 12.92, 20.13, 21.04 [09:03:21] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.16, 21.55, 19.50 [09:03:39] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 13.68, 18.15, 20.21 [09:03:40] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 13.21, 18.10, 20.00 [09:05:15] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.16, 19.73, 19.09 [09:08:25] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 14.33, 17.08, 20.15 [09:09:40] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.51, 19.74, 19.93 [09:11:40] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.96, 19.29, 19.76 [09:12:22] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.44, 19.72, 20.61 [09:14:21] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 13.43, 17.51, 19.70 [09:14:32] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 17.67, 22.29, 23.91 [09:14:54] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.58, 20.92, 19.40 [09:16:54] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 11.63, 17.34, 18.27 [09:18:12] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 11.52, 19.38, 23.95 [09:20:05] PROBLEM - test151 conntrack_table_size on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host [09:20:06] PROBLEM - mem151 Puppet on mem151 is CRITICAL: connect to address 10.0.15.113 port 5666: No route to hostconnect to host 10.0.15.113 port 5666: No route to host [09:20:06] PROBLEM - mem151 NTP time on mem151 is CRITICAL: connect to address 10.0.15.113 port 5666: No route to hostconnect to host 10.0.15.113 port 5666: No route to host [09:20:07] PROBLEM - graphite151 Puppet on graphite151 is CRITICAL: connect to address 10.0.15.145 port 5666: No route to hostconnect to host 10.0.15.145 port 5666: No route to host [09:20:08] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5627 bytes in 0.013 second response time [09:20:09] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:20:10] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [09:20:12] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5627 bytes in 0.012 second response time [09:20:13] PROBLEM - changeprop151 SSH on changeprop151 is CRITICAL: connect to address 10.0.15.148 and port 22: No route to host [09:20:16] PROBLEM - ping on changeprop151 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.148) [09:20:19] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 77% [09:20:19] PROBLEM - swiftobject151 SSH on swiftobject151 is CRITICAL: connect to address 10.0.15.117 and port 22: No route to host [09:20:20] PROBLEM - ping on mw152 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.115) [09:20:22] PROBLEM - ping on matomo151 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.112) [09:20:23] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is WARNING: WARNING - NGINX Error Rate is 59% [09:20:27] PROBLEM - test151 SSH on test151 is CRITICAL: connect to address 10.0.15.118 and port 22: No route to host [09:20:28] PROBLEM - ping on swiftobject151 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.117) [09:20:29] PROBLEM - Host mw152 is DOWN: CRITICAL - Host Unreachable (10.0.15.115) [09:20:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:20:30] PROBLEM - Host changeprop151 is DOWN: CRITICAL - Host Unreachable (10.0.15.148) [09:20:38] PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to matomo151.wikitide.net port 443 after 3078 ms: Couldn't connect to server [09:20:42] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 3.60, 15.90, 23.07 [09:20:43] PROBLEM - os151 NTP time on os151 is CRITICAL: connect to address 10.0.15.111 port 5666: No route to hostconnect to host 10.0.15.111 port 5666: No route to host [09:20:46] PROBLEM - os151 PowerDNS Recursor on os151 is CRITICAL: connect to address 10.0.15.111 port 5666: No route to hostconnect to host 10.0.15.111 port 5666: No route to host [09:20:46] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 1.84, 11.18, 18.80 [09:20:47] PROBLEM - Host mw151 is DOWN: CRITICAL - Host Unreachable (10.0.15.114) [09:20:49] PROBLEM - db151 Backups SQL on db151 is CRITICAL: connect to address 10.0.15.110 port 5666: No route to hostconnect to host 10.0.15.110 port 5666: No route to host [09:20:49] PROBLEM - db151 PowerDNS Recursor on db151 is CRITICAL: connect to address 10.0.15.110 port 5666: No route to hostconnect to host 10.0.15.110 port 5666: No route to host [09:20:49] PROBLEM - os151 Current Load on os151 is CRITICAL: connect to address 10.0.15.111 port 5666: No route to hostconnect to host 10.0.15.111 port 5666: No route to host [09:20:49] PROBLEM - os151 Puppet on os151 is CRITICAL: connect to address 10.0.15.111 port 5666: No route to hostconnect to host 10.0.15.111 port 5666: No route to host [09:20:50] PROBLEM - Host graphite151 is DOWN: CRITICAL - Host Unreachable (10.0.15.145) [09:20:51] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: connect to address 10.0.15.116 port 5666: No route to hostconnect to host 10.0.15.116 port 5666: No route to host [09:20:51] PROBLEM - rdb151 NTP time on rdb151 is CRITICAL: connect to address 10.0.15.142 port 5666: No route to hostconnect to host 10.0.15.142 port 5666: No route to host [09:20:51] PROBLEM - rdb151 Redis Process on rdb151 is CRITICAL: connect to address 10.0.15.142 port 5666: No route to hostconnect to host 10.0.15.142 port 5666: No route to host [09:20:52] PROBLEM - db151 Backups SQL mhglobal on db151 is CRITICAL: connect to address 10.0.15.110 port 5666: No route to hostconnect to host 10.0.15.110 port 5666: No route to host [09:20:52] PROBLEM - mem151 ferm_active on mem151 is CRITICAL: connect to address 10.0.15.113 port 5666: No route to hostconnect to host 10.0.15.113 port 5666: No route to host [09:20:52] PROBLEM - mem151 Disk Space on mem151 is CRITICAL: connect to address 10.0.15.113 port 5666: No route to hostconnect to host 10.0.15.113 port 5666: No route to host [09:20:54] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki [09:20:54] PROBLEM - rdb151 Disk Space on rdb151 is CRITICAL: connect to address 10.0.15.142 port 5666: No route to hostconnect to host 10.0.15.142 port 5666: No route to host [09:20:54] PROBLEM - rdb151 conntrack_table_size on rdb151 is CRITICAL: connect to address 10.0.15.142 port 5666: No route to hostconnect to host 10.0.15.142 port 5666: No route to host [09:20:54] PROBLEM - rdb151 poolcounter process on rdb151 is CRITICAL: connect to address 10.0.15.142 port 5666: No route to hostconnect to host 10.0.15.142 port 5666: No route to host [09:21:06] PROBLEM - prometheus151 ferm_active on prometheus151 is CRITICAL: connect to address 10.0.15.116 port 5666: No route to hostconnect to host 10.0.15.116 port 5666: No route to host [09:21:06] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: connect to address 10.0.15.116 port 5666: No route to hostconnect to host 10.0.15.116 port 5666: No route to host [09:21:07] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: connect to address 10.0.15.116 port 5666: No route to hostconnect to host 10.0.15.116 port 5666: No route to host [09:21:07] PROBLEM - test151 Disk Space on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host [09:21:08] PROBLEM - swiftobject151 Puppet on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [09:21:09] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [09:21:09] PROBLEM - mem151 SSH on mem151 is CRITICAL: connect to address 10.0.15.113 and port 22: No route to host [09:21:09] PROBLEM - ping on mem151 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.113) [09:21:10] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:21:10] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki [09:21:11] PROBLEM - swiftobject151 conntrack_table_size on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [09:21:11] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [09:21:12] PROBLEM - test151 PowerDNS Recursor on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host [09:21:13] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:21:15] PROBLEM - db151 Puppet on db151 is CRITICAL: connect to address 10.0.15.110 port 5666: No route to hostconnect to host 10.0.15.110 port 5666: No route to host [09:21:15] PROBLEM - db151 SSH on db151 is CRITICAL: connect to address 10.0.15.110 and port 22: No route to host [09:21:15] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5627 bytes in 0.013 second response time [09:21:16] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: connect to address 10.0.15.116 and port 22: No route to host [09:21:17] PROBLEM - rdb151 Current Load on rdb151 is CRITICAL: connect to address 10.0.15.142 port 5666: No route to hostconnect to host 10.0.15.142 port 5666: No route to host [09:21:17] PROBLEM - matomo151 PowerDNS Recursor on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [09:21:17] PROBLEM - matomo151 Redis Process on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [09:21:29] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: connect to address 10.0.15.116 port 5666: No route to hostconnect to host 10.0.15.116 port 5666: No route to host [09:21:30] PROBLEM - cloud15 SSH on cloud15 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:21:37] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [09:21:39] PROBLEM - matomo151 SSH on matomo151 is CRITICAL: connect to address 10.0.15.112 and port 22: No route to host [09:21:40] PROBLEM - matomo151 php-fpm on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [09:21:40] PROBLEM - Host matomo151 is DOWN: CRITICAL - Host Unreachable (10.0.15.112) [09:21:41] PROBLEM - db151 NTP time on db151 is CRITICAL: connect to address 10.0.15.110 port 5666: No route to hostconnect to host 10.0.15.110 port 5666: No route to host [09:21:41] PROBLEM - db151 MariaDB Connections on db151 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db151.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_conne [09:21:41] on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db151.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [09:21:42] PROBLEM - Host prometheus151 is DOWN: CRITICAL - Host Unreachable (10.0.15.116) [09:21:43] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki [09:21:46] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki [09:21:47] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:21:49] PROBLEM - Host db151 is DOWN: CRITICAL - Host Unreachable (10.0.15.110) [09:21:50] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [09:21:50] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5627 bytes in 0.017 second response time [09:21:50] PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 100% [09:21:56] PROBLEM - Host cloud15 is DOWN: PING CRITICAL - Packet loss = 100% [09:21:57] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki [09:22:12] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 1.44, 9.43, 18.81 [09:24:15] Is it just me, or is Miraheze 503/502-ing? meta.miraheze.org and rainverse.miraheze.org gives me 502, rainverse.wiki gives me 503 (Backend fetch failed via cp51.wikitide.net at Sun, 26 May 2024 09:22:31 GMT; Varnish XID 6265578, serving 2403:4800:3202:de23:f64a:875a:5b18:d5de, 127.0.0.1 (your IP!)) [09:24:35] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 1.48, 8.05, 18.24 [09:29:06] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is WARNING: WARNING - NGINX Error Rate is 43% [09:30:21] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [09:30:42] PROBLEM - os161 Puppet on os161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Opensearch_template[graylog-internal] [09:32:21] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is WARNING: WARNING - NGINX Error Rate is 58% [09:34:21] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [09:35:06] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 38% [09:35:57] RECOVERY - Host cloud15 is UP: PING OK - Packet loss = 0%, RTA = 0.39 ms [09:36:05] RECOVERY - cloud15 SSH on cloud15 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [09:36:21] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is WARNING: WARNING - NGINX Error Rate is 49% [09:36:41] RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.19 ms [09:42:21] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [09:55:09] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is WARNING: WARNING - NGINX Error Rate is 54% [09:55:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:59:09] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 39% [09:59:59] !log depool mw151/152 in cf from * [10:03:13] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is WARNING: WARNING - NGINX Error Rate is 43% [10:05:55] [02mw-config] 07redbluegreenhat opened pull request 03#5574: mark c1 as down - 13https://github.com/miraheze/mw-config/pull/5574 [10:06:56] [02mw-config] 07redbluegreenhat closed pull request 03#5574: mark c1 as down - 13https://github.com/miraheze/mw-config/pull/5574 [10:06:57] !log [@mwtask171] starting deploy of {'config': True} to all [10:06:58] [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/37beaa78521e...eb3356bd3b19 [10:07:00] [02mw-config] 07redbluegreenhat 03eb3356b - mark c1 as down (#5574) [10:07:05] miraheze/mw-config - redbluegreenhat the build passed. [10:07:36] !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [10:07:57] miraheze/mw-config - redbluegreenhat the build passed. [10:07:57] !log [alex@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw151.wikitide.net [10:08:42] !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [10:09:19] !log [@mwtask171] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw151.wikitide.net [10:11:30] !log [alex@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 168s [10:21:15] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 39% [10:25:16] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is WARNING: WARNING - NGINX Error Rate is 44% [10:29:17] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 35% [10:35:18] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is WARNING: WARNING - NGINX Error Rate is 46% [10:36:33] !log restarted nginx on mw161 [10:37:02] oh, right, logging doesn't work [10:37:59] !log restarted php8.2-fpm on mw161 [10:39:18] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 37% [10:45:19] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is WARNING: WARNING - NGINX Error Rate is 49% [10:53:28] RECOVERY - Host test151 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [10:53:33] RECOVERY - test151 Redis Process on test151 is OK: PROCS OK: 1 process with args 'redis-server' [10:53:34] RECOVERY - test151 PowerDNS Recursor on test151 is OK: DNS OK: 0.038 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [10:53:55] RECOVERY - test151 conntrack_table_size on test151 is OK: OK: nf_conntrack is 0 % full [10:53:56] PROBLEM - test151 NTP time on test151 is CRITICAL: NTP CRITICAL: Offset 0.9506341815 secs [10:54:01] RECOVERY - test151 poolcounter process on test151 is OK: PROCS OK: 1 process with UID = 999 (poolcounter), command name 'poolcounterd' [10:54:02] RECOVERY - test151 php-fpm on test151 is OK: PROCS OK: 13 processes with command name 'php-fpm8.2' [10:54:04] RECOVERY - test151 APT on test151 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [10:54:14] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 6.882 second response time [10:54:19] RECOVERY - Host changeprop151 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms [10:54:21] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is WARNING: WARNING - NGINX Error Rate is 58% [10:54:22] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.617 second response time [10:54:25] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 0.588 second response time [10:54:28] RECOVERY - Host mw151 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [10:54:28] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 0.072 second response time [10:54:29] RECOVERY - Host mw152 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [10:54:30] !log [@test151] starting deploy of {'config': True} to test151 [10:54:30] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 0.061 second response time [10:54:31] RECOVERY - ping on mw152 is OK: PING OK - Packet loss = 0%, RTA = 0.25 ms [10:54:31] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [10:54:32] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 0.062 second response time [10:54:32] RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 0.083 second response time [10:54:33] RECOVERY - test151 Current Load on test151 is OK: LOAD OK - total load average: 0.51, 0.16, 0.06 [10:54:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:54:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:54:45] RECOVERY - Host graphite151 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [10:54:50] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.703 second response time [10:54:51] PROBLEM - mw152 NTP time on mw152 is WARNING: NTP WARNING: Offset 0.4848535359 secs [10:54:54] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 0.059 second response time [10:54:55] RECOVERY - cp51 HTTPS on cp51 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3782 bytes in 1.041 second response time [10:55:04] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [10:55:06] PROBLEM - changeprop151 Puppet on changeprop151 is WARNING: WARNING: Puppet last ran 2 hours ago [10:55:11] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 17% [10:55:11] PROBLEM - mw152 Puppet on mw152 is WARNING: WARNING: Puppet last ran 2 hours ago [10:55:15] RECOVERY - Host swiftobject151 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [10:55:16] PROBLEM - mw151 NTP time on mw151 is CRITICAL: NTP CRITICAL: Offset 0.656027317 secs [10:55:17] RECOVERY - cp27 HTTPS on cp27 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3783 bytes in 0.894 second response time [10:55:17] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.324 second response time [10:55:21] RECOVERY - Host os151 is UP: PING OK - Packet loss = 0%, RTA = 3.08 ms [10:55:21] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 36% [10:55:26] RECOVERY - test151 SSH on test151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [10:55:26] RECOVERY - test151 Disk Space on test151 is OK: DISK OK - free space: / 63761MiB (72% inode=84%); [10:55:27] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [10:55:28] RECOVERY - Host rdb151 is UP: PING OK - Packet loss = 0%, RTA = 0.36 ms [10:55:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:55:31] RECOVERY - Host mem151 is UP: PING OK - Packet loss = 0%, RTA = 0.20 ms [10:55:37] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3784 bytes in 0.078 second response time [10:55:37] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.535 second response time [10:55:41] RECOVERY - Host matomo151 is UP: PING OK - Packet loss = 0%, RTA = 0.83 ms [10:55:41] RECOVERY - Host prometheus151 is UP: PING OK - Packet loss = 0%, RTA = 0.41 ms [10:55:43] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [10:55:43] RECOVERY - matomo151 NTP time on matomo151 is OK: NTP OK: Offset -0.001078605652 secs [10:55:43] RECOVERY - Host db151 is UP: PING OK - Packet loss = 0%, RTA = 0.65 ms [10:55:46] RECOVERY - db151 NTP time on db151 is OK: NTP OK: Offset 0.04454487562 secs [10:55:46] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [10:55:49] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.351 second response time [10:55:50] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3762 bytes in 0.072 second response time [10:55:52] RECOVERY - matomo151 Disk Space on matomo151 is OK: DISK OK - free space: / 9017MiB (50% inode=90%); [10:55:57] RECOVERY - test151 NTP time on test151 is OK: NTP OK: Offset 0.002850949764 secs [10:55:57] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [10:55:57] RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3784 bytes in 0.798 second response time [10:56:00] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.384 second response time [10:56:01] RECOVERY - changeprop151 SSH on changeprop151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [10:56:06] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:56:11] RECOVERY - graphite151 Puppet on graphite151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:56:16] RECOVERY - swiftobject151 SSH on swiftobject151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) RECOVERY - mem151 PowerDNS Recursor on mem151 is OK: DNS OK: 0.045 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [10:56:16] RECOVERY - db151 Backups SQL on db151 is OK: FILE_AGE OK: /var/log/sql-backup.log is 5885 seconds old and 106503 bytes [10:56:19] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 12% [10:56:21] RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 7% [10:56:21] [02mw-config] 07redbluegreenhat created branch 03revert-5574-c1-down - 13https://github.com/miraheze/mw-config [10:56:21] RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:56:21] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 0.59, 0.24, 0.09 [10:56:21] RECOVERY - ping on changeprop151 is OK: PING OK - Packet loss = 0%, RTA = 1.01 ms [10:56:24] RECOVERY - cp26 HTTPS on cp26 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3760 bytes in 1.264 second response time [10:56:24] [02mw-config] 07redbluegreenhat pushed 031 commit to 03revert-5574-c1-down [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/6bc8345e53a0 [10:56:26] [02mw-config] 07redbluegreenhat 036bc8345 - Revert "mark c1 as down (#5574)" [10:56:26] RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 552 bytes in 0.217 second response time [10:56:26] RECOVERY - mem151 NTP time on mem151 is OK: NTP OK: Offset 0.002567827702 secs [10:56:26] RECOVERY - ping on swiftobject151 is OK: PING OK - Packet loss = 0%, RTA = 0.26 ms [10:56:27] [02mw-config] 07redbluegreenhat opened pull request 03#5575: Revert "mark c1 as down" - 13https://github.com/miraheze/mw-config/pull/5575 [10:56:31] [02mw-config] 07redbluegreenhat closed pull request 03#5575: Revert "mark c1 as down" - 13https://github.com/miraheze/mw-config/pull/5575 [10:56:31] RECOVERY - rdb151 Redis Process on rdb151 is OK: PROCS OK: 1 process with args 'redis-server' [10:56:31] RECOVERY - mem151 Disk Space on mem151 is OK: DISK OK - free space: / 5810MiB (66% inode=86%); [10:56:31] RECOVERY - rdb151 NTP time on rdb151 is OK: NTP OK: Offset 0.006193190813 secs [10:56:31] RECOVERY - rdb151 conntrack_table_size on rdb151 is OK: OK: nf_conntrack is 0 % full [10:56:34] [02mw-config] 07redbluegreenhat deleted branch 03revert-5574-c1-down [10:56:36] RECOVERY - db151 conntrack_table_size on db151 is OK: OK: nf_conntrack is 0 % full [10:56:36] RECOVERY - rdb151 poolcounter process on rdb151 is OK: PROCS OK: 1 process with UID = 999 (poolcounter), command name 'poolcounterd' [10:56:36] RECOVERY - rdb151 Disk Space on rdb151 is OK: DISK OK - free space: / 5817MiB (66% inode=86%); [10:56:37] [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/eb3356bd3b19...be5ca9fb2ccf [10:56:39] [02mw-config] 07redbluegreenhat 03be5ca9f - Revert "mark c1 as down" (#5575) [10:56:41] RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [10:56:41] RECOVERY - os151 NTP time on os151 is OK: NTP OK: Offset 0.002594023943 secs [10:56:42] [02mw-config] 07redbluegreenhat deleted branch 03revert-5574-c1-down - 13https://github.com/miraheze/mw-config [10:56:43] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [10:56:46] RECOVERY - os151 PowerDNS Recursor on os151 is OK: DNS OK: 0.109 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [10:56:46]