[00:00:54] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.73, 22.26, 19.13 [00:02:53] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.28, 21.70, 19.31 [00:06:51] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.39, 18.27, 18.50 [00:15:43] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.57, 22.13, 20.06 [00:17:42] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.87, 21.12, 19.97 [00:25:37] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.19, 24.37, 21.71 [00:29:35] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.49, 23.38, 21.97 [00:30:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.99, 18.89, 16.24 [00:31:34] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.93, 24.39, 22.52 [00:32:01] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 19.91, 19.16, 16.66 [00:33:33] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.36, 23.91, 22.58 [00:46:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.02, 21.93, 23.91 [00:48:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.55, 23.75, 24.36 [00:52:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.42, 22.79, 23.87 [00:54:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.95, 24.65, 24.41 [00:55:23] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.36, 21.40, 21.34 [00:57:23] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.41, 20.84, 21.13 [01:05:23] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.02, 22.75, 21.87 [01:07:23] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.66, 21.80, 21.67 [01:13:23] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 13.72, 17.82, 20.09 [01:14:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.94, 20.74, 23.17 [01:20:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.96, 23.25, 23.43 [01:22:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.23, 21.85, 22.91 [01:24:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.43, 23.35, 23.31 [01:26:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 24.00, 23.16, 23.23 [01:27:17] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.71, 22.84, 20.77 [01:28:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.57, 24.51, 23.75 [01:29:28] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.81, 18.69, 15.64 [01:31:27] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.27, 18.06, 15.76 [01:40:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.65, 23.37, 23.98 [01:44:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.48, 23.36, 23.78 [01:46:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.32, 23.48, 23.79 [01:48:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.87, 23.83, 23.87 [01:50:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.28, 23.18, 23.63 [01:52:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.75, 23.06, 23.48 [01:54:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.36, 23.21, 23.52 [01:55:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:58:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.39, 23.39, 23.46 [02:00:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.60, 22.34, 23.08 [02:05:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:18:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.06, 22.06, 22.09 [02:24:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.73, 22.89, 22.83 [02:28:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.82, 24.93, 23.53 [02:30:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.67, 23.87, 23.32 [02:32:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.56, 24.21, 23.52 [02:36:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.36, 23.49, 23.44 [02:38:34] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 15.57, 20.47, 23.38 [02:54:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.50, 22.43, 22.16 [02:54:25] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.06, 22.93, 21.91 [02:56:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.78, 22.50, 22.23 [02:56:23] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.70, 23.10, 22.11 [02:58:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.76, 24.86, 23.13 [03:04:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.71, 23.89, 23.29 [03:06:17] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.85, 18.83, 20.35 [03:12:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.35, 22.36, 22.46 [03:14:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.10, 21.20, 21.99 [03:16:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.55, 22.73, 22.48 [03:18:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.28, 21.86, 22.17 [03:24:20] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.01, 18.31, 20.40 [03:28:21] [02mw-config] 07clarfonthey commented on pull request 03#5566: Changes to how special pages are treated - 13https://github.com/miraheze/mw-config/pull/5566#issuecomment-2119596238 [03:30:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.35, 21.67, 21.20 [03:34:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.46, 22.10, 21.40 [03:36:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.34, 21.11, 21.14 [03:36:57] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.51, 20.32, 19.20 [03:40:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.57, 22.32, 21.57 [03:40:55] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.35, 19.85, 19.31 [03:44:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.49, 22.58, 21.90 [03:50:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.69, 22.43, 21.77 [03:53:28] [02mw-config] 07colleirose commented on pull request 03#5566: Changes to how special pages are treated - 13https://github.com/miraheze/mw-config/pull/5566#issuecomment-2119612665 [03:54:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.09, 21.74, 21.75 [03:56:44] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.42, 20.96, 19.64 [03:58:43] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 20.22, 20.15, 19.48 [04:02:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.57, 22.80, 22.27 [04:04:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.30, 21.98, 22.03 [04:10:15] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 45.53, 25.87, 10.72 [04:12:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.64, 22.59, 22.14 [04:14:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.12, 21.57, 21.81 [04:16:13] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.72, 8.38, 7.55 [04:22:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.56, 22.56, 22.14 [04:24:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 12.51, 18.86, 20.86 [04:26:20] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 8.84, 15.21, 19.27 [04:41:56] [02MirahezeMagic] 07colleirose synchronize pull request 03#491: Add SEO tweaks and slight formatting enhancements - 13https://github.com/miraheze/MirahezeMagic/pull/491 [04:42:06] [02MirahezeMagic] 07coderabbitai[bot] edited a comment on pull request 03#491: Add SEO tweaks and slight formatting enhancements - 13https://github.com/miraheze/MirahezeMagic/pull/491#issuecomment-2119377566 [04:42:40] [02MirahezeMagic] 07colleirose commented on pull request 03#491: Add SEO tweaks and slight formatting enhancements - 13https://github.com/miraheze/MirahezeMagic/pull/491#issuecomment-2119655646 [04:43:29] [02MirahezeMagic] 07colleirose edited pull request 03#491: Add SEO tweaks and slight formatting enhancements - 13https://github.com/miraheze/MirahezeMagic/pull/491 [04:43:35] [02MirahezeMagic] 07colleirose edited pull request 03#491: Add SEO tweaks - 13https://github.com/miraheze/MirahezeMagic/pull/491 [04:43:36] [02MirahezeMagic] 07coderabbitai[bot] edited pull request 03#491: Add SEO tweaks - 13https://github.com/miraheze/MirahezeMagic/pull/491 [04:44:31] [02MirahezeMagic] 07coderabbitai[bot] reviewed pull request 03#491 commit - 13https://github.com/miraheze/MirahezeMagic/pull/491#discussion_r1606241123 [04:44:33] [02MirahezeMagic] 07coderabbitai[bot] reviewed pull request 03#491 commit - 13https://github.com/miraheze/MirahezeMagic/pull/491#discussion_r1606241119 [04:44:36] [02MirahezeMagic] 07coderabbitai[bot] edited a comment on pull request 03#491: Add SEO tweaks - 13https://github.com/miraheze/MirahezeMagic/pull/491#issuecomment-2119377566 [04:44:51] [02MirahezeMagic] 07colleirose synchronize pull request 03#491: Add SEO tweaks - 13https://github.com/miraheze/MirahezeMagic/pull/491 [04:44:56] [02MirahezeMagic] 07coderabbitai[bot] edited a comment on pull request 03#491: Add SEO tweaks - 13https://github.com/miraheze/MirahezeMagic/pull/491#issuecomment-2119377566 [04:47:35] [02MirahezeMagic] 07colleirose synchronize pull request 03#491: Add SEO tweaks - 13https://github.com/miraheze/MirahezeMagic/pull/491 [04:47:40] [02MirahezeMagic] 07coderabbitai[bot] edited a comment on pull request 03#491: Add SEO tweaks - 13https://github.com/miraheze/MirahezeMagic/pull/491#issuecomment-2119377566 [04:48:12] [02MirahezeMagic] 07colleirose reviewed pull request 03#491 commit - 13https://github.com/miraheze/MirahezeMagic/pull/491#discussion_r1606243085 [04:48:15] [02MirahezeMagic] 07coderabbitai[bot] reviewed pull request 03#491 commit - 13https://github.com/miraheze/MirahezeMagic/pull/491#discussion_r1606241123 [04:48:30] [02MirahezeMagic] 07coderabbitai[bot] reviewed pull request 03#491 commit - 13https://github.com/miraheze/MirahezeMagic/pull/491#discussion_r1606243259 [04:49:11] [02MirahezeMagic] 07coderabbitai[bot] edited pull request 03#491: Add SEO tweaks - 13https://github.com/miraheze/MirahezeMagic/pull/491 [04:49:14] [02MirahezeMagic] 07coderabbitai[bot] edited a comment on pull request 03#491: Add SEO tweaks - 13https://github.com/miraheze/MirahezeMagic/pull/491#issuecomment-2119377566 [04:49:35] [02MirahezeMagic] 07colleirose synchronize pull request 03#491: Add SEO tweaks - 13https://github.com/miraheze/MirahezeMagic/pull/491 [04:49:39] [02MirahezeMagic] 07coderabbitai[bot] edited a comment on pull request 03#491: Add SEO tweaks - 13https://github.com/miraheze/MirahezeMagic/pull/491#issuecomment-2119377566 [04:50:40] [02MirahezeMagic] 07colleirose edited pull request 03#491: Add SEO tweaks - 13https://github.com/miraheze/MirahezeMagic/pull/491 [05:09:17] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 141.35 ms [05:11:17] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 140.53 ms [05:44:57] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 141.00 ms [05:48:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.66, 17.21, 12.13 [05:48:57] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 141.06 ms [05:50:32] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.57, 18.44, 14.11 [05:52:15] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#491 commit - 13https://github.com/miraheze/MirahezeMagic/pull/491#discussion_r1606278636 [05:52:16] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#491 commit - 13https://github.com/miraheze/MirahezeMagic/pull/491#discussion_r1606277877 [05:52:19] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#491 commit - 13https://github.com/miraheze/MirahezeMagic/pull/491#discussion_r1606279592 [05:52:22] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#491 commit - 13https://github.com/miraheze/MirahezeMagic/pull/491#discussion_r1606278716 [05:52:23] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#491 commit - 13https://github.com/miraheze/MirahezeMagic/pull/491#discussion_r1606278351 [05:52:24] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#491 commit - 13https://github.com/miraheze/MirahezeMagic/pull/491#discussion_r1606278946 [05:52:27] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#491 commit - 13https://github.com/miraheze/MirahezeMagic/pull/491#discussion_r1606279301 [05:54:29] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.22, 20.80, 15.94 [05:54:45] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#491 commit - 13https://github.com/miraheze/MirahezeMagic/pull/491#discussion_r1606282997 [05:56:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.93, 22.60, 16.75 [05:56:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.29, 19.36, 15.94 [06:12:15] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.14, 21.55, 19.10 [06:16:12] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.73, 22.73, 20.20 [06:24:08] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.79, 22.79, 21.04 [06:28:05] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.22, 23.06, 21.69 [06:53:50] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.05, 22.46, 21.48 [06:55:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.80, 22.69, 21.75 [07:11:39] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.33, 18.86, 19.99 [07:14:33] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 141.11 ms [07:16:36] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 141.21 ms [07:24:29] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.40, 20.88, 19.77 [07:26:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.39, 20.32, 19.72 [07:34:23] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.73, 21.17, 19.81 [07:36:22] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.60, 20.96, 19.91 [07:44:17] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.42, 20.29, 20.11 [07:48:14] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.75, 23.78, 21.56 [07:50:12] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.17, 23.79, 21.89 [07:52:11] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.23, 24.36, 22.35 [07:54:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.15, 22.02, 21.71 [07:54:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.08, 22.69, 23.81 [07:56:09] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.17, 23.17, 22.16 [07:56:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.04, 24.77, 24.42 [07:58:08] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.29, 23.36, 22.36 [08:06:03] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.17, 18.13, 20.17 [08:40:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.66, 21.26, 23.67 [08:50:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.12, 23.75, 23.49 [08:51:33] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.74, 21.40, 18.87 [08:53:32] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.69, 20.88, 18.98 [08:59:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.47, 23.22, 20.52 [09:01:27] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.94, 22.35, 20.50 [09:21:23] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.69, 22.73, 21.52 [09:27:23] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.98, 23.08, 22.21 [09:33:23] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 13.07, 17.09, 19.92 [09:37:23] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.91, 19.71, 20.34 [09:39:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:47:23] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.04, 21.84, 21.00 [09:49:23] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.45, 21.68, 21.06 [09:53:23] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.66, 23.78, 21.95 [10:01:23] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.66, 23.73, 23.10 [10:09:23] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.12, 22.99, 22.79 [10:11:23] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.07, 22.45, 22.63 [10:14:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:17:23] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.20, 22.88, 22.65 [10:19:23] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.04, 22.94, 22.70 [10:23:23] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.06, 23.16, 22.81 [10:29:23] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.99, 22.68, 22.99 [10:39:23] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.65, 20.83, 21.31 [10:41:23] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.48, 20.78, 21.20 [10:43:23] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.29, 22.65, 21.80 [10:47:23] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.85, 23.30, 22.37 [11:09:23] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.15, 22.52, 21.65 [11:11:23] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.96, 23.03, 21.96 [11:21:23] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.05, 21.48, 21.13 [11:25:23] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.48, 23.30, 22.00 [11:30:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.39, 22.33, 23.91 [11:37:23] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.83, 18.65, 20.14 [11:46:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.06, 20.46, 21.31 [11:48:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.13, 21.40, 21.61 [11:56:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.98, 23.17, 22.00 [12:00:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.15, 22.64, 22.20 [12:02:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.08, 24.60, 22.96 [12:03:12] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.92, 21.40, 19.99 [12:05:11] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.08, 21.44, 20.11 [12:07:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.57, 21.36, 20.30 [12:11:02] [02ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/ImportDump/compare/815ce1239132...7d62a4c3098e [12:11:03] [02ImportDump] 07translatewiki 037d62a4c - Localisation updates from https://translatewiki.net. [12:11:06] [02MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/MirahezeMagic/compare/87e487517fed...1775d037e39c [12:11:07] [02MirahezeMagic] 07translatewiki 031775d03 - Localisation updates from https://translatewiki.net. [12:11:08] [02DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/dd5d6df8b79a...99ebb9e132cd [12:11:11] [02DataDump] 07translatewiki 0399ebb9e - Localisation updates from https://translatewiki.net. [12:11:13] [02WikiDiscover] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/WikiDiscover/compare/292a228b59cf...2dcd9f9089ee [12:11:15] [02WikiDiscover] 07translatewiki 032dcd9f9 - Localisation updates from https://translatewiki.net. [12:12:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.41, 23.60, 23.64 [12:14:51] miraheze/MirahezeMagic - translatewiki the build has errored. [12:15:05] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 20.01, 20.33, 20.40 [12:15:59] miraheze/WikiDiscover - translatewiki the build passed. [12:16:03] miraheze/DataDump - translatewiki the build passed. [12:19:01] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.17, 20.87, 20.53 [12:19:06] miraheze/ImportDump - translatewiki the build passed. [12:20:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.86, 22.31, 22.81 [12:22:59] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.79, 24.23, 21.89 [12:23:51] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [12:23:52] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:03] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [12:24:04] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:16] !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 [12:24:17] !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:25:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:25:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:26:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.42, 23.73, 23.47 [12:28:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.50, 25.71, 24.23 [12:32:26] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:32:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:39] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 13s [12:32:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:51] !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:33:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:04] !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 12s [12:33:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:52] !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:37:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:03] !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 11s [12:38:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:15] !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:38:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:24] !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 9s [12:38:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:40:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.68, 22.81, 23.59 [12:43:15] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 141.77 ms [12:44:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.30, 22.73, 23.83 [12:46:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.87, 23.15, 23.81 [12:47:15] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 142.38 ms [12:48:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.36, 22.06, 23.37 [12:51:24] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 141.93 ms [12:52:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.10, 22.54, 23.27 [12:52:41] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.52, 21.99, 22.07 [12:54:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.21, 22.77, 23.27 [12:58:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.11, 22.96, 23.14 [13:00:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.76, 21.54, 22.59 [13:02:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.18, 23.62, 23.22 [13:02:35] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.89, 23.64, 23.14 [13:12:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.95, 23.06, 23.52 [13:13:50] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 143.00 ms [13:16:27] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.03, 17.33, 20.08 [13:18:00] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 145.74 ms [13:36:20] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.83, 18.22, 20.15 [13:40:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.18, 21.22, 21.01 [13:42:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.19, 22.17, 21.37 [13:44:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.70, 20.62, 20.89 [13:44:24] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 144.45 ms [13:47:05] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.52, 22.40, 20.44 [13:52:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.36, 24.51, 22.55 [13:54:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.34, 22.53, 22.05 [13:56:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.65, 24.08, 22.67 [14:00:57] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.31, 23.63, 22.79 [14:04:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.90, 23.16, 23.15 [14:04:54] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.93, 24.10, 23.16 [14:06:53] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.41, 23.82, 23.17 [14:10:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.09, 22.47, 22.66 [14:12:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.14, 22.03, 22.49 [14:14:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.08, 23.70, 23.02 [14:16:47] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.83, 23.41, 22.99 [14:24:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.55, 23.52, 23.73 [14:30:39] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.54, 22.70, 23.73 [14:36:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.18, 21.69, 22.34 [14:38:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.99, 21.14, 22.07 [14:42:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.11, 23.31, 22.72 [14:44:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.78, 22.34, 22.41 [14:52:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.23, 22.65, 22.45 [14:52:26] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.92, 23.31, 22.29 [14:54:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.22, 22.83, 22.54 [14:56:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.44, 24.35, 23.14 [14:56:23] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.26, 23.85, 22.79 [14:58:22] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.52, 24.67, 23.24 [15:00:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.46, 23.63, 23.25 [15:02:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.37, 23.49, 23.16 [15:02:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.68, 23.81, 23.31 [15:04:18] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.26, 24.93, 23.71 [15:06:17] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.87, 22.37, 22.94 [15:06:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.24, 22.95, 23.15 [15:08:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.91, 24.99, 23.88 [15:10:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.04, 23.40, 23.43 [15:12:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.37, 25.36, 24.14 [15:20:49] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.401 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [15:22:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.54, 23.85, 23.86 [15:24:06] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.12, 20.69, 21.12 [15:24:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.59, 23.87, 23.86 [15:26:05] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.02, 20.64, 21.11 [15:30:03] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.84, 19.10, 20.32 [15:33:59] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.39, 22.96, 21.52 [15:37:57] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.99, 23.32, 22.14 [15:49:50] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.41, 24.27, 22.67 [15:51:16] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 29.09, 21.94, 18.00 [15:52:27] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 29.19, 22.70, 17.03 [15:52:59] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw181 [15:54:59] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [15:55:36] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw182 [15:56:28] PROBLEM - en.religiononfire.mar.in.ua - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.religiononfire.mar.in.ua' expires in 15 day(s) (Wed 05 Jun 2024 03:36:41 PM GMT +0000). [15:56:39] PROBLEM - uk.religiononfire.mar.in.ua - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'uk.religiononfire.mar.in.ua' expires in 15 day(s) (Wed 05 Jun 2024 03:42:41 PM GMT +0000). [15:56:41] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3868c54a3f27...d79e30726a29 [15:56:44] [02ssl] 07WikiTideSSLBot 03d79e307 - Bot: Update SSL cert for en.religiononfire.mar.in.ua [15:56:55] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d79e30726a29...42703f0b1b4b [15:56:56] [02ssl] 07WikiTideSSLBot 0342703f0 - Bot: Update SSL cert for uk.religiononfire.mar.in.ua [15:58:59] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw182 [15:59:31] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [16:00:59] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [16:09:34] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw182 [16:10:26] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw181 [16:10:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:11:34] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [16:12:23] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [16:12:45] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw182 [16:14:41] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [16:14:50] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/42703f0b1b4b...fc45962fb50e [16:14:53] [02ssl] 07WikiTideSSLBot 03fc45962 - Bot: Add SSL cert for wiki.neonai.dev [16:15:34] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw182 [16:18:17] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw182 [16:18:50] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw151 [16:18:58] [02dns] 07MacFan4000 opened pull request 03#519: Add encyclopbd.info zone - 13https://github.com/miraheze/dns/pull/519 [16:19:10] [02dns] 07coderabbitai[bot] commented on pull request 03#519: Add encyclopbd.info zone - 13https://github.com/miraheze/dns/pull/519#issuecomment-2120777039 [16:19:34] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [16:20:01] [02dns] 07coderabbitai[bot] edited pull request 03#519: Add encyclopbd.info zone - 13https://github.com/miraheze/dns/pull/519 [16:20:05] [02dns] 07coderabbitai[bot] edited a comment on pull request 03#519: Add encyclopbd.info zone - 13https://github.com/miraheze/dns/pull/519#issuecomment-2120777039 [16:20:47] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [16:22:12] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [16:23:23] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw182 [16:25:27] RECOVERY - en.religiononfire.mar.in.ua - LetsEncrypt on sslhost is OK: OK - Certificate 'en.religiononfire.mar.in.ua' will expire on Sun 18 Aug 2024 02:56:34 PM GMT +0000. [16:25:34] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 3 backends are down. mw152 mw181 mw182 [16:25:48] RECOVERY - uk.religiononfire.mar.in.ua - LetsEncrypt on sslhost is OK: OK - Certificate 'uk.religiononfire.mar.in.ua' will expire on Sun 18 Aug 2024 02:56:49 PM GMT +0000. [16:26:27] MacFan4000: FYI I believe I disabled coderabbit entirely for dns and ssl repos. If it happens let me know and I'll see why. There is absolutely no reason for it there... [16:27:13] [02dns] 07Universal-Omega closed pull request 03#519: Add encyclopbd.info zone - 13https://github.com/miraheze/dns/pull/519 [16:27:16] [02dns] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/ee03367cd368...43707f6dc8fe [16:27:16] [02dns] 07MacFan4000 0343707f6 - Add encyclopbd.info zone (#519) [16:27:34] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [16:28:07] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw182 [16:29:20] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [16:32:09] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw152 mw182 [16:33:59] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [16:35:20] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 3 backends are down. mw151 mw181 mw182 [16:36:33] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 2 backends are down. mw181 mw182 [16:36:35] PROBLEM - www.cgradegames.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.cgradegames.net could not be found [16:37:21] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [16:38:32] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [16:38:56] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [16:40:10] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [16:40:45] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw182 [16:41:20] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [16:46:45] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [16:48:15] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw181 [16:48:48] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [16:49:02] PROBLEM - www.cgradegames.net - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [16:50:11] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [16:50:39] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw182 [16:52:36] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [16:52:38] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw181 [16:54:00] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw181 mw182 [16:54:35] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [16:55:59] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [16:59:24] !log reboot mw182 [16:59:39] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 5.62, 1.23, 0.40 [16:59:48] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw181 [16:59:54] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw181 [17:00:06] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 0.24, 0.05, 0.02 [17:01:14] !log reboot mw181 (sorry opposite order) [17:01:48] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw182 [17:02:15] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw182 [17:02:46] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw181 [17:03:08] !log reboot mw172 [17:03:39] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.68, 17.03, 7.10 [17:04:12] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [17:04:40] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [17:04:53] !log reboot mw171 [17:05:57] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.32, 17.24, 7.25 [17:06:26] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:06:28] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [17:07:17] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query. [17:07:30] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [17:07:32] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [17:07:48] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [17:08:23] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw182 [17:08:31] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw182 [17:08:36] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 8.336 second response time [17:08:49] Log failed it seems [17:10:20] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [17:10:26] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [17:11:43] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw182 [17:13:42] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [17:15:32] RECOVERY - cloud18 IPMI Sensors on cloud18 is OK: IPMI Status: OK [17:17:15] RECOVERY - cloud16 IPMI Sensors on cloud16 is OK: IPMI Status: OK [17:20:04] RECOVERY - cloud17 IPMI Sensors on cloud17 is OK: IPMI Status: OK [17:20:25] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/fc45962fb50e...55d47f0abf34 [17:20:26] [02ssl] 07WikiTideSSLBot 0355d47f0 - Bot: Add SSL cert for wiki.encyclopbd.info [17:22:16]