[00:01:32] PROBLEM - cp31 Disk Space on cp31 is CRITICAL: DISK CRITICAL - free space: / 2247 MB (5% inode=96%); [00:02:41] PROBLEM - cp30 Disk Space on cp30 is CRITICAL: DISK CRITICAL - free space: / 2278 MB (5% inode=96%); [00:03:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.36, 3.74, 3.70 [00:07:32] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 9251 MB (24% inode=96%); [00:07:42] PROBLEM - roblox-wiki.tk - LetsEncrypt on sslhost is CRITICAL: connect to address roblox-wiki.tk and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:07:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.97, 2.87, 3.35 [00:07:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.43, 2.44, 3.20 [00:08:41] RECOVERY - cp30 Disk Space on cp30 is OK: DISK OK - free space: / 10647 MB (27% inode=96%); [00:13:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.22, 8.72, 9.97 [00:27:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.42, 5.58, 4.31 [00:29:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.12, 3.59, 3.29 [00:31:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.98, 10.26, 11.79 [00:35:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.09, 10.89, 11.63 [00:35:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.64, 3.80, 3.98 [00:37:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.77, 10.52, 11.41 [00:37:07] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.66, 2.80, 3.12 [00:37:17] RECOVERY - roblox-wiki.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'roblox-wiki.tk' will expire on Thu 27 Oct 2022 15:10:45 GMT +0000. [00:39:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.64, 11.60, 11.66 [00:39:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.62, 3.87, 3.93 [00:41:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.73, 3.46 [00:41:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.42, 11.29, 11.55 [00:41:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.05, 3.22, 3.68 [00:44:52] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.59, 3.24, 3.32 [00:47:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.48, 11.41, 11.36 [00:47:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.85, 2.71, 3.36 [00:49:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.89, 11.12, 11.25 [00:57:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.13, 8.16, 9.78 [01:05:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 19.62, 14.14, 11.68 [01:05:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.83, 11.35, 9.81 [01:13:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.89, 11.59, 11.43 [01:13:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.10, 11.16, 10.42 [01:13:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.75, 3.94, 3.20 [01:15:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.04, 12.60, 11.79 [01:15:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.53, 5.05, 3.66 [01:19:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.93, 9.68, 10.04 [01:20:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.64, 4.14, 3.44 [01:22:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.41, 3.76, 3.40 [01:22:37] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/be790a6a6e36 [01:22:38] [02miraheze/mw-config] 07paladox 03be790a6 - Replace db111 with db141 [01:22:40] [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/mw-config [01:22:40] [url] Page not found · GitHub · GitHub | github.com [01:22:41] [02mw-config] 07paladox opened pull request 03#4869: Replace db111 with db141 - 13https://github.com/miraheze/mw-config/pull/4869 [01:22:42] [url] Page not found · GitHub · GitHub | github.com [01:23:23] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/be790a6a6e36...6f66c690f1b5 [01:23:24] [url] Comparing be790a6a6e36...6f66c690f1b5 · miraheze/mw-config · GitHub | github.com [01:23:25] [02miraheze/mw-config] 07paladox 036f66c69 - Update LocalSettings.php [01:23:26] [02mw-config] 07paladox synchronize pull request 03#4869: Replace db111 with db141 - 13https://github.com/miraheze/mw-config/pull/4869 [01:23:27] [url] Page not found · GitHub · GitHub | github.com [01:23:35] [02mw-config] 07paladox commented on pull request 03#4868: Replace db111 with db141 - 13https://github.com/miraheze/mw-config/pull/4868#issuecomment-1211460781 [01:23:35] [url] Replace db111 with db141 by paladox · Pull Request #4868 · miraheze/mw-config · GitHub | github.com [01:23:36] [02mw-config] 07paladox closed pull request 03#4868: Replace db111 with db141 - 13https://github.com/miraheze/mw-config/pull/4868 [01:23:37] [url] Page not found · GitHub · GitHub | github.com [01:23:38] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/mw-config [01:23:38] [url] Page not found · GitHub · GitHub | github.com [01:23:39] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [01:23:55] miraheze/mw-config - paladox the build passed. [01:23:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.23, 3.48, 3.55 [01:24:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.66, 4.00, 3.52 [01:24:36] miraheze/mw-config - paladox the build passed. [01:25:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.73, 4.38, 3.86 [01:26:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.27, 3.97, 3.59 [01:27:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.40, 1.75, 2.00 [01:28:23] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.08, 4.06, 3.69 [01:29:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.37, 1.99, 2.06 [01:30:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.57, 3.67, 3.60 [01:31:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.13, 1.66, 1.93 [01:33:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.05, 1.78, 1.93 [01:34:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.50, 3.72, 3.60 [01:36:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.76, 3.91, 3.70 [01:37:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.66, 10.52, 10.09 [01:38:02] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.38, 4.39, 3.90 [01:38:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.14, 1.73, 1.99 [01:39:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.26, 1.87, 1.97 [01:39:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.06, 11.49, 10.48 [01:40:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.07, 1.86, 2.00 [01:41:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.62, 11.64, 10.67 [01:43:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.03, 12.65, 11.22 [01:45:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.00, 2.15, 2.01 [01:45:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.23, 3.45, 3.73 [01:45:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.80, 11.43, 10.93 [01:45:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.61, 3.19, 3.81 [01:47:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.86, 3.86, 3.84 [01:47:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.80, 4.95, 4.36 [01:49:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.35, 2.88, 3.48 [01:51:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.23, 3.84, 4.00 [01:53:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.57, 9.25, 10.12 [01:59:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 0.93, 2.45, 3.20 [01:59:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.18, 2.83, 3.37 [02:00:02] !log set db111 and db141 into read only [02:00:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:00:30] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/ae3bebca4cce...e0d446b78b4a [02:00:31] [url] Comparing ae3bebca4cce...e0d446b78b4a · miraheze/mw-config · GitHub | github.com [02:00:31] [02miraheze/mw-config] 07paladox 03e0d446b - Replace db111 with db141 (#4869) [02:00:33] [02mw-config] 07paladox closed pull request 03#4869: Replace db111 with db141 - 13https://github.com/miraheze/mw-config/pull/4869 [02:00:33] ... [02:00:34] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3 [02:00:36] [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/mw-config [02:00:36] [url] Page not found · GitHub · GitHub | github.com [02:00:48] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [02:00:52] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to test131 [02:00:53] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to test131 - SUCCESS in 1s [02:00:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:00:57] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [02:01:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:01:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:01:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:01:38] miraheze/mw-config - paladox the build passed. [02:01:55] !log remove read only from db141 [02:02:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:04:54] PROBLEM - db111 MariaDB on db111 is CRITICAL: Can't connect to MySQL server on 'db111.miraheze.org' (115) [02:04:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.40, 1.79, 1.97 [02:06:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.61, 1.93, 1.99 [02:07:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.57, 1.82, 1.99 [02:07:31] !log stopped mariadb on db111 [02:07:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:09:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.12, 1.89, 2.00 [02:11:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.14, 1.64, 1.90 [02:11:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.22, 11.00, 10.28 [02:13:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.13, 1.95, 1.98 [02:13:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.15, 9.93, 9.98 [02:15:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.71, 1.85, 1.94 [02:19:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.12, 9.85, 11.77 [02:21:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.49, 2.17, 2.04 [02:24:08] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/bd7aba33f5da...978e42f17013 [02:24:09] [url] Comparing bd7aba33f5da...978e42f17013 · miraheze/dns · GitHub | github.com [02:24:09] [02miraheze/dns] 07paladox 03978e42f - Revert "Depool cp20/21 (#328)" [02:24:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.62, 5.03, 3.45 [02:25:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.35, 10.87, 11.45 [02:25:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.21, 3.78, 2.96 [02:27:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.92, 3.66, 3.01 [02:29:47] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.50, 4.15, 3.28 [02:29:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.44, 11.01, 10.27 [02:31:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.59, 1.51, 1.92 [02:31:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.88, 3.92, 3.30 [02:33:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.26, 4.09, 3.43 [02:33:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.69, 12.81, 11.18 [02:35:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.18, 3.59, 3.32 [02:35:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.19, 12.00, 11.07 [02:37:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.18, 12.67, 11.47 [02:41:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 4.02, 3.61 [02:43:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.74, 3.92, 3.62 [02:45:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.95, 1.25, 1.60 [02:45:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.04, 4.09, 3.69 [02:47:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.15, 3.71, 3.60 [02:49:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.94, 11.28, 11.83 [02:51:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.17, 1.74, 1.71 [02:51:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.64, 3.09, 3.36 [02:53:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.65, 1.65, 1.67 [02:53:35] !log shutdown db111 [02:53:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:54:42] !log shutdown gluster111 [02:54:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:56:51] PROBLEM - gluster111 SSH on gluster111 is CRITICAL: connect to address 2a10:6740::6:203 and port 22: No route to host [02:57:23] PROBLEM - db111 NTP time on db111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: No route to hostconnect to host 2a10:6740::6:201 port 5666: No route to host [02:57:29] PROBLEM - db111 conntrack_table_size on db111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: No route to hostconnect to host 2a10:6740::6:201 port 5666: No route to host [02:57:29] PROBLEM - db111 Disk Space on db111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: No route to hostconnect to host 2a10:6740::6:201 port 5666: No route to host [02:57:29] PROBLEM - db111 PowerDNS Recursor on db111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: No route to hostconnect to host 2a10:6740::6:201 port 5666: No route to host [02:57:33] PROBLEM - gluster111 PowerDNS Recursor on gluster111 is CRITICAL: connect to address 2a10:6740::6:203 port 5666: No route to hostconnect to host 2a10:6740::6:203 port 5666: No route to host [02:57:33] PROBLEM - gluster111 Disk Space on gluster111 is CRITICAL: connect to address 2a10:6740::6:203 port 5666: No route to hostconnect to host 2a10:6740::6:203 port 5666: No route to host [02:57:39] PROBLEM - db111 Current Load on db111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: No route to hostconnect to host 2a10:6740::6:201 port 5666: No route to host [02:57:43] PROBLEM - gluster111 ferm_active on gluster111 is CRITICAL: connect to address 2a10:6740::6:203 port 5666: No route to hostconnect to host 2a10:6740::6:203 port 5666: No route to host [02:57:45] PROBLEM - db111 Puppet on db111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: No route to hostconnect to host 2a10:6740::6:201 port 5666: No route to host [02:57:49] PROBLEM - gluster111 Puppet on gluster111 is CRITICAL: connect to address 2a10:6740::6:203 port 5666: No route to hostconnect to host 2a10:6740::6:203 port 5666: No route to host [02:57:51] PROBLEM - db111 ferm_active on db111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: No route to hostconnect to host 2a10:6740::6:201 port 5666: No route to host [02:57:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.45, 2.68, 3.89 [02:57:59] PROBLEM - gluster111 NTP time on gluster111 is CRITICAL: connect to address 2a10:6740::6:203 port 5666: No route to hostconnect to host 2a10:6740::6:203 port 5666: No route to host [02:58:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: connect to address 2a10:6740::6:203 port 5666: No route to hostconnect to host 2a10:6740::6:203 port 5666: No route to host [02:58:10] PROBLEM - gluster111 conntrack_table_size on gluster111 is CRITICAL: connect to address 2a10:6740::6:203 port 5666: No route to hostconnect to host 2a10:6740::6:203 port 5666: No route to host [02:58:21] PROBLEM - ping6 on db111 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:201) [02:58:25] PROBLEM - Host gluster111 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:203) [02:58:29] PROBLEM - Host db111 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:201) [02:58:29] PROBLEM - db111 SSH on db111 is CRITICAL: connect to address 2a10:6740::6:201 and port 22: No route to host [02:59:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.21, 10.28, 10.89 [02:59:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.15, 3.91 [03:00:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.10, 1.50, 1.90 [03:01:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.73, 2.65, 3.63 [03:03:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.43, 10.62, 10.95 [03:05:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.53, 11.12, 11.08 [03:07:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.36, 2.61, 3.29 [03:11:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.85, 12.00, 11.48 [03:11:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.88, 3.68 [03:12:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.96, 1.22, 1.60 [03:15:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.20, 2.87, 3.30 [03:22:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.59, 1.83, 1.65 [03:24:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.88, 1.42, 1.52 [03:31:16] !log depool and repool mw141 [03:31:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:35:46] !log enable numa on graylog121 [03:35:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:35:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.86, 11.00, 10.71 [03:36:31] !log enable numa on jobchron121 [03:36:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:37:19] !log enable numa on mail121 [03:37:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:38:24] !log set phab121 sockets to 2 and vcpu to 1 manually [03:38:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:38:53] !log enable numa on bast101/bast121 [03:38:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:39:56] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 2.09, 0.95, 0.35 [03:41:56] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 1.14, 0.96, 0.43 [03:43:50] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[git_pull_JobRunner],Exec[git_pull_mathoid],Exec[git_pull_3d2png],Exec[git_pull_femiwiki-deploy] [03:43:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.98, 11.66, 11.45 [03:44:25] PROBLEM - bast101 NTP time on bast101 is CRITICAL: NTP CRITICAL: Offset 0.6309202611 secs [03:46:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 11.38, 6.59, 4.10 [03:48:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.95, 3.47, 2.84 [03:50:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.46, 3.07, 2.76 [03:50:07] paladox: I think numa might be what is causing NTP time to flap? It started on other servers when tbey were done before, and now somewhat on bast101 also. Could it actually cause it though? [03:50:45] not as far as i'm aware. [03:51:48] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:51:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.84, 11.12, 11.05 [03:52:33] paladox: I have never seen it on bast until now though, and it very rarely happened on other servers but after numa it seemed to be more common. Is there maybe just a way to update the check to make it less flappy maybe? Either way, not a huge issue, I don't think. [03:53:20] https://icinga.miraheze.org/dashboard#!/monitoring/service/history?host=bast101&service=bast101%20NTP%20time [03:53:20] [url] Icinga Web 2 Login | icinga.miraheze.org [03:53:28] seems to have it as happening before the NUMA change [03:53:33] thus it isn't related to NUMA [03:53:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.85, 1.58, 1.44 [03:53:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.58, 10.18, 10.73 [03:54:18] paladox: Well it seemed to be more common after numa, either way I am not concerned with it. It is possible I only noticed after numa and just assumed also. [03:54:34] yeh [03:54:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.83, 3.84, 3.72 [03:55:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.78, 1.24, 1.33 [03:57:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.99, 10.57, 11.91 [04:02:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.58, 4.05, 3.82 [04:04:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.09, 3.66, 3.70 [04:05:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 21.11, 13.48, 12.32 [04:05:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.10, 10.79, 10.55 [04:07:35] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.81, 1.56, 1.35 [04:07:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.37, 10.63, 10.53 [04:08:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.52, 2.49, 3.22 [04:09:33] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.30, 1.54, 1.37 [04:12:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.26, 3.93, 3.67 [04:13:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.45, 9.07, 9.93 [04:14:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.80, 3.66 [04:15:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.87, 10.01, 11.69 [04:16:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.17, 4.11, 3.77 [04:16:25] RECOVERY - bast101 NTP time on bast101 is OK: NTP OK: Offset 0.003490358591 secs [04:21:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 23.49, 12.03, 11.52 [04:22:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.41, 3.74, 3.78 [04:25:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.61, 3.71, 3.72 [04:27:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.57, 2.10, 1.64 [04:27:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.89, 3.11, 2.69 [04:29:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.76, 1.92, 1.63 [04:29:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.97, 2.95, 2.68 [04:31:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.76, 1.50, 1.51 [04:31:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.61, 10.40, 9.78 [04:32:39] !log install numactl on cloud13 & 14 [04:32:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:33:34] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.03, 10.28, 9.81 [04:37:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 11.25, 5.21, 2.82 [04:37:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.20, 8.70, 9.32 [04:37:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.10, 3.65, 3.97 [04:41:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.56, 10.81, 11.89 [04:45:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.71, 3.55, 3.66 [04:47:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.76, 3.11, 3.49 [04:49:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.42, 1.66, 1.97 [04:49:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.54, 4.83, 4.09 [04:53:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.46, 3.84, 3.91 [04:55:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/64d7116a1073 [04:55:07] [02miraheze/puppet] 07paladox 0364d7116 - base: Add numa facter [04:55:09] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [04:55:09] [url] Page not found · GitHub · GitHub | github.com [04:55:10] [02puppet] 07paladox opened pull request 03#2776: base: Add numa facter - 13https://github.com/miraheze/puppet/pull/2776 [04:55:11] [url] Page not found · GitHub · GitHub | github.com [04:55:22] [02puppet] 07paladox closed pull request 03#2776: base: Add numa facter - 13https://github.com/miraheze/puppet/pull/2776 [04:55:23] [url] Page not found · GitHub · GitHub | github.com [04:55:24] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [04:55:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/46c9f52fc157...0ca9747fd6b7 [04:55:26] [url] Comparing 46c9f52fc157...0ca9747fd6b7 · miraheze/puppet · GitHub | github.com [04:55:27] [02miraheze/puppet] 07paladox 030ca9747 - base: Add numa facter (#2776) [04:55:28] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [04:55:29] [url] Page not found · GitHub · GitHub | github.com [04:57:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.40, 1.82, 1.86 [04:59:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.47, 9.21, 10.10 [05:01:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.34, 1.79, 1.85 [05:03:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.17, 12.08, 10.96 [05:05:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.84, 1.25, 1.62 [05:05:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.40, 2.61, 3.35 [05:07:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.91, 10.68, 10.67 [05:11:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.86, 9.36, 10.18 [05:22:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.88, 3.29, 2.97 [05:25:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 18.10, 12.40, 10.85 [05:26:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.16, 2.88, 2.89 [05:27:37] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.248 second response time [05:27:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.06, 9.38, 8.62 [05:27:57] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.235 second response time [05:28:00] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [05:28:25] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 9.66, 11.23, 7.65 [05:28:51] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.007 second response time [05:29:16] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.014 second response time [05:30:25] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 4.69, 8.80, 7.19 [05:30:33] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw142 [05:30:52] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw142 [05:31:39] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw142 [05:31:51] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw142 [05:31:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 10.16, 9.96, 9.05 [05:35:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.97, 10.60, 9.53 [05:37:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.49, 11.46, 10.00 [05:41:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.43, 11.38, 10.37 [05:47:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.37, 10.11, 10.16 [05:48:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.28, 2.82 [05:50:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.08, 3.18, 2.84 [05:51:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.08, 11.78, 10.80 [05:55:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.12, 10.11, 10.38 [05:57:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.86, 9.18, 10.01 [06:09:42] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.74, 1.34, 1.09 [06:11:38] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.34, 1.43, 1.15 [06:12:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.62, 2.88 [06:13:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.58, 11.03, 10.36 [06:14:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.77, 2.99, 2.74 [06:15:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.42, 10.16, 11.80 [06:17:31] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.85, 10.05, 10.13 [06:19:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.79, 11.79, 12.09 [06:21:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.64, 10.49, 11.58 [06:28:11] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.72, 1.67, 1.26 [06:30:11] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.39, 1.56, 1.27 [06:31:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.52, 11.23, 11.36 [06:32:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.20, 10.44, 10.04 [06:33:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.04, 9.93, 9.91 [06:34:11] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.92, 1.74, 1.40 [06:37:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.93, 10.73, 10.27 [06:39:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.51, 11.65, 10.65 [06:41:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.83, 10.55, 10.37 [06:42:11] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.07, 1.53, 1.47 [06:43:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.20, 9.43, 9.98 [07:03:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.98, 9.78, 11.40 [07:05:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.52, 12.08, 12.05 [07:07:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.52, 10.59, 11.51 [07:15:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.04, 10.82, 10.93 [07:17:35] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.53, 2.50, 1.63 [07:19:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.75, 11.70, 11.29 [07:19:33] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.28, 1.98, 1.54 [07:23:29] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.51, 1.39, 1.41 [07:27:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.34, 8.91, 10.10 [07:35:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.63, 11.03, 10.52 [07:37:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 28.91, 15.69, 11.09 [07:39:16] PROBLEM - cloud12 Current Load on cloud12 is CRITICAL: CRITICAL - load average: 106.15, 57.21, 31.98 [07:39:52] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 7.39, 10.67, 7.82 [07:41:52] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 5.21, 8.67, 7.42 [07:45:16] PROBLEM - cloud12 Current Load on cloud12 is WARNING: WARNING - load average: 17.77, 44.09, 36.51 [07:47:16] RECOVERY - cloud12 Current Load on cloud12 is OK: OK - load average: 16.52, 34.88, 34.06 [07:53:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 2.73, 2.09 [07:55:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.91, 2.55, 2.09 [08:01:06] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [08:13:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.42, 11.27, 11.99 [08:26:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [08:29:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.03, 8.54, 10.04 [08:31:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.53, 10.31, 11.63 [08:33:13] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.02, 2.22, 1.36 [08:35:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.37, 12.76, 12.26 [08:35:08] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.52, 1.58, 1.23 [08:39:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.23, 11.68, 11.96 [08:40:53] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:47:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.76, 11.20, 11.21 [08:53:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.15, 11.82, 10.20 [08:57:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.44, 11.54, 10.53 [09:00:12] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK [09:01:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.76, 11.71, 10.81 [09:05:18] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.56, 10.98, 10.79 [09:11:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.93, 11.27, 10.82 [09:12:57] PROBLEM - revi.wiki - PositiveSSLDV on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:13:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.79, 9.52, 11.79 [09:13:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.02, 9.74, 10.33 [09:13:43] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [09:15:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.62, 8.08, 9.66 [09:17:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 4.95, 6.87, 10.17 [09:17:42] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:41:28] RECOVERY - revi.wiki - PositiveSSLDV on sslhost is OK: OK - Certificate 'revi.wiki' will expire on Thu 29 Dec 2022 23:59:59 GMT +0000. [09:46:03] [02puppet] 07RhinosF1 opened pull request 03#2777: check_rdns: update cf_at_root logic - 13https://github.com/miraheze/puppet/pull/2777 [09:46:03] [url] Page not found · GitHub · GitHub | github.com [09:46:20] [02puppet] 07RhinosF1 commented on pull request 03#2777: check_rdns: update cf_at_root logic - 13https://github.com/miraheze/puppet/pull/2777#issuecomment-1211764443 [09:46:20] [url] check_rdns: update cf_at_root logic by RhinosF1 · Pull Request #2777 · miraheze/puppet · GitHub | github.com [11:10:26] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/c66926ac2048...bd19f13a4182 [11:10:27] [url] Comparing c66926ac2048...bd19f13a4182 · miraheze/MirahezeMagic · GitHub | github.com [11:10:28] [02miraheze/MirahezeMagic] 07translatewiki 03bd19f13 - Localisation updates from https://translatewiki.net. [11:10:28] [url] Main page - translatewiki.net | translatewiki.net [11:10:29] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/landing/compare/8e463ebbc863...d06a5449c19e [11:10:30] [url] Comparing 8e463ebbc863...d06a5449c19e · miraheze/landing · GitHub | github.com [11:10:31] [02miraheze/landing] 07translatewiki 03d06a544 - Localisation updates from https://translatewiki.net. [11:10:31] [url] Main page - translatewiki.net | translatewiki.net [11:11:22] miraheze/landing - translatewiki the build passed. [11:19:58] miraheze/MirahezeMagic - translatewiki the build passed. [11:38:51] PROBLEM - landofliberos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:39:01] PROBLEM - www.fossilfighterswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:41:26] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:41:29] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [11:41:45] PROBLEM - yokaiwatchwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:43:20] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [11:43:29] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:43:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.23, 3.03, 2.07 [11:44:25] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[Landing Sync] [11:45:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.47, 11.24, 8.81 [11:45:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.05, 2.78, 2.09 [11:53:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.35, 12.11, 9.96 [11:57:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.31, 11.42, 10.21 [12:07:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.56, 10.06, 10.03 [12:07:47] RECOVERY - landofliberos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'landofliberos.com' will expire on Tue 30 Aug 2022 16:38:50 GMT +0000. [12:08:05] RECOVERY - www.fossilfighterswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.fossilfighterswiki.com' will expire on Sat 24 Sep 2022 13:17:51 GMT +0000. [12:09:43] PROBLEM - fortis.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fortis.wiki' expires in 11 day(s) (Mon 22 Aug 2022 15:42:16 GMT +0000). [12:10:27] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [12:10:34] RECOVERY - yokaiwatchwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'yokaiwatchwiki.com' will expire on Tue 01 Nov 2022 02:48:17 GMT +0000. [12:19:56]