[00:01:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.79, 3.79, 3.75 [00:02:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.12, 3.83, 3.85 [00:02:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.93, 10.25, 9.79 [00:02:53] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.52, 1.53, 1.03 [00:04:27] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 10472 MB (26% inode=96%); [00:04:38] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.28, 9.72, 9.64 [00:04:50] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.26, 1.44, 1.06 [00:05:04] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 10342 MB (26% inode=96%); [00:05:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.36, 4.58, 3.74 [00:09:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.04, 3.81, 3.61 [00:10:59] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.81, 3.13, 3.39 [00:11:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.26, 3.77, 3.98 [00:13:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.85, 4.30, 4.13 [00:14:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.58, 3.72, 3.57 [00:18:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.48, 3.78, 3.63 [00:20:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.68, 10.56, 9.86 [00:22:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.64, 11.15, 10.15 [00:23:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.11, 1.68, 1.32 [00:24:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.67, 11.35, 10.35 [00:25:06] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.95, 1.39, 1.26 [00:28:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.14, 4.03, 3.75 [00:28:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.43, 11.38, 10.53 [00:30:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.71, 3.59, 3.62 [00:30:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.76, 10.46, 10.29 [00:32:29] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.40, 9.03, 9.79 [00:36:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.05, 10.58, 10.23 [00:37:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.90, 4.16, 3.77 [00:38:27] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.37, 10.16, 10.12 [00:41:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.25, 3.71, 3.74 [00:43:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.51, 4.47, 4.01 [00:45:30] !log [@test101] finished deploy of {'l10nupdate': True} to all - SUCCESS in 6321s [00:45:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:47:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.44, 3.49, 3.72 [00:49:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.48, 2.85, 3.77 [00:53:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.51, 3.51, 3.59 [00:55:40] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.80, 1.77, 1.42 [00:55:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.52, 4.32, 4.05 [00:57:40] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.55, 1.61, 1.40 [00:59:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.45, 3.89, 3.86 [01:03:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.41, 3.86, 3.84 [01:08:25] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.27, 4.68, 3.28 [01:09:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.25, 3.47, 3.75 [01:09:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.67, 3.70, 3.97 [01:10:23] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.46, 3.79, 3.12 [01:11:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.23, 4.12, 4.08 [01:12:21] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 1.85, 3.21, 2.99 [01:15:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.97, 3.93, 4.00 [01:19:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.19, 3.47 [01:21:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.15, 3.09, 3.39 [01:21:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.29, 2.66, 3.37 [01:27:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.52, 3.65, 3.53 [01:27:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 3.91, 3.70 [01:29:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.08, 3.33, 3.52 [01:31:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.32, 3.06, 3.33 [01:34:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.55, 3.97 [01:36:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.66, 4.00, 4.09 [01:37:45] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.46, 3.00, 3.35 [01:41:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.84, 3.65, 3.55 [01:44:13] [02ssl] 07MacFan4000 closed pull request 03#525: Add studyabroad.wiki cert - 13https://github.com/miraheze/ssl/pull/525 [01:44:13] [url] Page not found · GitHub · GitHub | github.com [01:44:14] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/4dcfe7efa13e...ab7e1e5b28ea [01:44:15] [url] Comparing 4dcfe7efa13e...ab7e1e5b28ea · miraheze/ssl · GitHub | github.com [01:44:16] [02miraheze/ssl] 07Universal-Omega 03ab7e1e5 - Add studyabroad.wiki cert (#525) [01:44:19] [02ssl] 07MacFan4000 closed pull request 03#524: Remove no longer pointing domains - 13https://github.com/miraheze/ssl/pull/524 [01:44:20] [url] Page not found · GitHub · GitHub | github.com [01:44:21] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-3/±2] 13https://github.com/miraheze/ssl/compare/ab7e1e5b28ea...6af968499712 [01:44:21] [url] Comparing ab7e1e5b28ea...6af968499712 · miraheze/ssl · GitHub | github.com [01:44:22] [02miraheze/ssl] 07Universal-Omega 036af9684 - Remove no longer pointing domains (#524) [01:46:22] Thanks MacFan4000! [01:51:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.33, 3.93, 3.64 [01:51:47] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.80, 3.16, 2.82 [01:51:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.39, 1.74, 1.29 [01:52:43] PROBLEM - mw112 Puppet on mw112 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 3 minutes ago with 5 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wiki.mastodon.kr.key],File[/etc/ssl/localcerts/pi.miraheze.org.crt],File[/etc/ssl/private/pi.miraheze.org.key],File[/etc/ssl/localcerts/www.recyclerie-uzege.org.crt] [01:53:45] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.81, 3.03, 2.82 [01:53:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.78, 3.85, 3.65 [01:53:48] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.23, 1.45, 1.23 [01:54:46] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/wiki.mastodon.kr.crt],File[/etc/ssl/private/wiki.mastodon.kr.key],File[/etc/ssl/localcerts/pi.miraheze.org.crt],File[/etc/ssl/private/pi.miraheze.org.key] [02:01:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.32, 3.50, 3.92 [02:03:33] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.24, 3.15, 1.64 [02:04:54] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.71, 2.96, 3.34 [02:07:29] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.45, 1.64, 1.35 [02:13:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.88, 4.17, 3.81 [02:17:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.82, 3.76 [02:19:25] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 4.29, 2.64, 1.72 [02:21:25] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [02:22:21] RECOVERY - mw112 Puppet on mw112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:23:22] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.45, 1.58, 1.50 [02:27:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 3.52, 3.22 [02:29:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.97, 3.35, 3.03 [02:31:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.63, 3.16, 3.14 [02:31:16] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.83, 3.17, 3.01 [02:33:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.51, 3.49 [02:35:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.46, 3.15 [02:35:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.72, 3.58 [02:37:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.34, 3.38, 3.15 [02:37:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 3.87, 3.65 [02:39:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.56, 4.18, 3.57 [02:40:07] [02dns] 07Universal-Omega opened pull request 03#276: Add miraheze.cf zone - 13https://github.com/miraheze/dns/pull/276 [02:40:08] [url] Page not found · GitHub · GitHub | github.com [02:41:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.08, 3.62, 3.62 [02:41:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.65, 3.72, 3.48 [02:43:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.88, 4.25, 3.86 [02:45:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.36, 3.82, 3.57 [02:51:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.95, 3.96, 3.54 [02:53:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.87, 4.17, 3.66 [02:55:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.75, 3.83, 3.90 [02:55:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 4.00, 3.80, 3.56 [02:57:10] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.84, 4.25, 4.05 [02:58:09] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [02:59:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.95, 3.66, 3.85 [03:03:36] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.98, 3.22, 3.38 [03:06:09] [02dns] 07Universal-Omega edited pull request 03#274: Remove petrawiki.org zone - 13https://github.com/miraheze/dns/pull/274 [03:06:10] [url] Page not found · GitHub · GitHub | github.com [03:08:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.07, 2.87, 3.32 [03:08:48] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.14, 1.81, 1.41 [03:10:47] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.40, 1.68, 1.41 [03:11:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.84, 3.45, 3.96 [03:21:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.38, 3.69, 3.39 [03:23:16] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.71, 3.19, 3.24 [03:25:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 3.77, 3.83 [03:26:45] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [03:27:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.64, 3.92, 3.90 [03:33:40] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.18, 2.61, 3.30 [03:39:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.81, 3.42, 3.38 [03:41:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 4.00, 3.50, 3.18 [03:43:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.49, 3.44 [03:45:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.33, 3.14, 3.12 [03:51:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.83, 4.50, 3.78 [03:53:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.90, 3.59, 3.29 [03:57:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.74, 1.73, 1.42 [03:57:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.93, 3.86, 3.75 [03:58:58] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.81, 3.11, 3.19 [03:59:07] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.51, 1.53, 1.38 [03:59:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.43, 4.93, 4.11 [04:01:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.96, 3.88, 3.37 [04:02:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.64, 3.97, 3.64 [04:05:25] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.55, 3.28, 3.25 [04:06:42] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.25, 2.93, 3.27 [04:11:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.31, 3.62, 3.40 [04:12:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.89, 3.73, 3.52 [04:13:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.31, 3.74, 3.46 [04:14:32] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.45, 3.32, 3.40 [04:15:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.38, 3.34 [04:17:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.34, 2.94, 3.19 [04:19:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.90, 3.61, 3.94 [04:23:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.15, 3.49, 3.77 [04:25:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.88, 3.52, 3.74 [04:33:40] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.00, 3.00, 3.36 [04:48:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.39, 3.45, 3.31 [04:48:47] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 3.46, 3.02 [04:50:32] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.00, 3.20, 3.23 [04:50:45] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 3.01, 2.90 [04:53:03] [02dns] 07Reception123 closed pull request 03#274: Remove petrawiki.org zone - 13https://github.com/miraheze/dns/pull/274 [04:53:04] [url] Page not found · GitHub · GitHub | github.com [04:53:05] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/6ca96e2cd9ea...af9e3ad709a6 [04:53:05] [url] Comparing 6ca96e2cd9ea...af9e3ad709a6 · miraheze/dns · GitHub | github.com [04:53:06] [02miraheze/dns] 07Universal-Omega 03af9e3ad - Remove petrawiki.org zone (#274) [04:53:11] [02dns] 07Reception123 closed pull request 03#276: Add miraheze.cf zone - 13https://github.com/miraheze/dns/pull/276 [04:53:12] ... [04:53:13] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/af9e3ad709a6...f44e8cd7c67d [04:53:14] [url] Comparing af9e3ad709a6...f44e8cd7c67d · miraheze/dns · GitHub | github.com [04:53:14] [02miraheze/dns] 07Universal-Omega 03f44e8cd - Add miraheze.cf zone (#276) [04:54:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.04, 4.08, 3.57 [04:58:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.94, 3.59, 3.22 [05:00:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.78, 3.87, 3.66 [05:02:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.67, 4.47, 3.90 [05:02:32] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.06, 3.01, 3.09 [05:06:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.91, 3.80 [05:06:35] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.78, 1.44, 0.92 [05:08:33] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.79, 1.13, 0.87 [05:10:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.58, 3.74, 3.71 [05:12:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.07, 3.39, 3.58 [05:25:52] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.32, 3.12, 3.38 [05:35:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.87, 3.46, 3.37 [05:39:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.15, 3.48, 3.38 [05:41:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.57, 3.32, 3.32 [05:43:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.59, 3.90, 3.51 [05:46:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.87, 3.61, 3.22 [05:47:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.98, 3.67 [05:49:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.35, 4.52, 3.86 [05:50:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.42, 4.37, 3.58 [05:51:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.22, 3.51, 3.12 [05:54:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.85, 1.76, 1.24 [05:55:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.05, 3.82, 3.29 [05:56:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.48, 3.83, 3.61 [05:57:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.13, 3.90, 3.89 [05:58:05] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.60, 1.67, 1.32 [05:58:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.11, 4.48, 3.88 [05:59:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.41, 4.44, 4.11 [06:01:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.71, 3.61, 3.41 [06:01:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.49, 3.91, 3.97 [06:02:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.74, 3.25, 3.55 [06:03:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.07, 3.15, 3.27 [06:06:32] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.24, 2.95, 3.34 [06:11:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.32, 3.48, 3.64 [06:13:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.04, 3.19, 3.51 [06:19:40] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.18, 2.87, 3.31 [06:24:03] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.04, 3.15, 2.33 [06:26:03] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.90, 3.10, 2.42 [06:56:14] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.37, 1.52, 1.15 [06:58:12] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.82, 1.47, 1.17 [07:00:10] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.47, 1.55, 1.24 [07:04:05] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.95, 1.83, 1.43 [07:06:03] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.82, 1.47, 1.35 [07:08:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.64, 3.65, 3.08 [07:10:15] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.71, 3.02, 2.92 [07:13:19] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:13:42] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.10, 10.72, 7.76 [07:13:56] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:14:17] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 18.41, 12.95, 9.16 [07:15:16] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 2.361 second response time [07:15:42] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.06, 9.82, 7.82 [07:15:52] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.332 second response time [07:18:17] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.65, 10.29, 9.05 [07:18:45] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.59, 2.18, 1.20 [07:20:17] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.44, 9.01, 8.73 [07:20:44] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.15, 1.69, 1.14 [07:25:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.08, 1.56, 1.30 [07:26:07] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [07:27:46] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.67, 1.59, 1.34 [07:33:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.55, 3.17 [07:35:44] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.61, 2.88, 2.97 [07:47:44] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:52] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 22.19, 13.02, 8.50 [07:48:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.39, 10.51, 8.63 [07:49:48] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 7.189 second response time [07:50:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.22, 8.82, 8.24 [07:53:51] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.76, 9.91, 8.93 [07:54:43] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [07:57:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.47, 1.81, 1.61 [07:59:46] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.22, 1.55, 1.54 [07:59:56] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/53a3763be6e5 [07:59:57] [02miraheze/mediawiki] 07RhinosF1 0353a3763 - emails: fix JQ spam [07:59:59] [02mediawiki] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://github.com/miraheze/mediawiki [07:59:59] [url] Page not found · GitHub · GitHub | github.com [08:00:00] [02mediawiki] 07RhinosF1 opened pull request 03#7044: emails: fix JQ spam - 13https://github.com/miraheze/mediawiki/pull/7044 [08:00:01] [url] Page not found · GitHub · GitHub | github.com [08:04:39] [02mediawiki] 07Universal-Omega reviewed pull request 03#7044 commit - 13https://github.com/miraheze/mediawiki/pull/7044#discussion_r878814417 [08:04:39] [url] emails: fix JQ spam by RhinosF1 · Pull Request #7044 · miraheze/mediawiki · GitHub | github.com [08:05:11] [02mediawiki] 07RhinosF1 synchronize pull request 03#7044: emails: fix JQ spam - 13https://github.com/miraheze/mediawiki/pull/7044 [08:05:11] [url] Page not found · GitHub · GitHub | github.com [08:05:12] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/53a3763be6e5...3c69b8ac179f [08:05:13] [url] Comparing 53a3763be6e5...3c69b8ac179f · miraheze/mediawiki · GitHub | github.com [08:05:14] [02miraheze/mediawiki] 07RhinosF1 033c69b8a - Update includes/mail/EmailNotification.php [08:10:35] [02mediawiki] 07RhinosF1 commented on pull request 03#7044: emails: fix JQ spam - 13https://github.com/miraheze/mediawiki/pull/7044#issuecomment-1133841576 [08:10:35] [url] emails: fix JQ spam by RhinosF1 · Pull Request #7044 · miraheze/mediawiki · GitHub | github.com [08:11:10] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/manageJobs.php --type=enotifNotify --action=repush-abandoned (START) [08:11:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:11:26] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/manageJobs.php --type=enotifNotify --action=repush-abandoned (END - exit=2) [08:11:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:11:39] !log [rhinos@mwtask111] starting deploy of {'world': True} to all [08:11:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:12:18] RhinosF1: you didn't merge. [08:13:40] [02puppet] 07RhinosF1 opened pull request 03#2581: jobrunner: set maxjobs to 1 - 13https://github.com/miraheze/puppet/pull/2581 [08:13:40] [url] Page not found · GitHub · GitHub | github.com [08:14:08] [02mediawiki] 07RhinosF1 closed pull request 03#7044: emails: fix JQ spam - 13https://github.com/miraheze/mediawiki/pull/7044 [08:14:08] [url] Page not found · GitHub · GitHub | github.com [08:14:08] !log [rhinos@mwtask111] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [08:14:09] [02mediawiki] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://github.com/miraheze/mediawiki [08:14:09] ... [08:14:11] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [08:14:12] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/56ecf6a893f8...a7929968b9df [08:14:13] [url] Comparing 56ecf6a893f8...a7929968b9df · miraheze/mediawiki · GitHub | github.com [08:14:14] [02miraheze/mediawiki] 07RhinosF1 03a792996 - emails: fix JQ spam (#7044) [08:14:14] !log [rhinos@mwtask111] starting deploy of {'world': True} to all [08:14:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:14:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:14:29] CosmicAlpha: https://github.com/miraheze/puppet/pull/2581 too [08:14:30] [url] jobrunner: set maxjobs to 1 by RhinosF1 · Pull Request #2581 · miraheze/puppet · GitHub | github.com [08:14:46] RhinosF1: I've seen. [08:15:01] CosmicAlpha: is that ok? [08:15:08] Yes. [08:22:04] !log [rhinos@mwtask111] finished deploy of {'world': True} to all - SUCCESS in 469s [08:22:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:23:14] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/manageJobs.php --type=enotifNotify --action=repush-abandoned (START) [08:23:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:26:24] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.25, 1.78, 1.54 [08:28:22] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.01, 1.44, 1.44 [08:29:50] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/manageJobs.php --type=enotifNotify --action=repush-abandoned (END - exit=2) [08:29:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:30:17] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/manageJobs.php --type=enotifNotify --action=repush-abandoned (START) [08:30:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:30:26] moving to screen [08:41:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.25, 3.64, 2.88 [08:43:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.94, 2.95, 2.71 [08:46:32] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.81, 1.52, 1.25 [08:47:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.46, 3.59, 3.05 [08:47:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.39, 3.88, 3.34 [08:48:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.51, 1.94, 1.59 [08:49:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.70, 3.14, 2.94 [08:49:31] PROBLEM - es101 APT on es101 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (1 critical updates). [08:49:31] PROBLEM - cp31 APT on cp31 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (1 critical updates). [08:50:27] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (1 critical updates). [08:50:28] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.12, 1.38, 1.25 [08:50:48] PROBLEM - mw112 APT on mw112 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:51:19] PROBLEM - test101 APT on test101 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:51:29] PROBLEM - db121 APT on db121 is CRITICAL: APT CRITICAL: 46 packages available for upgrade (1 critical updates). [08:51:40] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.10, 3.00, 3.11 [08:51:54] PROBLEM - cp30 APT on cp30 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (1 critical updates). [08:52:10] PROBLEM - mem101 APT on mem101 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (1 critical updates). [08:52:13] PROBLEM - matomo101 APT on matomo101 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (1 critical updates). [08:52:50] PROBLEM - prometheus101 APT on prometheus101 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (1 critical updates). [08:53:09] PROBLEM - cloud11 APT on cloud11 is CRITICAL: APT CRITICAL: 71 packages available for upgrade (1 critical updates). [08:53:18] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:54:24] PROBLEM - db101 APT on db101 is CRITICAL: APT CRITICAL: 46 packages available for upgrade (1 critical updates). [08:54:57] PROBLEM - es111 APT on es111 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (1 critical updates). [08:56:20] PROBLEM - puppet111 APT on puppet111 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (1 critical updates). [08:56:24] PROBLEM - cp20 APT on cp20 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (1 critical updates). [08:57:05] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (1 critical updates). [08:58:26] PROBLEM - mon111 APT on mon111 is CRITICAL: APT CRITICAL: 44 packages available for upgrade (1 critical updates). [08:58:32] PROBLEM - cp21 APT on cp21 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (1 critical updates). [08:59:05] PROBLEM - graylog121 APT on graylog121 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (1 critical updates). [08:59:32] PROBLEM - gluster121 APT on gluster121 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (1 critical updates). [09:00:15] PROBLEM - gluster111 APT on gluster111 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (1 critical updates). [09:02:37] PROBLEM - mem121 APT on mem121 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (1 critical updates). [09:03:03] PROBLEM - db112 APT on db112 is CRITICAL: APT CRITICAL: 46 packages available for upgrade (1 critical updates). [09:03:14] PROBLEM - mw101 APT on mw101 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:05:32] PROBLEM - mw122 APT on mw122 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:06:23] PROBLEM - bast101 APT on bast101 is CRITICAL: APT CRITICAL: 37 packages available for upgrade (1 critical updates). [09:07:23] PROBLEM - mw111 APT on mw111 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:07:47] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.37, 1.86, 1.95 [09:07:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 3.02, 2.81 [09:09:56] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.35, 2.72, 2.72 [09:11:18] PROBLEM - cloud10 APT on cloud10 is CRITICAL: APT CRITICAL: 71 packages available for upgrade (1 critical updates). [09:11:20] PROBLEM - db111 APT on db111 is CRITICAL: APT CRITICAL: 46 packages available for upgrade (1 critical updates). [09:11:38] PROBLEM - phab121 APT on phab121 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (1 critical updates). [09:11:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.39, 1.80, 1.88 [09:12:02] PROBLEM - mw121 APT on mw121 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:12:15] PROBLEM - mw102 APT on mw102 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:12:21] PROBLEM - cloud12 APT on cloud12 is CRITICAL: APT CRITICAL: 71 packages available for upgrade (1 critical updates). [09:12:55] PROBLEM - ldap111 APT on ldap111 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (1 critical updates). [09:13:30] PROBLEM - bast121 APT on bast121 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (1 critical updates). [09:13:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.34, 1.61, 1.80 [09:15:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.95, 3.70, 3.30 [09:15:42] PROBLEM - mail121 APT on mail121 is CRITICAL: APT CRITICAL: 45 packages available for upgrade (1 critical updates). [09:16:15] PROBLEM - es121 APT on es121 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (1 critical updates). [09:16:45] PROBLEM - gluster101 APT on gluster101 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (1 critical updates). [09:17:14] PROBLEM - jobchron121 APT on jobchron121 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:19:17] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.49, 3.12, 3.21 [09:21:55] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/b3041396df81 [09:21:57] [02miraheze/mw-config] 07Universal-Omega 03b304139 - Add ExternalData to extension-list [09:21:58] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [09:21:59] [url] Page not found · GitHub · GitHub | github.com [09:22:00] [02mw-config] 07Universal-Omega opened pull request 03#4701: Add ExternalData to extension-list - 13https://github.com/miraheze/mw-config/pull/4701 [09:22:00] [url] Page not found · GitHub · GitHub | github.com [09:22:35] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b3041396df81...017ee52fd622 [09:22:35] [url] Comparing b3041396df81...017ee52fd622 · miraheze/mw-config · GitHub | github.com [09:22:37] [02miraheze/mw-config] 07github-actions 03017ee52 - CI: lint code to MediaWiki standards [09:22:38] [02mw-config] 07github-actions[bot] synchronize pull request 03#4701: Add ExternalData to extension-list - 13https://github.com/miraheze/mw-config/pull/4701 [09:22:38] [url] Page not found · GitHub · GitHub | github.com [09:23:13] miraheze/mw-config - Universal-Omega the build passed. [09:23:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.79, 3.55, 3.11 [09:23:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.08, 1.89, 1.84 [09:25:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.93, 3.27, 3.06 [09:26:38] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [09:26:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/dcd59ec60891 [09:26:53] [02miraheze/mw-config] 07Universal-Omega 03dcd59ec - Add ExternalData to ManageWikiExtensions [09:26:55] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [09:26:55] [url] Page not found · GitHub · GitHub | github.com [09:27:59] [02mw-config] 07Universal-Omega opened pull request 03#4702: Add ExternalData to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4702 [09:27:59] [url] Page not found · GitHub · GitHub | github.com [09:28:36] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/dcd59ec60891...fea29e5e944f [09:28:36] [url] Comparing dcd59ec60891...fea29e5e944f · miraheze/mw-config · GitHub | github.com [09:28:37] [02miraheze/mw-config] 07github-actions 03fea29e5 - CI: lint code to MediaWiki standards [09:28:39] [02mw-config] 07github-actions[bot] synchronize pull request 03#4702: Add ExternalData to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4702 [09:28:39] ... [09:28:40] PROBLEM - gluster111 Disk Space on gluster111 is WARNING: DISK WARNING - free space: / 88085 MB (10% inode=80%); [09:28:47] [02mw-config] 07Universal-Omega synchronize pull request 03#4702: Add ExternalData to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4702 [09:28:47] ... [09:28:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/fea29e5e944f...658e456c3c2b [09:28:49] [url] Comparing fea29e5e944f...658e456c3c2b · miraheze/mw-config · GitHub | github.com [09:28:50] [02miraheze/mw-config] 07Universal-Omega 03658e456 - Update ManageWikiExtensions.php [09:29:14] miraheze/mw-config - Universal-Omega the build passed. [09:29:55] miraheze/mw-config - Universal-Omega the build passed. [09:30:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/bda9cc64ede2...e0eb5ad7f4fd [09:30:08] [url] Comparing bda9cc64ede2...e0eb5ad7f4fd · miraheze/mw-config · GitHub | github.com [09:30:09] [02miraheze/mw-config] 07Universal-Omega 03e0eb5ad - Formatting [09:30:42] [02miraheze/mw-config] 07Universal-Omega pushed 032 commits to 03Universal-Omega-patch-2 [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/658e456c3c2b...90d702fd0932 [09:30:42] [url] Comparing 658e456c3c2b...90d702fd0932 · miraheze/mw-config · GitHub | github.com [09:30:43] [02miraheze/mw-config] 07Universal-Omega 037b92668 - Add ExternalData to ManageWikiExtensions [09:30:45] [02miraheze/mw-config] 07Universal-Omega 0390d702f - Update ManageWikiExtensions.php [09:30:46] [02mw-config] 07Universal-Omega synchronize pull request 03#4702: Add ExternalData to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4702 [09:30:47] [url] Page not found · GitHub · GitHub | github.com [09:30:58] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/017ee52fd622...db6967d95401 [09:30:58] [url] Comparing 017ee52fd622...db6967d95401 · miraheze/mw-config · GitHub | github.com [09:30:59] [02miraheze/mw-config] 07Universal-Omega 03db6967d - Add ExternalData to extension-list [09:31:01] [02mw-config] 07Universal-Omega synchronize pull request 03#4701: Add ExternalData to extension-list - 13https://github.com/miraheze/mw-config/pull/4701 [09:31:01] [url] Page not found · GitHub · GitHub | github.com [09:31:12] miraheze/mw-config - Universal-Omega the build passed. [09:31:52] miraheze/mw-config - Universal-Omega the build passed. [09:32:07] miraheze/mw-config - Universal-Omega the build passed. [09:33:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.59, 1.98, 1.99 [09:34:49] !log [@test101] starting deploy of {'config': True} to all [09:34:52] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 3s [09:35:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:35:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:37:25] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/manageJobs.php --type=enotifNotify --action=repush-abandoned (END - exit=0) [09:37:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:39:47] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.37, 1.81, 1.89 [09:41:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.76, 1.72, 1.84 [09:43:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.03, 2.25, 2.05 [09:45:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.58, 1.99, 1.98 [09:47:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.65, 2.31, 2.10 [09:49:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.56, 1.99, 2.00 [09:52:21] !log [@mwtask111] starting deploy of {'config': True} to all [09:52:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:52:30] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 8s [09:52:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:53:47] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.12, 2.22, 2.07 [09:55:15] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [09:55:15] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:02:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.42, 3.25, 2.79 [10:04:32] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.62, 2.94, 2.73 [10:19:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 3.97, 3.19 [10:22:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.46, 2.97 [10:24:32] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.93, 2.98, 2.86 [10:25:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.43, 3.52, 3.30 [10:26:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.60, 3.71, 3.38 [10:28:46] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.27, 1.75, 1.44 [10:28:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.13, 3.22, 3.25 [10:32:43] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.38, 1.59, 1.45 [10:43:02] [02puppet] 07JohnFLewis closed pull request 03#2581: jobrunner: set maxjobs to 1 - 13https://github.com/miraheze/puppet/pull/2581 [10:43:03] [url] Page not found · GitHub · GitHub | github.com [10:43:04] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5c454b9730f7...7c0cf30aaaee [10:43:04] [url] Comparing 5c454b9730f7...7c0cf30aaaee · miraheze/puppet · GitHub | github.com [10:43:05] [02miraheze/puppet] 07RhinosF1 037c0cf30 - jobrunner: set maxjobs to 1 (#2581) [10:43:43] [02puppet] 07JohnFLewis closed pull request 03#2580: ulogd: remove old OS support - 13https://github.com/miraheze/puppet/pull/2580 [10:43:43] [url] Page not found · GitHub · GitHub | github.com [10:43:44] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7c0cf30aaaee...47161bc5156b [10:43:45] [url] Comparing 7c0cf30aaaee...47161bc5156b · miraheze/puppet · GitHub | github.com [10:43:46] [02miraheze/puppet] 07Universal-Omega 0347161bc - ulogd: remove old OS support (#2580) [10:45:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.66, 3.31 [10:47:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.70, 3.54, 3.30 [10:49:40] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.00, 3.01, 3.14 [10:54:07] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [11:04:19] [02mediawiki] 07RhinosF1 commented on commit 03a7929968b9dfcfafdaef4ad8189f3038416768bb - 13https://github.com/miraheze/mediawiki/commit/a7929968b9dfcfafdaef4ad8189f3038416768bb#commitcomment-74221680 [11:04:20] [url] emails: fix JQ spam (#7044) · miraheze/mediawiki@a792996 · GitHub | github.com [11:16:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.48, 3.28 [11:18:12] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.30, 4.09, 3.52 [11:20:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.88, 3.77, 3.46 [11:22:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.68, 4.35, 3.70 [11:25:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.61, 3.63, 3.23 [11:26:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.70, 3.24, 3.13 [11:28:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.57, 3.62 [11:30:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.02, 3.97, 3.75 [11:31:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.56, 3.18 [11:32:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.27, 3.42, 3.23 [11:33:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.56, 3.91, 3.36 [11:34:39] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.77, 1.80, 1.52 [11:34:41] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.75, 4.16, 3.54 [11:36:37] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.50, 1.57, 1.46 [11:37:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.34, 3.99, 3.54 [11:37:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.40, 1.74, 2.00 [11:38:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.24, 3.50, 3.44 [11:41:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.11, 1.83, 1.96 [11:42:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.63, 4.03, 3.66 [11:43:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.38, 3.38, 3.39 [11:43:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.57, 1.71, 1.90 [11:49:47] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.52, 1.83, 1.85 [11:50:21] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.77, 1.73, 1.51 [11:50:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.79, 3.81, 3.76 [11:51:47] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.70, 1.70, 1.80 [11:52:20] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.63, 1.70, 1.52 [11:53:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.98, 3.41, 3.41 [11:53:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.12, 1.94, 1.88 [11:55:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.69, 3.32, 3.38 [11:56:07] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [11:56:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.19, 3.62, 3.67 [12:02:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.22, 3.76, 3.72 [12:07:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.84, 3.29, 3.28 [12:08:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.27, 4.30, 3.94 [12:09:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.98, 3.38, 3.33 [12:10:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.79, 3.87, 3.82 [12:12:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.06, 3.94, 3.85 [12:12:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.07, 1.64, 1.53 [12:14:53] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.17, 1.52, 1.51 [12:16:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.62, 3.40, 3.66 [12:19:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.79, 3.84, 3.57 [12:23:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.39, 3.22, 3.35 [12:24:42] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.85, 1.70, 1.58 [12:26:40] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.27, 1.80, 1.62 [12:28:38] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.83, 1.50, 1.54 [12:29:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.42, 4.14, 3.68 [12:30:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 3.81, 3.67 [12:31:25] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.98, 3.70, 3.58 [12:35:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.73, 3.91, 3.68 [12:36:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.15, 3.92, 3.80 [12:38:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.55, 4.12, 3.91 [12:39:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.78, 3.53, 3.59 [12:40:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.05, 3.82, 3.83 [12:42:24] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.93, 1.72, 1.52 [12:44:22] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.03, 1.48, 1.46 [12:45:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.03, 3.13, 3.39 [12:46:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.27, 3.72, 3.74 [12:50:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.82, 3.89, 3.80 [12:52:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.00, 4.33, 3.98 [12:54:07] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [12:57:07] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.46, 1.86, 1.58 [12:58:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.10, 3.76, 3.84 [12:59:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.77, 3.60 [12:59:05] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.63, 1.65, 1.53 [13:01:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 3.80, 3.62 [13:02:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.52, 4.14, 3.94 [13:03:00] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.97, 1.88, 1.66 [13:04:58] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.36, 1.66, 1.60 [13:05:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.12, 3.81, 3.69 [13:05:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.90, 1.49, 1.94 [13:06:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.72, 3.70, 3.85 [13:07:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.43, 2.08, 2.13 [13:08:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.12, 4.78, 4.23 [13:12:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.86, 3.87, 3.95 [13:13:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.80, 4.02, 3.73 [13:13:47] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.64, 1.85, 2.00 [13:15:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.77, 3.68 [13:15:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.39, 2.02, 2.04 [13:16:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.31, 4.32, 4.11 [13:17:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.88, 4.10, 3.83 [13:17:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.78, 1.87, 1.98 [13:18:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.98, 3.82, 3.96 [13:19:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.45, 3.65, 3.71 [13:19:48] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 23.01, 6.52, 3.49 [13:20:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.04, 4.32, 4.12 [13:27:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.48, 3.55, 3.56 [13:35:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.75, 3.77 [13:38:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.62, 3.57, 3.96 [13:44:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.97, 3.75, 3.85 [13:45:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 3.53, 3.61 [13:47:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.61, 3.51, 3.59 [13:54:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.47, 3.69, 3.80 [13:57:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.30, 3.44 [13:58:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.12, 4.34, 3.98 [13:58:59] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.03, 2.01, 1.66 [14:00:57] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.34, 1.77, 1.61 [14:03:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.98, 3.77 [14:06:52] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.50, 1.42, 1.55 [14:07:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.67, 4.26, 3.91 [14:27:46] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.73, 2.07, 1.65 [14:29:46] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.71, 1.96, 1.67 [14:31:46] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.33, 1.69, 1.60 [14:33:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.96, 3.33, 3.85 [14:37:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.10, 1.70, 2.00 [14:39:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 3.39, 3.66 [14:41:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.98, 3.91, 3.85 [14:41:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.08, 1.80, 1.96 [14:43:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.96, 4.19, 3.95 [14:43:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.44, 1.62, 1.87 [14:47:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.16, 2.46, 2.11 [14:50:50] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.53, 3.34, 3.02 [14:51:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.64, 1.94, 1.97 [14:52:50] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.60, 3.18, 3.00 [14:53:47] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.21, 2.24, 2.07 [14:56:07] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [14:56:07] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:56:30] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.18, 3.28, 2.34 [15:02:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.00, 3.59, 3.93 [15:03:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.41, 3.62, 3.95 [15:04:22] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.55, 1.72, 1.94 [15:08:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.07, 4.52, 4.16 [15:10:16] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.09, 1.34, 1.70 [15:13:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.32, 3.65, 3.74 [15:14:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.83, 3.94, 3.99 [15:16:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.76, 4.10, 4.06 [15:18:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.97, 3.89, 3.97 [15:21:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 3.93, 3.91 [15:22:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.57, 3.95, 3.95 [15:23:59] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.97, 2.55, 2.07 [15:24:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.14, 3.51, 3.79 [15:24:43] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [15:25:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.03, 4.03, 3.94 [15:27:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.97, 3.98, 3.93 [15:28:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.10, 4.14, 3.97 [15:31:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.62, 3.76 [15:31:52] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.72, 1.99, 2.00 [15:33:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.67, 3.61, 3.73 [15:35:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.36, 10.03, 8.93 [15:39:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.70, 3.87, 3.77 [15:39:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.04, 10.79, 9.47 [15:41:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.74, 3.73 [15:41:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.96, 10.83, 9.65 [15:42:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.87, 3.58, 3.89 [15:43:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.99, 4.12, 3.87 [15:43:33] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.41, 9.96, 9.48 [15:43:46] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.39, 2.38, 2.05 [15:44:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.21, 4.36, 4.15 [15:51:46] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.27, 1.80, 1.95 [15:53:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.36, 3.98, 3.93 [15:53:10] [02puppet] 07Universal-Omega opened pull request 03#2582: deploy-mediawiki: remove l10nupdate argument - 13https://github.com/miraheze/puppet/pull/2582 [15:53:11] [url] Page not found · GitHub · GitHub | github.com [15:54:42] [02puppet] 07JohnFLewis closed pull request 03#2578: Absent l10n-modern-deploy cron - 13https://github.com/miraheze/puppet/pull/2578 [15:54:42] [url] Page not found · GitHub · GitHub | github.com [15:54:43] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/47161bc5156b...f2b2e024fa32 [15:54:44] [url] Comparing 47161bc5156b...f2b2e024fa32 · miraheze/puppet · GitHub | github.com [15:54:44] [02miraheze/puppet] 07Universal-Omega 03f2b2e02 - Absent l10n-modern-deploy cron (#2578) [15:54:57] [02puppet] 07JohnFLewis closed pull request 03#2582: deploy-mediawiki: remove l10nupdate argument - 13https://github.com/miraheze/puppet/pull/2582 [15:54:57] [url] Page not found · GitHub · GitHub | github.com [15:54:58] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f2b2e024fa32...eff9ac7e8e1d [15:54:59] [url] Comparing f2b2e024fa32...eff9ac7e8e1d · miraheze/puppet · GitHub | github.com [15:55:00] [02miraheze/puppet] 07Universal-Omega 03eff9ac7 - deploy-mediawiki: remove l10nupdate argument (#2582) [15:56:03] [02puppet] 07Universal-Omega opened pull request 03#2583: mediawiki: remove absented l10n-modern-deploy cron - 13https://github.com/miraheze/puppet/pull/2583 [15:56:04] [url] Page not found · GitHub · GitHub | github.com [15:57:46] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.91, 2.18, 2.03 [15:59:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.45, 4.94, 4.24 [15:59:46] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.41, 1.83, 1.92 [16:00:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/0ab66f36d1c8 [16:00:32] [02miraheze/mw-config] 07Universal-Omega 030ab66f3 - Update GlobalExtensions.php [16:00:33] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [16:00:34] [url] Page not found · GitHub · GitHub | github.com [16:01:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0ab66f36d1c8...bc6026ff6878 [16:01:15] [url] Comparing 0ab66f36d1c8...bc6026ff6878 · miraheze/mw-config · GitHub | github.com [16:01:16] [02miraheze/mw-config] 07Universal-Omega 03bc6026f - Update GlobalSettings.php [16:01:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/bc6026ff6878...81a1e51e751b [16:01:41] [url] Comparing bc6026ff6878...81a1e51e751b · miraheze/mw-config · GitHub | github.com [16:01:42] [02miraheze/mw-config] 07Universal-Omega 0381a1e51 - Update extension-list [16:02:41] [02mw-config] 07Universal-Omega opened pull request 03#4703: Remove LocalisationUpdate - 13https://github.com/miraheze/mw-config/pull/4703 [16:02:42] [url] Page not found · GitHub · GitHub | github.com [16:03:46] miraheze/mw-config - Universal-Omega the build passed. [16:06:27] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.74, 10.96, 9.82 [16:07:46] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.07, 1.46, 1.70 [16:10:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.20, 11.08, 10.12 [16:14:26] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.54, 9.96, 9.93 [16:26:06] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [16:28:29] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.89, 1.87, 1.61 [16:32:26] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.07, 1.57, 1.56 [16:46:50] [02puppet] 07Universal-Omega opened pull request 03#2584: jobrunner: don't set server twice in redis.queues - 13https://github.com/miraheze/puppet/pull/2584 [16:46:51] [url] Page not found · GitHub · GitHub | github.com [16:54:43] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [16:54:43] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:10:50] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.70, 3.35, 2.92 [17:12:50] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.06, 3.51, 3.02 [17:14:50] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.72, 3.08, 2.92 [17:17:47] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.70, 1.85, 2.00 [17:22:05] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.18, 2.42, 1.48 [17:23:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.92, 2.02, 1.98 [17:24:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.28, 1.86, 1.38 [17:24:07] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [17:26:05] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.14, 1.62, 1.35 [17:27:35] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.60, 2.13, 1.62 [17:28:12] [02mw-config] 07Universal-Omega closed pull request 03#4703: Remove LocalisationUpdate - 13https://github.com/miraheze/mw-config/pull/4703 [17:28:13] [url] Page not found · GitHub · GitHub | github.com [17:28:14] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [17:28:14] ... [17:28:15] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [17:28:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/mw-config/compare/e0eb5ad7f4fd...d2f027e06dc2 [17:28:17] [url] Comparing e0eb5ad7f4fd...d2f027e06dc2 · miraheze/mw-config · GitHub | github.com [17:28:18] [02miraheze/mw-config] 07Universal-Omega 03d2f027e - Remove LocalisationUpdate (#4703) [17:28:27] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'l10n': True} to all [17:28:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:28:43] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True, 'l10n': True} to all [17:28:50] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.16, 3.46, 3.08 [17:28:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:29:16] miraheze/mw-config - Universal-Omega the build passed. [17:30:23] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/a7929968b9df...6789c801a608 [17:30:23] [url] Comparing a7929968b9df...6789c801a608 · miraheze/mediawiki · GitHub | github.com [17:30:24] [02miraheze/mediawiki] 07Universal-Omega 036789c80 - Update .branches.json [17:30:39] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/0cb1d27be4e6...bc67296fcc9b [17:30:40] [url] Comparing 0cb1d27be4e6...bc67296fcc9b · miraheze/mediawiki · GitHub | github.com [17:30:40] [02miraheze/mediawiki] 07Universal-Omega 03bc67296 - Update .branches.json [17:30:50] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.57, 3.06, 2.98 [17:31:21] [02puppet] 07JohnFLewis closed pull request 03#2583: mediawiki: remove absented l10n-modern-deploy cron - 13https://github.com/miraheze/puppet/pull/2583 [17:31:22] [url] Page not found · GitHub · GitHub | github.com [17:31:23] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/eff9ac7e8e1d...e67a5f8e6007 [17:31:23] [url] Comparing eff9ac7e8e1d...e67a5f8e6007 · miraheze/puppet · GitHub | github.com [17:31:24] [02miraheze/puppet] 07Universal-Omega 03e67a5f8 - mediawiki: remove absented l10n-modern-deploy cron (#2583) [17:31:31] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.16, 1.86, 1.64 [17:34:07] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-1/±1] 13https://github.com/miraheze/mediawiki/compare/6789c801a608...a8580df41ceb [17:34:08] [url] Comparing 6789c801a608...a8580df41ceb · miraheze/mediawiki · GitHub | github.com [17:34:09] [02miraheze/mediawiki] 07Universal-Omega 03a8580df - Remove LocalisationUpdate [17:35:10] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-1/±1] 13https://github.com/miraheze/mediawiki/compare/bc67296fcc9b...621047345e1a [17:35:11] [url] Comparing bc67296fcc9b...621047345e1a · miraheze/mediawiki · GitHub | github.com [17:35:11] [02miraheze/mediawiki] 07Universal-Omega 036210473 - Remove LocalisationUpdate [17:35:28] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.34, 1.99, 1.74 [17:37:26] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.05, 1.70, 1.67 [17:43:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.64, 3.97 [17:45:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.14, 4.16, 4.11 [17:53:16] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:53:43] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:55:04] PROBLEM - prometheus101 Puppet on prometheus101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:56:07] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [17:59:03] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.11, 1.88, 1.56 [17:59:59] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'l10n': True} to all - SUCCESS in 1892s [18:00:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:00:27] !log [universalomega@test101] starting deploy of {'world': True, 'extensionlist': True} to all [18:00:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:01:01] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.52, 1.73, 1.54 [18:01:06] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True, 'l10n': True} to all - SUCCESS in 1942s [18:01:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:01:15] !log [universalomega@mwtask111] starting deploy of {'world': True, 'extensionlist': True} to all [18:01:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:02:59] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.25, 2.04, 1.69 [18:08:37] !log [universalomega@test101] finished deploy of {'world': True, 'extensionlist': True} to all - SUCCESS in 490s [18:08:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:09:43] !log [universalomega@mwtask111] finished deploy of {'world': True, 'extensionlist': True} to all - SUCCESS in 506s [18:09:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:10:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.79, 3.33, 2.96 [18:10:29] !log sudo -u www-data rm -rf /srv/mediawiki-staging/w/Extensions/LocalisationUpdate on mwtask111 and test101 [18:10:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:10:43] !log [universalomega@mwtask111] starting deploy of {'world': True, 'l10n': True, 'extensionlist': True} to all [18:10:48] !log [universalomega@test101] starting deploy of {'world': True, 'l10n': True, 'extensionlist': True} to all [18:10:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:10:51] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 2.00, 1.99, 1.81 [18:10:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:11:56] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.49, 3.05, 2.90 [18:16:46] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.43, 1.60, 1.69 [18:17:21] !log [universalomega@mwtask111] finished deploy of {'world': True, 'l10n': True, 'extensionlist': True} to all - SUCCESS in 397s [18:17:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:20:26] !log [universalomega@test101] finished deploy of {'world': True, 'l10n': True, 'extensionlist': True} to all - SUCCESS in 578s [18:20:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:20:37] !log sudo -u www-data rm /srv/mediawiki/cache/gitinfo/* on mwtask111 and test101 [18:20:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:20:49] !log [universalomega@mwtask111] starting deploy of {'world': True, 'l10n': True, 'extensionlist': True} to all [18:20:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:20:55] !log [universalomega@test101] starting deploy of {'world': True, 'l10n': True, 'extensionlist': True} to all [18:21:05] RECOVERY - prometheus101 Puppet on prometheus101 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [18:21:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:05] !log [universalomega@mwtask111] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [18:23:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:24] !log [universalomega@mwtask111] starting deploy of {'world': True, 'l10n': True, 'extensionlist': True} to all [18:23:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.48, 1.67, 1.95 [18:24:36] !log [universalomega@mwtask111] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [18:24:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:43] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [18:24:53] !log [universalomega@mwtask111] starting deploy of {'world': True, 'l10n': True, 'extensionlist': True} to all [18:24:59] !log [universalomega@test101] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [18:25:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:12] !log [universalomega@test101] starting deploy of {'world': True, 'l10n': True, 'extensionlist': True} to all [18:25:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:16] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:25:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:38] !log sudo -u www-data rm -rf /srv/mediawiki-staging/w/extensions/LocalisationUpdate on mwtask111 and test101 [18:25:47] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.34, 2.20, 2.11 [18:25:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:28:31] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.84, 1.77, 1.67 [18:32:29] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.22, 1.52, 1.59 [18:32:31] !log [universalomega@mwtask111] finished deploy of {'world': True, 'l10n': True, 'extensionlist': True} to all - SUCCESS in 458s [18:32:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:33:47] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.86, 1.78, 1.95 [18:34:20] !log [universalomega@test101] finished deploy of {'world': True, 'l10n': True, 'extensionlist': True} to all - SUCCESS in 548s [18:34:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:37:15] !log sudo -u www-data rm /srv/mediawiki/cache/l10nupdate-* on mwtask111 and test101 [18:37:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:37:43] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:37:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.57, 2.00, 1.97 [18:39:47] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.68, 1.72, 1.87 [18:40:42] !log DELETE FROM oldimage WHERE oi_archive_name=''; on dcmultiversewiki [18:40:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:41:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.33, 1.91, 1.91 [18:43:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.41, 1.69, 1.83 [18:45:19] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+1/-0/±1] 13https://github.com/miraheze/mediawiki/compare/a8580df41ceb...359f330ddb2b [18:45:19] [url] Comparing a8580df41ceb...359f330ddb2b · miraheze/mediawiki · GitHub | github.com [18:45:20] [02miraheze/mediawiki] 07Universal-Omega 03359f330 - Install ExternalData [18:45:56] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+1/-0/±1] 13https://github.com/miraheze/mediawiki/compare/621047345e1a...b81574c4d559 [18:45:56] [url] Comparing 621047345e1a...b81574c4d559 · miraheze/mediawiki · GitHub | github.com [18:45:57] [02miraheze/mediawiki] 07Universal-Omega 03b81574c - Install ExternalData [18:46:05] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/359f330ddb2b...4dfd145cc035 [18:46:05] [url] Comparing 359f330ddb2b...4dfd145cc035 · miraheze/mediawiki · GitHub | github.com [18:46:06] [02miraheze/mediawiki] 07Universal-Omega 034dfd145 - Update .branches.json [18:46:24] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/b81574c4d559...452eb4769b64 [18:46:24] [url] Comparing b81574c4d559...452eb4769b64 · miraheze/mediawiki · GitHub | github.com [18:46:25] [02miraheze/mediawiki] 07Universal-Omega 03452eb47 - Update .branches.json [18:47:14] !log [universalomega@test101] starting deploy of {'world': True, 'extensionlist': True} to all [18:47:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:47:24] !log [universalomega@mwtask111] starting deploy of {'world': True, 'extensionlist': True} to all [18:47:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:47:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.85, 2.21, 2.00 [18:52:42] !log [universalomega@test101] finished deploy of {'world': True, 'extensionlist': True} to all - SUCCESS in 328s [18:52:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:55] !log [universalomega@mwtask111] finished deploy of {'world': True, 'extensionlist': True} to all - SUCCESS in 450s [18:55:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:55:54] [02mw-config] 07Universal-Omega closed pull request 03#4701: Add ExternalData to extension-list - 13https://github.com/miraheze/mw-config/pull/4701 [18:55:54] [url] Page not found · GitHub · GitHub | github.com [18:55:55] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [18:55:55] [url] Page not found · GitHub · GitHub | github.com [18:55:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d2f027e06dc2...2042c12f8702 [18:55:57] [url] Comparing d2f027e06dc2...2042c12f8702 · miraheze/mw-config · GitHub | github.com [18:55:58] [02miraheze/mw-config] 07Universal-Omega 032042c12 - Add ExternalData to extension-list (#4701) [18:56:00] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'l10n': True} to all [18:56:00] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [18:56:04] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True, 'l10n': True} to all [18:56:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:56:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:56:53] miraheze/mw-config - Universal-Omega the build passed. [18:58:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.97, 3.70, 2.35 [19:04:17] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.19, 2.01, 1.36 [19:06:17] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.39, 1.75, 1.34 [19:07:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.68, 1.76, 1.99 [19:08:13] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.40, 1.68, 1.99 [19:08:17] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.02, 1.45, 1.28 [19:09:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:09:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.17, 2.08, 2.07 [19:10:48] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.09, 1.66, 1.40 [19:12:46] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.21, 1.56, 1.41 [19:12:57] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.51, 3.17, 2.87 [19:14:36] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:14:53] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.21, 2.84, 2.78 [19:16:08] PROBLEM - cp21 Disk Space on cp21 is WARNING: DISK WARNING - free space: / 4267 MB (10% inode=96%); [19:16:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/90d702fd0932...7ebba9a0f151 [19:16:25] [url] Comparing 90d702fd0932...7ebba9a0f151 · miraheze/mw-config · GitHub | github.com [19:16:26] [02miraheze/mw-config] 07Universal-Omega 037ebba9a - Update LocalSettings.php [19:16:28] [02mw-config] 07Universal-Omega synchronize pull request 03#4702: Add ExternalData to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4702 [19:16:28] [url] Page not found · GitHub · GitHub | github.com [19:17:33] miraheze/mw-config - Universal-Omega the build passed. [19:20:07] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.08, 1.32, 1.64 [19:21:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7ebba9a0f151...e3e88d42467f [19:21:04] [url] Comparing 7ebba9a0f151...e3e88d42467f · miraheze/mw-config · GitHub | github.com [19:21:05] [02miraheze/mw-config] 07Universal-Omega 03e3e88d4 - add punctuation [19:21:07] [02mw-config] 07Universal-Omega synchronize pull request 03#4702: Add ExternalData to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4702 [19:21:07] [url] Page not found · GitHub · GitHub | github.com [19:22:09] miraheze/mw-config - Universal-Omega the build passed. [19:26:03] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:26:07] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [19:26:07] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:26:18] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True, 'l10n': True} to all - SUCCESS in 1813s [19:26:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:29] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.27, 2.01, 1.70 [19:28:07] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 52095 bytes in 5.835 second response time [19:28:13] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'l10n': True} to all - SUCCESS in 1933s [19:28:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:28:27] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.40, 1.72, 1.63 [19:29:20] !log [universalomega@test101] starting deploy of {'config': True} to all [19:29:21] !log [universalomega@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [19:29:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:29:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:30:26] [02mw-config] 07Universal-Omega synchronize pull request 03#4702: Add ExternalData to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4702 [19:30:26] [url] Page not found · GitHub · GitHub | github.com [19:30:27] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e3e88d42467f...3f0e12e7923f [19:30:28] [url] Comparing e3e88d42467f...3f0e12e7923f · miraheze/mw-config · GitHub | github.com [19:30:29] [02miraheze/mw-config] 07Universal-Omega 033f0e12e - Update ManageWikiExtensions.php [19:30:37] !log [universalomega@test101] starting deploy of {'config': True} to all [19:30:38] !log [universalomega@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [19:30:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:30:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:31:29] miraheze/mw-config - Universal-Omega the build passed. [19:32:23] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.12, 1.51, 1.57 [19:33:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.32, 3.14, 3.87 [19:35:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.60, 3.70, 3.99 [19:36:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.44, 10.55, 9.76 [19:37:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.62, 3.31, 3.81 [19:39:32] !log [@test101] starting deploy of {'config': True} to all [19:39:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:39:43] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 10s [19:39:47] PROBLEM - cp20 Disk Space on cp20 is WARNING: DISK WARNING - free space: / 4277 MB (10% inode=96%); [19:39:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:41:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.93, 3.97, 3.95 [19:43:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.78, 3.89 [19:44:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.89, 9.52, 9.67 [19:45:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.58, 4.06, 3.95 [19:49:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.91, 3.93 [19:50:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.48, 1.90, 1.50 [19:51:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.51, 4.20, 4.04 [19:54:43] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [19:58:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.27, 1.93, 1.76 [19:58:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.09, 1.74, 1.46 [20:00:53] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.08, 1.53, 1.42 [20:01:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.70, 10.03, 8.81 [20:02:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.30, 1.55, 1.64 [20:03:18] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.60, 8.97, 8.58 [20:04:47] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.28, 1.73, 1.55 [20:06:44] !log [@test101] starting deploy of {'config': True} to all [20:06:45] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.49, 1.31, 1.42 [20:06:46] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 2s [20:06:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:21:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.64, 1.75, 1.99 [20:23:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.70, 2.52, 2.27 [20:32:50] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.65, 3.26, 2.79 [20:34:50] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.00, 3.22, 2.84 [20:36:12] !log [@test101] starting deploy of {'config': True} to all [20:36:15] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 3s [20:36:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:36:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:38:50] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.79, 3.40, 2.99 [20:40:50] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.24, 3.19, 2.98 [20:44:05] [02mw-config] 07Universal-Omega closed pull request 03#4702: Add ExternalData to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4702 [20:44:05] [url] Page not found · GitHub · GitHub | github.com [20:44:06] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [20:44:08] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [20:44:08] [url] Page not found · GitHub · GitHub | github.com [20:44:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/2042c12f8702...f79ac8233a2e [20:44:10] [url] Comparing 2042c12f8702...f79ac8233a2e · miraheze/mw-config · GitHub | github.com [20:44:11] [02miraheze/mw-config] 07Universal-Omega 03f79ac82 - Add ExternalData to ManageWikiExtensions (#4702) [20:44:19] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [20:44:22] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 2s [20:44:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:44:25] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [20:44:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:44:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:44:36] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [20:44:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:45:10] miraheze/mw-config - Universal-Omega the build passed. [20:49:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.93, 1.22, 1.95 [20:53:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.64, 1.38, 1.82 [20:56:54] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.01, 1.62, 1.30 [20:58:52] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.31, 1.51, 1.30 [21:03:49] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.63, 1.59, 1.87 [21:05:07] [02ssl] 07Universal-Omega opened pull request 03#527: add www.cgradegames.net cert - 13https://github.com/miraheze/ssl/pull/527 [21:05:08] [url] Page not found · GitHub · GitHub | github.com [21:06:12] [02ssl] 07Universal-Omega synchronize pull request 03#527: add www.cgradegames.net cert - 13https://github.com/miraheze/ssl/pull/527 [21:06:13] ... [21:13:47] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.03, 1.36, 1.64 [21:21:48] [02puppet] 07JohnFLewis closed pull request 03#2579: img-csp: allow all of wikimedia.org - 13https://github.com/miraheze/puppet/pull/2579 [21:21:48] [url] Page not found · GitHub · GitHub | github.com [21:21:49] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e67a5f8e6007...9666dc1163c4 [21:21:50] [url] Comparing e67a5f8e6007...9666dc1163c4 · miraheze/puppet · GitHub | github.com [21:21:51] [02miraheze/puppet] 07RhinosF1 039666dc1 - img-csp: allow all of wikimedia.org (#2579) [21:21:51] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 14821 MB (11% inode=99%); [21:23:13] RECOVERY - cp21 APT on cp21 is OK: APT OK: 33 packages available for upgrade (0 critical updates). [21:23:14] RECOVERY - jobchron121 APT on jobchron121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:23:17] RECOVERY - cloud10 APT on cloud10 is OK: APT OK: 70 packages available for upgrade (0 critical updates). [21:23:25] RECOVERY - matomo101 APT on matomo101 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [21:23:28] RECOVERY - bast121 APT on bast121 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [21:23:28] RECOVERY - cp31 APT on cp31 is OK: APT OK: 33 packages available for upgrade (0 critical updates). [21:23:30] RECOVERY - db121 APT on db121 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [21:23:31] RECOVERY - mw122 APT on mw122 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:23:45] RECOVERY - mail121 APT on mail121 is OK: APT OK: 44 packages available for upgrade (0 critical updates). [21:23:45] RECOVERY - mem121 APT on mem121 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [21:23:52] RECOVERY - cp30 APT on cp30 is OK: APT OK: 33 packages available for upgrade (0 critical updates). [21:23:56] RECOVERY - test101 APT on test101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:24:02] RECOVERY - mw121 APT on mw121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:24:10] RECOVERY - mem101 APT on mem101 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [21:24:12] RECOVERY - phab121 APT on phab121 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [21:24:14] RECOVERY - mw102 APT on mw102 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:24:15] RECOVERY - es121 APT on es121 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [21:24:20] RECOVERY - cloud12 APT on cloud12 is OK: APT OK: 70 packages available for upgrade (0 critical updates). [21:24:24] RECOVERY - bast101 APT on bast101 is OK: APT OK: 36 packages available for upgrade (0 critical updates). [21:24:24] RECOVERY - db101 APT on db101 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [21:24:27] RECOVERY - ns2 APT on ns2 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [21:24:28] !log upgrade libxml2 on all servers [21:24:33] RECOVERY - gluster111 APT on gluster111 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [21:24:33] RECOVERY - graylog121 APT on graylog121 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [21:24:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:24:47] RECOVERY - prometheus101 APT on prometheus101 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [21:24:49] RECOVERY - gluster101 APT on gluster101 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [21:24:53] RECOVERY - ldap111 APT on ldap111 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [21:24:58] RECOVERY - es111 APT on es111 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [21:25:04] RECOVERY - db112 APT on db112 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [21:25:04] RECOVERY - mw112 APT on mw112 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:25:05] RECOVERY - ns1 APT on ns1 is OK: APT OK: 36 packages available for upgrade (0 critical updates). [21:25:06] RECOVERY - mon111 APT on mon111 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [21:25:09] RECOVERY - cloud11 APT on cloud11 is OK: APT OK: 70 packages available for upgrade (0 critical updates). [21:25:13] RECOVERY - mw101 APT on mw101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:25:16] RECOVERY - db111 APT on db111 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [21:25:16] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:25:23] RECOVERY - mw111 APT on mw111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:25:29] RECOVERY - es101 APT on es101 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [21:25:36] RECOVERY - gluster121 APT on gluster121 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [21:26:57] RECOVERY - cp20 APT on cp20 is OK: APT OK: 33 packages available for upgrade (0 critical updates). [21:31:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.22, 3.30, 3.95 [21:33:15] [02ssl] 07Universal-Omega opened pull request 03#528: Add miraheze.cf cert - 13https://github.com/miraheze/ssl/pull/528 [21:33:15] [url] Page not found · GitHub · GitHub | github.com [21:35:27] [02ssl] 07Universal-Omega synchronize pull request 03#528: Add miraheze.cf cert - 13https://github.com/miraheze/ssl/pull/528 [21:35:27] [url] Page not found · GitHub · GitHub | github.com [21:35:40] [02ssl] 07Universal-Omega edited pull request 03#528: Add miraheze.cf cert + redirect - 13https://github.com/miraheze/ssl/pull/528 [21:35:40] ... [21:39:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.06, 3.90, 3.91 [21:40:06] RECOVERY - puppet111 APT on puppet111 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [21:41:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.70, 1.87, 1.60 [21:43:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.80, 3.87 [21:43:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.98, 1.59, 1.53 [21:44:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.75, 3.05, 3.83 [21:46:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.05, 3.60, 3.92 [21:47:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.12, 3.70, 3.78 [21:48:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.64, 3.56, 3.87 [21:49:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.57, 3.72 [21:52:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.76, 4.09, 4.01 [21:53:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 3.72, 3.73 [21:54:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.55, 3.97, 3.98 [21:55:36] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.37, 1.80, 1.67 [21:56:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.85, 4.70, 4.25 [21:57:35] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.75, 1.38, 1.52 [22:05:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.37, 3.52, 3.88 [22:09:26] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.95, 2.66, 1.91 [22:09:51] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14600 MB (10% inode=99%); [22:11:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.03, 4.78, 4.24 [22:19:20] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.63, 1.44, 1.84 [22:23:17] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.33, 1.91, 1.93 [22:25:16] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.12, 1.58, 1.80 [22:27:14] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.89, 1.32, 1.68 [22:29:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.41, 3.27, 3.88 [22:38:15] [02mw-config] 07dmehus opened pull request 03#4704: Temporarily adjust setting to allow me to reset my Betaheze password - 13https://github.com/miraheze/mw-config/pull/4704 [22:38:15] [url] Page not found · GitHub · GitHub | github.com [22:39:47] miraheze/mw-config - dmehus the build has errored. [22:41:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.36, 4.28, 3.91 [22:44:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.99, 3.32, 3.96 [22:45:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.86, 3.97, 3.86 [22:52:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.34, 3.65, 3.81 [22:53:35] dmehus: re that pull request what is the full error that is shown when you try to do a password reset [22:54:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.12, 3.00, 3.55 [22:56:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.12, 3.66, 3.71 [22:57:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.35, 3.61, 3.63 [23:00:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.87, 3.94, 3.82 [23:03:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.54, 3.37, 3.62 [23:03:15] PROBLEM - prometheus101 APT on prometheus101 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [23:03:49] i do see that you are somehow part of the steward and interwiki-admin global groups even though they don't actually exist on beta [23:03:50] MacFan4000: `Passwords must be at least 12 characters.` when I try to do a password reset [23:04:23] It shouldn't be doing that, it should at least send me a temporary password to my e-mail address, allowing me to choose a new password of at least 12 characters [23:05:15] RECOVERY - prometheus101 APT on prometheus101 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [23:05:34] MacFan4000, yeah, from my conversations with CosmicAlpha, and JohnLewis as well, we're wondering if it was a mistake to use a separate CentralAuth table from the main table [23:06:32] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.31, 2.70, 3.27 [23:06:37] That's because of session issues. Purging your session should've worked but didn't for you. Even though the exact same thing happened to me originally and purging it once worked properly, then I was able to login, and the CA groups reflected properly. [23:07:50] Tbh, migrating to a split CentralAuth was a huge mistake, we don't have the setup intended to support it like that really. It's a mess with session cache. [23:07:53] We shouldn't have copied over the existing users from `test3wiki`, imho. We should've started fresh, if we're going to have a separate CA table and separate groups, and require user accounts on Betaheze be created by an existing user [23:08:38] > Tbh, migrating to a split CentralAuth was a huge mistake, we don't have the setup intended to support it like that really. It's a mess with session cache. [23:08:38] yeah, I'd definitely strongly support a reset. There's not much to lose on Betaheze in doing so [23:09:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.37, 3.38, 3.48 [23:09:27] dmehus: A reset isn't going to fix anything. Only reverting the CA split would, and that'd actually take some work at this point. [23:09:59] It’s all just a fragile setup, even in production at this point :P [23:10:10] CosmicAlpha, how would it take some work? Just dump the Betaheze databases, update the configuration, and recreate the wiki(s) [23:11:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.48, 3.07, 3.36 [23:11:41] test101 is almost unusable now also, because somehow at somepoint it became a redis server also, so running redis and jobrunner, and pretty much everything else, and has less resources then any single production server. So it's so slow to actually use. [23:12:02] CosmicAlpha, interesting [23:12:35] tbf JohnLewis did raise objections to Betaheze before it was created, so perhaps he foresaw all of these issues :P [23:12:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.75, 3.62, 3.50 [23:13:53] dmehus: it's not that simple because of the caching setup, primarily session cache, would be a mess trying to revert that, and then the config needs changed, and may have some other effects. During the initial CA split, something caused issues with production, I can't fully remember what happened though, my concern is reverting now could have unforseen effects on production. [23:13:58] wikimedia somehow has a working seperate ca database for their beta cluster [23:14:21] They have dedicated caching servers for beta cluster. [23:14:26] ah [23:14:32] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.49, 3.21, 3.37 [23:17:58] CosmicAlpha, so don't revert the split (i.e., keep separate CA tables), but rather dump the Betaheze user tables, and require users to create separate accounts on Betaheze (or have SRE create accounts for them) [23:18:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.13, 3.82, 3.53 [23:18:45] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.82, 1.97, 1.57 [23:18:48] That's what should've been done initially. [23:18:58] yep [23:19:12] this is why I don't like copying over user tables from existing databases [23:19:27] Yeah no, that wasn’t a good idea [23:19:41] Who did that anyway? [23:19:55] Yeah, that makes sense (re: dedicated caching infrastructure). We don't have the resources to support separate cache proxies for Betaheze lol [23:20:10] Wasn’t Rhinos in charge of the initial CA stuff for Betaheze? [23:20:16] I'm guessing RhinosF1 as he has pushed for direct imports of .sql databases from external wikis in the past [23:20:43] dmehus: It would be memcached we'd need separate for, not varnish cache proxies really. [23:20:52] CosmicAlpha, oh ok [23:21:28] Agent: yeah I guess so. Oh well, what's done is done, but we need to fix it. I'm mobile for the day now though. [23:22:15] CosmicAlpha, yeah [23:22:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.83, 2.40, 1.80 [23:22:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 3.76, 3.55 [23:24:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.70, 3.59 [23:24:41] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.09, 1.86, 1.67 [23:24:48] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.37, 9.84, 8.93 [23:24:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.05, 3.14, 3.34 [23:26:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.91, 4.40, 3.85 [23:26:48] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.77, 9.37, 8.88 [23:28:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.41, 1.63, 1.62 [23:30:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.32, 3.88, 3.77 [23:37:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.85, 3.45, 3.94 [23:38:17] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.97, 1.86, 1.42 [23:38:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.05, 4.15, 3.87 [23:40:17] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.29, 1.88, 1.47 [23:40:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.92, 3.49, 3.66 [23:42:17] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.45, 1.79, 1.49 [23:43:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.24, 3.56, 3.79 [23:44:17] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.49, 1.70, 1.50 [23:44:32] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.90, 3.02, 3.40 [23:45:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.90, 3.52, 3.77 [23:48:53] CosmicAlpha, your BackendPerformance.js script doesn't seem to work with New Vector :( [23:49:06] It doesn't display the wiki database name or jobs outstanding for me [23:49:54] dmehus: it's in the user personal menu drop down. [23:50:17] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.47, 1.93, 1.65 [23:51:43] CosmicAlpha, hrm? I mean, I didn't have Classic/Legacy Vector set as a global preference, as I didn't think that was a global preference setable setting, so I've now done that, but we should still fix the script to work for people who use New Vector, no? [23:54:17] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.80, 1.35, 1.48 [23:55:08] CosmicAlpha, that script has a personal menu? [23:55:28] link? [23:55:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 3.48, 3.59 [23:57:40] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.97, 2.99, 3.40 [23:59:27] dmehus: no, New vector does, where the stuff from the script appears.