[00:00:14] RECOVERY - ldap141 APT on ldap141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:01:17] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.86, 1.96, 1.97 [00:01:26] !log apt dist-upgrade and reboot ldap141 (includes kernel upgrade) [00:01:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:01:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.93, 3.98, 3.73 [00:02:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.92, 9.70, 10.05 [00:03:17] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.31, 2.12, 2.02 [00:04:11] PROBLEM - cp30 Disk Space on cp30 is WARNING: DISK WARNING - free space: / 3818 MB (9% inode=96%); [00:05:40] PROBLEM - mail121 APT on mail121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [00:06:11] RECOVERY - cp30 Disk Space on cp30 is OK: DISK OK - free space: / 10932 MB (28% inode=96%); [00:07:41] RECOVERY - mail121 APT on mail121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:07:55] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.98, 2.87, 3.30 [00:08:02] PROBLEM - mail121 SMTP on mail121 is CRITICAL: connect to address 2a10:6740::6:307 and port 25: Connection refusedSMTP CRITICAL - 0.001 sec. response time [00:08:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.49, 10.86, 11.92 [00:09:17] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.84, 1.77, 1.98 [00:09:58] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.88, 1.91, 1.92 [00:10:02] RECOVERY - mail121 SMTP on mail121 is OK: SMTP OK - 0.136 sec. response time [00:11:58] !log apt dist-upgrade and reboot mail121 (includes kernel upgrade) [00:12:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:14:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.81, 10.39, 11.16 [00:15:55] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.55, 4.15, 3.69 [00:15:58] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.79, 2.16, 2.00 [00:16:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.97, 10.48, 11.08 [00:17:17] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.24, 1.92, 1.90 [00:17:24] RECOVERY - jobchron121 APT on jobchron121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:18:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.96, 12.71, 11.84 [00:18:57] !log apt dist-upgrade and reboot jobchron121 (includes kernel upgrade) [00:19:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:19:17] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.74, 1.79, 1.85 [00:19:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.08, 3.50, 3.56 [00:20:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.46, 11.98, 11.67 [00:21:08] !log restart jobrunner service on mw* and mwtask141 [00:21:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:21:17] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.88, 2.20, 1.99 [00:23:55] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.01, 2.67, 3.22 [00:24:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.90, 12.25, 11.84 [00:25:58] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.35, 1.85, 1.99 [00:26:26] !log apt dist-upgrade and reboot mem131 (includes kernel upgrade) [00:26:27] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 5.36, 4.23, 2.93 [00:26:53] RECOVERY - mem131 APT on mem131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:26:57] !log apt dist-upgrade and reboot mem101 (includes kernel upgrade) [00:28:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:28:27] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.30, 3.90, 2.97 [00:28:32] !log apt dist-upgrade and reboot mem131 (includes kernel upgrade) [00:28:38] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [00:28:44] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 20.04, 9.30, 5.35 [00:28:56] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:29:17] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.19, 1.91, 1.97 [00:29:39] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:29:48] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:29:55] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.54, 4.62, 3.91 [00:29:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:30:11] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:30:34] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:31:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:31:38] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.463 second response time [00:31:45] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.339 second response time [00:31:46] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw122 [00:31:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.18, 3.83, 3.71 [00:32:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 3.97, 9.47, 11.46 [00:32:33] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 2.469 second response time [00:32:56] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.322 second response time [00:33:46] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [00:33:58] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.08, 1.86, 1.89 [00:34:19] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.316 second response time [00:34:42] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.510 second response time [00:35:00] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.318 second response time [00:35:17] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.08, 2.01, 1.94 [00:36:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.47, 11.35, 11.65 [00:36:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.54, 11.53, 10.35 [00:37:55] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.71, 2.96, 3.40 [00:41:58] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.16, 1.69, 1.86 [00:42:16] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.03, 12.07, 10.80 [00:45:58] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.99, 2.07, 1.98 [00:46:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.42, 11.55, 10.92 [00:47:18] RECOVERY - mon141 APT on mon141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:49:45] RECOVERY - mon141 APT on mon141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:50:02] !log apt dist-upgrade and reboot mon141 (includes kernel upgrade) [00:50:04] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.12, 1.94, 1.94 [00:50:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:52:34] !log restart irclogserverbot service on mon141 [00:52:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:53:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.83, 9.22, 10.16 [00:53:49] !log restart ircrcbot service on mon141 [00:53:53] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.50, 1.79, 1.89 [00:53:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.18, 10.14, 11.69 [00:55:30] RECOVERY - es131 APT on es131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:59:19] PROBLEM - es131 Puppet on es131 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[puppet-agent],Package[elasticsearch] [00:59:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.00, 1.63, 1.99 [01:01:16] RECOVERY - es131 Puppet on es131 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [01:02:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.24, 11.18, 11.37 [01:02:59] !log apt dist-upgrade and reboot es131 (includes kernel upgrade) [01:03:09] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.43, 2.44, 3.87 [01:03:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:03:49] !log apt dist-upgrade and reboot es141 (includes kernel upgrade) [01:03:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:06:56] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.37, 11.60, 11.66 [01:07:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.66, 2.03, 3.39 [01:07:17] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.56, 1.58, 1.69 [01:07:28] RECOVERY - puppet141 APT on puppet141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [01:08:17] !log apt dist-upgrade and reboot puppet141 (includes kernel upgrade) [01:08:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:11:01] RECOVERY - ns2 APT on ns2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [01:11:13] RECOVERY - ns1 APT on ns1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [01:11:17] !log apt dist-upgrade and reboot ns1 (includes kernel upgrade) [01:11:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:12:59] !log apt dist-upgrade and reboot ns2 (includes kernel upgrade) [01:13:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:14:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.43, 11.61, 11.55 [01:15:33] RECOVERY - db141 APT on db141 is OK: APT OK: 10 packages available for upgrade (0 critical updates). [01:16:23] !log upgrade libgnutls30 on db131 and db141 [01:16:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:16:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.25, 10.87, 11.28 [01:16:57] RECOVERY - db131 APT on db131 is OK: APT OK: 10 packages available for upgrade (0 critical updates). [01:20:42] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.03, 2.01, 1.79 [01:22:05] !log upgrade gpg-wks-client, dirmngr, gpg-wks-server, gnupg-utils, gpg-agent, gpg, gpgconf, gnupg-l10n, gnupg, gpgsm, gpgv, libssl1.1, gsasl-common, libgsasl7, and openssl on gluster101 and gluster121 [01:22:08] RECOVERY - gluster121 APT on gluster121 is OK: APT OK: 14 packages available for upgrade (0 critical updates). [01:22:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:22:20] RECOVERY - gluster101 APT on gluster101 is OK: APT OK: 14 packages available for upgrade (0 critical updates). [01:22:37] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.95, 1.92, 1.77 [01:22:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.66, 10.54, 10.87 [01:23:23] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.17, 1.81, 1.84 [01:24:31] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.41, 2.28, 1.93 [01:25:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.94, 1.91, 1.88 [01:25:38] RECOVERY - cloud14 APT on cloud14 is OK: APT OK: 14 packages available for upgrade (0 critical updates). [01:25:45] !log upgrade libgnutlsxx28, libgnutls-dane0, libgnutls30, curl, libcurl4, gnutls-bin, libcurl3-gnutls, libxslt1.1, and xsltproc on cloud* [01:25:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.04, 10.40, 9.54 [01:25:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:25:56] RECOVERY - cloud11 APT on cloud11 is OK: APT OK: 49 packages available for upgrade (0 critical updates). [01:26:06] RECOVERY - cloud12 APT on cloud12 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [01:26:26] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.23, 1.81, 1.79 [01:26:45] RECOVERY - cloud13 APT on cloud13 is OK: APT OK: 14 packages available for upgrade (0 critical updates). [01:27:23] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.12, 1.91, 1.88 [01:27:48] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.97, 9.31, 9.25 [01:28:37] !log also upgrade libsnmp-base, snmp, snmpd, libsnmp40, libtirpc-common, libtirpc-dev, libtirpc3, linux-libc-dev, and unzip on cloud10 [01:28:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:28:52] RECOVERY - cloud10 APT on cloud10 is OK: APT OK: 49 packages available for upgrade (0 critical updates). [01:29:23] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.73, 1.83, 1.85 [01:29:48] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/commit/1c0e287a6280 [01:29:50] [02miraheze/ssl] 07Universal-Omega 031c0e287 - Remove no-longer pointing custom domain [01:29:51] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [01:29:52] [url] Page not found · GitHub · GitHub | github.com [01:29:53] [02ssl] 07Universal-Omega opened pull request 03#598: Remove no-longer pointing custom domain - 13https://github.com/miraheze/ssl/pull/598 [01:29:53] [url] Page not found · GitHub · GitHub | github.com [01:30:16] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.07, 1.80, 1.77 [01:30:20] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1c0e287a6280...f310ff26158c [01:30:20] [url] Comparing 1c0e287a6280...f310ff26158c · miraheze/ssl · GitHub | github.com [01:30:21] [02miraheze/ssl] 07Universal-Omega 03f310ff2 - Update certs.yaml [01:30:23] [02ssl] 07Universal-Omega synchronize pull request 03#598: Remove no-longer pointing custom domain - 13https://github.com/miraheze/ssl/pull/598 [01:30:23] [url] Page not found · GitHub · GitHub | github.com [01:31:02] [02ssl] 07Universal-Omega closed pull request 03#598: Remove no-longer pointing custom domain - 13https://github.com/miraheze/ssl/pull/598 [01:31:03] [url] Page not found · GitHub · GitHub | github.com [01:31:04] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/aa4fe55621de...e8a72df089be [01:31:04] [url] Comparing aa4fe55621de...e8a72df089be · miraheze/ssl · GitHub | github.com [01:31:05] [02miraheze/ssl] 07Universal-Omega 03e8a72df - Remove no-longer pointing custom domain (#598) [01:31:07] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [01:31:08] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [01:31:09] ... [01:31:22] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.45, 1.96, 1.89 [01:31:34] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/494a476f46f7...dbb370b03ca5 [01:31:35] [url] Comparing 494a476f46f7...dbb370b03ca5 · miraheze/dns · GitHub | github.com [01:31:36] [02miraheze/dns] 07Universal-Omega 03dbb370b - Remove no-longer pointing custom domain zone [01:32:35] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 5.61, 4.38, 3.14 [01:33:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.14, 1.62, 1.77 [01:33:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.00, 11.41, 10.08 [01:34:37] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/commit/57b7f8d44788 [01:34:38] [02miraheze/ssl] 07Universal-Omega 0357b7f8d - Remove no-longer pointing custom domain [01:34:40] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [01:34:41] [url] Page not found · GitHub · GitHub | github.com [01:34:41] [02ssl] 07Universal-Omega opened pull request 03#599: Remove no-longer pointing custom domain - 13https://github.com/miraheze/ssl/pull/599 [01:34:42] [url] Page not found · GitHub · GitHub | github.com [01:35:12] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/57b7f8d44788...2c59ada06f8c [01:35:12] [url] Comparing 57b7f8d44788...2c59ada06f8c · miraheze/ssl · GitHub | github.com [01:35:13] [02miraheze/ssl] 07Universal-Omega 032c59ada - Update certs.yaml [01:35:15] [02ssl] 07Universal-Omega synchronize pull request 03#599: Remove no-longer pointing custom domain - 13https://github.com/miraheze/ssl/pull/599 [01:35:15] [url] Page not found · GitHub · GitHub | github.com [01:35:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.27, 10.87, 10.04 [01:35:52] [02ssl] 07Universal-Omega closed pull request 03#599: Remove no-longer pointing custom domain - 13https://github.com/miraheze/ssl/pull/599 [01:35:53] [url] Page not found · GitHub · GitHub | github.com [01:35:54] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [01:35:55] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [01:35:56] [url] Page not found · GitHub · GitHub | github.com [01:35:57] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/e8a72df089be...9cbc83ed1d29 [01:35:57] [url] Comparing e8a72df089be...9cbc83ed1d29 · miraheze/ssl · GitHub | github.com [01:35:58] [02miraheze/ssl] 07Universal-Omega 039cbc83e - Remove no-longer pointing custom domain (#599) [01:36:07] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.97, 1.61, 1.72 [01:36:56] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/tensegritywiki.com.key] [01:38:11] !log restart nutcracker service on mw*, mwtask141, and test131 [01:38:26] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 6.22, 5.41, 3.98 [01:38:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:39:07] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/tensegritywiki.com.key] [01:39:18] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/tensegritywiki.com.key],File[/etc/ssl/private/wiki.minebox.fr.key] [01:40:26] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.55, 4.33, 3.76 [01:41:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.88, 1.52, 1.68 [01:42:50] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [01:43:05] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:43:16] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:43:27] PROBLEM - puppet141 Puppet on puppet141 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/tensegritywiki.com.key],File[/etc/ssl/localcerts/wiki.minebox.fr.crt],File[/etc/ssl/private/wiki.minebox.fr.key] [01:44:07] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.22, 1.48, 1.64 [01:45:22] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.86, 2.34, 1.98 [01:47:25] RECOVERY - puppet141 Puppet on puppet141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:47:33] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.42, 9.19, 9.89 [01:50:07] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.70, 2.33, 1.95 [01:50:42] PROBLEM - cloud11 PowerDNS Recursor on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host [01:50:50] PROBLEM - cloud11 Current Load on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host [01:50:50] PROBLEM - cloud11 NTP time on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host [01:51:02] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host [01:51:02] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host [01:51:05] PROBLEM - cloud11 conntrack_table_size on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host [01:51:13] PROBLEM - Host cloud11 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:200) [01:51:13] PROBLEM - cloud11 SSH on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 and port 22: No route to host [01:53:58] !log apt upgrade and reboot cloud11 (includes kernel upgrade) [01:54:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:54:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.21, 9.49, 11.87 [01:54:43] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.18, 1.83, 1.96 [01:55:28] RECOVERY - Host cloud11 is UP: PING OK - Packet loss = 0%, RTA = 1.11 ms [01:56:31] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [01:56:36] RECOVERY - cloud11 SSH on cloud11 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [01:56:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.04, 2.09, 2.05 [01:56:41] RECOVERY - cloud11 SMART on cloud11 is OK: OK: [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean| [01:56:51] RECOVERY - cloud11 PowerDNS Recursor on cloud11 is OK: DNS OK: 0.024 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [01:57:01] RECOVERY - cloud11 Current Load on cloud11 is OK: OK - load average: 0.59, 0.99, 0.49 [01:57:11] RECOVERY - cloud11 conntrack_table_size on cloud11 is OK: OK: nf_conntrack is 0 % full [01:57:15] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.99, 1.97, 1.91 [01:57:21] RECOVERY - cloud11 NTP time on cloud11 is OK: NTP OK: Offset 0.0007894039154 secs [02:00:19] PROBLEM - mon141 grafana.miraheze.org HTTPS on mon141 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 398 bytes in 0.016 second response time [02:00:24] PROBLEM - cp30 Stunnel HTTP for matomo131 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2418 bytes in 0.371 second response time [02:00:24] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4167 bytes in 0.060 second response time [02:00:33] PROBLEM - cp31 Stunnel HTTP for mon141 on cp31 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 398 bytes in 0.255 second response time [02:00:40] PROBLEM - mon141 HTTPS on mon141 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 398 bytes in 0.012 second response time [02:00:43] PROBLEM - db112 MariaDB on db112 is CRITICAL: Can't connect to MySQL server on 'db112.miraheze.org' (115) [02:00:56] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [02:00:57] PROBLEM - cp31 Stunnel HTTP for matomo131 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2418 bytes in 0.269 second response time [02:01:04] PROBLEM - cp30 Stunnel HTTP for mon141 on cp30 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 398 bytes in 0.267 second response time [02:01:12] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.75, 2.06, 1.94 [02:01:20] PROBLEM - matomo131 HTTPS on matomo131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.006 second response time [02:03:10] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.65, 1.97, 1.93 [02:03:36] !log apt upgrade and reboot db112 (includes kernel upgrade) [02:03:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:03:50] !log restart mariadb service on db112 [02:03:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:04:18] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 18991 bytes in 0.169 second response time [02:04:19] RECOVERY - mon141 grafana.miraheze.org HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 200 OK - 37532 bytes in 0.060 second response time [02:04:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.73, 10.89, 11.17 [02:04:23] RECOVERY - cp30 Stunnel HTTP for matomo131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 100150 bytes in 0.628 second response time [02:04:27] RECOVERY - cp31 Stunnel HTTP for mon141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 37532 bytes in 0.346 second response time [02:04:35] RECOVERY - mon141 HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.032 second response time [02:04:43] RECOVERY - db112 MariaDB on db112 is OK: Uptime: 82 Threads: 8 Questions: 58645 Slow queries: 788 Opens: 176 Open tables: 170 Queries per second avg: 715.182 [02:04:49] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17684 bytes in 0.106 second response time [02:04:52] RECOVERY - cp31 Stunnel HTTP for matomo131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 100150 bytes in 0.743 second response time [02:05:01] RECOVERY - cp30 Stunnel HTTP for mon141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 37547 bytes in 0.353 second response time [02:05:19] RECOVERY - matomo131 HTTPS on matomo131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.116 second response time [02:07:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.38, 9.94, 10.77 [02:07:25] RECOVERY - cp31 Stunnel HTTP for mon141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 37547 bytes in 0.344 second response time [02:07:33] RECOVERY - mon141 HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.029 second response time [02:07:43] RECOVERY - db112 MariaDB on db112 is OK: Uptime: 262 Threads: 9 Questions: 115624 Slow queries: 1692 Opens: 188 Open tables: 182 Queries per second avg: 441.312 [02:07:50] RECOVERY - cp31 Stunnel HTTP for matomo131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 100150 bytes in 0.506 second response time [02:07:51] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17684 bytes in 0.130 second response time [02:07:55] RECOVERY - cp30 Stunnel HTTP for mon141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 37532 bytes in 0.357 second response time [02:08:06] RECOVERY - matomo131 HTTPS on matomo131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.131 second response time [02:09:23] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.11, 1.64, 1.94 [02:11:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.30, 8.61, 10.03 [02:11:20] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.07, 1.81, 1.96 [02:13:16] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.68, 1.67, 1.89 [02:14:20] !log db112: MariaDB [(none)]> set global read_only = 0; [02:14:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:17:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.37, 1.43, 1.65 [02:19:05] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.39, 1.70, 1.79 [02:20:52] MacFan4000, Reception123: all APT alerts resolved now. [02:21:27] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.67, 2.08, 1.87 [02:22:45] I wonder if it will be possible to fix the issues on cloud13/14 with smart and ipmi [02:22:53] (At some point) [02:23:22] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.11, 1.77, 1.79 [02:23:27] But awesome, it’s been so long since we’ve had this few alerts in icinga [02:27:12] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.66, 1.58, 1.70 [02:31:03] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.65, 2.07, 1.87 [02:32:40] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.00, 1.66, 1.93 [02:32:58] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.04, 1.63, 1.73 [02:34:53] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.20, 2.25, 1.97 [02:38:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.09, 1.80, 1.87 [02:41:48] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.03, 10.31, 8.50 [02:42:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.49, 1.47, 1.69 [02:43:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.00, 9.31, 8.34 [02:46:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/609565025e38...ba57660cb98c [02:46:37] [url] Comparing 609565025e38...ba57660cb98c · miraheze/mw-config · GitHub | github.com [02:46:37] [02miraheze/mw-config] 07Universal-Omega 03ba57660 - T9692: configure `$wgDefaultUserOptions` for crashspyrowiki and segawiki [02:46:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.93, 1.34, 1.63 [02:47:47] miraheze/mw-config - Universal-Omega the build passed. [02:56:36] !log [@test131] starting deploy of {'config': True} to all [02:56:37] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [02:56:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:56:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:57:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.75, 10.27, 9.56 [02:59:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.52, 11.01, 9.87 [03:01:25] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.83, 1.67, 1.62 [03:01:37] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.65, 10.14, 9.70 [03:05:15] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.42, 2.03, 1.77 [03:05:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.67, 10.52, 10.00 [03:07:10] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.29, 1.76, 1.70 [03:07:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.19, 9.63, 9.76 [03:09:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.16, 1.78, 1.74 [03:09:21] !log [@mwtask141] starting deploy of {'config': True} to all [03:09:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:09:30] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 9s [03:09:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:11:00] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.09, 1.54, 1.64 [03:11:00] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.95, 1.52, 1.65 [03:14:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.67, 10.30, 9.33 [03:16:41] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.02, 9.06, 8.98 [03:23:40] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.43, 1.76, 1.68 [03:25:25] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.53, 2.28, 1.85 [03:25:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.48, 1.98, 1.77 [03:27:33] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.34, 1.66, 1.67 [03:29:15] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.18, 1.75, 1.74 [03:31:10] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.40, 1.58, 1.67 [03:33:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.08, 1.90, 1.84 [03:41:10] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.06, 1.43, 1.65 [03:45:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.41, 1.77, 1.77 [03:48:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [03:48:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.31, 1.52, 1.67 [03:51:35] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.14, 1.98, 1.66 [03:52:53] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.78, 1.85, 1.79 [03:53:30] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.15, 1.60, 1.56 [03:56:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:56:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.18, 1.83, 1.78 [04:01:11] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 19.49, 8.82, 4.17 [04:02:35] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.60, 1.68, 1.73 [04:04:32] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.09, 1.82, 1.77 [04:18:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.28, 1.80, 1.93 [04:24:09] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.71, 2.08, 1.97 [04:24:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.63, 1.32, 2.00 [04:26:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.72, 1.99, 1.96 [04:26:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [04:30:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.77, 1.03, 1.66 [04:34:09] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.71, 1.67, 1.75 [04:38:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.64, 1.81, 1.80 [04:39:40] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.92, 1.87, 1.77 [04:41:35] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.06, 1.93, 1.80 [04:42:09] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.98, 1.38, 1.63 [04:47:20] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.46, 1.79, 1.82 [04:50:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:51:10] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.06, 1.95, 1.88 [04:53:05] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.68, 1.79, 1.83 [04:56:55] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.29, 1.45, 1.67 [05:00:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.76, 2.99, 2.28 [05:10:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.94, 1.73, 2.00 [05:12:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [05:18:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.08, 1.30, 1.69 [05:21:47] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=rosettacodewiki --summary="Imported from rosettacode.org" --user=Miraheze_maintenance_script --search-recursively /home/reception/rosettacode [05:21:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:24:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.02, 1.93, 1.67 [05:26:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.30, 1.74, 1.64 [05:28:40] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 9.63, 6.40, 3.69 [05:28:49] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=darkwitchwiki --summary="Imported from darkwitch.fandom.com" --user=Miraheze_maintenance_script --search-recursively /home/reception/darkwitch [05:29:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:32:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.23, 1.57, 1.62 [05:34:33] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=milkywayidlewiki --summary="Imported from milkywayidle.fandom.com" --search-recursively /home/reception/milkywayidle [05:34:37] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 2.78, 5.62, 4.37 [05:34:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:36:36] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.46, 4.09, 3.95 [05:40:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:42:28] [02puppet] 07Universal-Omega closed pull request 03#2791: Update typos - 13https://github.com/miraheze/puppet/pull/2791 [05:42:29] [url] Page not found · GitHub · GitHub | github.com [05:42:30] [02miraheze/puppet] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [05:42:31] [02puppet] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/puppet [05:42:32] ... [05:42:33] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±6] 13https://github.com/miraheze/puppet/compare/c6856e9317de...32895eceaca1 [05:42:33] [url] Comparing c6856e9317de...32895eceaca1 · miraheze/puppet · GitHub | github.com [05:42:34] [02miraheze/puppet] 07Universal-Omega 0332895ec - Update typos (#2791) [05:43:17] MacFan4000: yeah it would be nice to get those fixed as well. [05:48:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.26, 1.78, 1.56 [05:50:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [05:50:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.47, 1.64, 1.54 [05:53:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.58, 1.91, 1.71 [06:01:34] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.62, 1.63, 1.65 [06:04:20] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 5.95, 4.39, 3.10 [06:06:19] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.69, 4.11, 3.16 [06:09:22] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.07, 2.06, 1.82 [06:09:30] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.92, 1.73, 1.54 [06:09:45] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.83, 3.13, 2.08 [06:11:25] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 5.79, 2.89, 1.96 [06:21:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.20, 3.62, 2.52 [06:29:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.31, 3.73, 3.15 [06:31:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.73, 3.02, 2.96 [06:32:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:37:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 1.26, 2.82, 3.68 [06:39:45] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.16, 2.28, 3.37 [06:40:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [06:46:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.86, 1.30, 1.89 [06:48:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 5.10, 2.52, 2.25 [06:54:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.61, 1.75, 2.00 [06:54:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.07, 1.61, 1.92 [06:56:09] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.26, 1.79, 1.97 [06:58:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.54, 1.60, 1.88 [07:00:09] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.09, 1.70, 1.87 [07:02:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.18, 1.43, 1.75 [07:04:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.78, 1.24, 1.65 [07:08:09] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.29, 1.61 [07:18:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.06, 1.60, 1.53 [07:20:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.89, 1.69, 1.57 [07:22:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.37, 1.55, 1.53 [07:26:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:28:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.32, 1.78, 1.64 [07:32:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [07:32:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.16, 1.53, 1.58 [08:18:13] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:23:11] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.58, 2.20, 1.70 [08:24:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [08:27:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.66, 1.81, 1.65 [08:29:00] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.05, 1.77, 1.65 [08:30:56] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.10, 1.56, 1.59 [09:02:55] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - fc.songcontests.eu All nameservers failed to answer the query. [09:05:02] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.82, 1.89, 1.68 [09:06:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.03, 1.56, 1.58 [09:09:07] PROBLEM - wiki.thunis.eu - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.thunis.eu. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:09:50] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mineland.eu All nameservers failed to answer the query. [09:10:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:13:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.60, 1.95, 1.75 [09:15:10] PROBLEM - wiki.hsins.eu - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query hsins.eu. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:15:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.80, 1.55, 1.63 [09:16:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [09:19:18] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.98, 1.63, 1.34 [09:21:13] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.29, 1.97, 1.50 [09:21:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.69, 2.21, 1.89 [09:23:08] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.95, 1.65, 1.44 [09:23:34] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.36, 1.90, 1.82 [09:32:20] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query fc.songcontests.eu. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:33:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.39, 2.09, 1.85 [09:35:12] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.66, 1.93, 1.83 [09:37:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.82, 1.55, 1.70 [09:38:28] RECOVERY - wiki.thunis.eu - reverse DNS on sslhost is OK: SSL OK - wiki.thunis.eu reverse DNS resolves to cp30.miraheze.org - CNAME OK [09:38:37] RECOVERY - wiki.mineland.eu - reverse DNS on sslhost is OK: SSL OK - wiki.mineland.eu reverse DNS resolves to cp30.miraheze.org - CNAME OK [09:44:16] RECOVERY - wiki.hsins.eu - reverse DNS on sslhost is OK: SSL OK - wiki.hsins.eu reverse DNS resolves to cp31.miraheze.org - CNAME OK [09:47:53] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=sctoolszhwiki --no-updates --username-prefix=en /mnt/mediawiki-static/metawiki/ImportDump/sctoolszhwiki-20220825090625.xml (START) [09:48:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:51:58] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.75, 1.56, 1.41 [09:52:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.96, 2.07, 1.70 [09:53:53] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.82, 1.37, 1.36 [09:54:38] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.55, 1.81, 1.65 [09:56:35] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.76, 1.40, 1.52 [10:01:45] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - fc.songcontests.eu All nameservers failed to answer the query. [10:04:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:08:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [10:31:09] RECOVERY - fc.songcontests.eu - reverse DNS on sslhost is OK: SSL OK - fc.songcontests.eu reverse DNS resolves to cp30.miraheze.org - CNAME OK [10:39:15] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.39, 1.79, 1.49 [10:41:10] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.67, 1.71, 1.49 [10:41:26] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:43:05] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.80, 1.41, 1.41 [10:50:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.90, 1.57, 1.41 [10:52:23] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.42, 2.27, 1.72 [10:52:41] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=sctoolszhwiki --no-updates --username-prefix=en /mnt/mediawiki-static/metawiki/ImportDump/sctoolszhwiki-20220825090625.xml (END - exit=0) [10:52:42] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=sctoolszhwiki (START) [10:52:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:52:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.05, 1.32, 1.34 [10:52:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:56:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:56:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.15, 1.77, 1.65 [11:00:09] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.71, 2.23, 1.86 [11:02:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [11:10:17] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [11:10:29] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/bcd484139844...1ee008db3e2e [11:10:29] [url] Comparing bcd484139844...1ee008db3e2e · miraheze/WikiDiscover · GitHub | github.com [11:10:30] [02miraheze/WikiDiscover] 07translatewiki 031ee008d - Localisation updates from https://translatewiki.net. [11:10:31] [url] Main page - translatewiki.net | translatewiki.net [11:10:32] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/c1d7d0fe31e2...d9d2ec5a200b [11:10:32] [url] Comparing c1d7d0fe31e2...d9d2ec5a200b · miraheze/ManageWiki · GitHub | github.com [11:10:33] [02miraheze/ManageWiki] 07translatewiki 03d9d2ec5 - Localisation updates from https://translatewiki.net. [11:10:34] [url] Main page - translatewiki.net | translatewiki.net [11:10:35] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/5afdda732657...fdf83f99471e [11:10:35] [url] Comparing 5afdda732657...fdf83f99471e · miraheze/MirahezeMagic · GitHub | github.com [11:10:36] [02miraheze/MirahezeMagic] 07translatewiki 03fdf83f9 - Localisation updates from https://translatewiki.net. [11:10:37] [url] Main page - translatewiki.net | translatewiki.net [11:14:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.30, 1.63, 1.88 [11:15:18] miraheze/WikiDiscover - translatewiki the build passed. [11:15:59] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.56, 2.39 [11:16:06] miraheze/ManageWiki - translatewiki the build passed. [11:17:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.78, 2.94, 2.31 [11:18:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.80, 1.98, 1.57 [11:20:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.77, 1.82, 1.55 [11:24:04] miraheze/MirahezeMagic - translatewiki the build passed. [11:24:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.21, 1.69, 1.57 [11:28:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.08, 1.37, 1.62 [11:32:55] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query fc.songcontests.eu. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:35:48] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mineland.eu. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:38:45] PROBLEM - wiki.thunis.eu - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query thunis.eu. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:39:07] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp31.miraheze.org - NS RECORDS OK [11:44:40] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 6.03, 4.29, 2.36 [11:45:48] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.60, 1.76, 1.64 [11:48:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:49:41] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.20, 1.54, 1.59 [11:52:18] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.55, 1.88, 1.66 [11:53:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.02, 2.06, 1.81 [11:54:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [11:56:08] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.12, 1.55, 1.59 [11:56:20] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.16, 3.89, 3.36 [11:57:29] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.23, 1.80, 1.78 [11:58:17] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.08, 4.45, 3.62 [12:02:20] RECOVERY - fc.songcontests.eu - reverse DNS on sslhost is OK: SSL OK - fc.songcontests.eu reverse DNS resolves to cp30.miraheze.org - CNAME OK [12:04:06] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 1.77, 3.48, 3.49 [12:04:36] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mineland.eu All nameservers failed to answer the query. [12:06:03] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.30, 3.96, 3.65 [12:07:11] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.18, 1.54, 1.68 [12:10:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.40, 9.49, 8.31 [12:11:06] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.94, 1.80, 1.75 [12:12:55] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.98, 8.85, 8.22 [12:13:03] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.21, 1.58, 1.68 [12:16:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.58, 1.70, 1.72 [12:18:54] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.22, 1.52, 1.65 [12:22:50] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.28, 1.86, 1.75 [12:24:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.60, 1.81, 1.74 [12:26:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.90, 1.47, 1.62 [12:30:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.54, 2.35, 1.90 [12:32:18] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.23, 1.99, 1.82 [12:33:24] RECOVERY - wiki.mineland.eu - reverse DNS on sslhost is OK: SSL OK - wiki.mineland.eu reverse DNS resolves to cp30.miraheze.org - CNAME OK [12:37:25] RECOVERY - wiki.thunis.eu - reverse DNS on sslhost is OK: SSL OK - wiki.thunis.eu reverse DNS resolves to cp31.miraheze.org - CNAME OK [12:38:03] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.96, 1.43, 1.63 [12:39:23] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.98, 1.81, 1.68 [12:40:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:41:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.45, 1.67, 1.64 [12:41:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.47, 11.51, 9.69 [12:42:38] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=sctoolszhwiki (END - exit=0) [12:42:39] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=sctoolszhwiki --active --update (END - exit=0) [12:42:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:42:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:43:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.03, 10.24, 9.44 [12:43:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.44, 1.80, 1.77 [12:44:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [12:45:14] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.58, 2.03, 1.78 [12:45:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.92, 9.11, 9.13 [12:47:39] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.36, 2.07, 1.88 [12:53:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 18.83, 12.93, 10.17 [12:53:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 22.70, 13.62, 10.68 [12:55:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.23, 4.58, 3.04 [12:55:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 0.17, 0.93, 3.67 [12:57:45] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 0.40, 0.76, 3.27 [12:59:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.76, 11.37, 10.58 [12:59:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.24, 3.94, 3.13 [13:01:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 19.12, 13.80, 11.52 [13:01:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 4.08, 3.28 [13:05:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.73, 3.82, 3.40 [13:07:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.41, 11.59, 11.39 [13:09:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.93, 3.13, 3.23 [13:11:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.84, 11.85, 11.42 [13:13:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.68, 11.14, 11.20 [13:19:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.61, 10.16, 11.75 [13:21:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.11, 3.04 [13:25:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.69, 9.00, 9.98 [13:25:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.96, 2.89, 2.99 [13:29:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.83, 8.59, 9.95 [13:31:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 19.32, 13.27, 11.29 [13:34:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:35:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.69, 10.84, 10.79 [13:38:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [13:39:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.89, 11.01, 10.79 [13:41:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.51, 11.41, 10.98 [13:41:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.13, 10.13, 9.78 [13:44:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.37, 1.68, 1.99 [13:45:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.50, 10.04, 9.86 [13:48:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.68, 2.03, 2.04 [13:50:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.75, 1.93, 2.00 [13:57:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 19.85, 14.23, 11.98 [13:57:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.39, 11.86, 10.48 [13:59:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.43, 11.78, 10.66 [14:01:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.15, 11.72, 11.44 [14:01:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.81, 12.63, 11.13 [14:03:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.69, 12.10, 11.56 [14:03:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.73, 11.44, 10.87 [14:04:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.62, 1.91, 1.88 [14:05:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.05, 11.51, 11.40 [14:05:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.66, 11.81, 11.06 [14:06:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.35, 1.61, 1.77 [14:07:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.56, 11.82, 11.17 [14:09:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.46, 13.08, 12.03 [14:09:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 19.78, 14.42, 12.19 [14:10:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.67, 2.05, 1.87 [14:14:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.16, 1.72, 1.79 [14:16:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.78, 2.27, 1.99 [14:18:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.43, 1.95, 1.90 [14:20:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.44, 2.43, 2.10 [14:26:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:26:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.53, 3.46, 2.93 [14:30:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [14:39:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.45, 3.83, 3.69 [14:47:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.59, 3.02, 3.33 [15:12:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.66, 3.72, 3.24 [15:16:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.22, 4.66, 3.21 [15:18:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:20:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.91, 3.49, 3.09 [15:22:24] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.02, 2.88, 2.90 [15:22:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.31, 3.85, 3.70 [15:24:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [15:30:07] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.29, 2.75, 3.23 [15:36:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.11, 1.67, 2.00 [15:48:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 4.43, 2.53, 2.12 [16:12:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:16:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [16:23:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.45, 9.56, 11.92 [16:29:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.58, 11.82, 12.07 [16:31:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.42, 11.27, 11.83 [16:37:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.38, 9.57, 11.54 [16:39:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.43, 10.95, 11.79 [16:41:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.21, 10.95, 11.72 [16:41:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.89, 11.26, 11.18 [16:43:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.75, 9.63, 10.60 [16:44:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.64, 1.83, 1.98 [16:46:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.23, 2.03, 2.04 [16:47:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.10, 10.88, 11.21 [16:49:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.46, 10.92, 10.77 [16:51:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.27, 11.15, 10.88 [16:54:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.14, 1.79, 1.98 [16:55:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.94, 13.23, 11.67 [16:59:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.24, 3.65, 3.02 [17:01:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.81, 3.34, 2.99 [17:04:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:04:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.17, 1.68, 1.80 [17:06:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.72, 1.64, 1.77 [17:08:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [17:08:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.84, 1.40, 1.67 [17:18:33] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.99, 2.11, 1.90 [17:28:08] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.94, 2.00, 1.96 [17:30:03] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.46, 2.14, 2.02 [17:33:53] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.47, 1.86, 1.94 [17:35:48] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.32, 2.06, 2.00 [17:37:43] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.73, 1.96, 1.98 [17:39:38] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.71, 2.28, 2.09 [17:43:15] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=artmechanicumwiki (END - exit=65280) [17:43:17] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=artmechanicumwiki deploy-mediawiki --files=../mediawiki/w/extensions/SemanticMediaWiki/artmechanicumwiki.smw.json --servers=all (END - exit=256) [17:43:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:43:28] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.28, 1.86, 1.97 [17:43:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:49:13] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 4.14, 2.60, 2.19 [17:51:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.72, 10.20, 11.70 [17:51:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.93, 10.12, 11.73 [17:53:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.36, 13.02, 12.54 [17:53:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.16, 12.93, 12.58 [17:56:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:00:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [18:01:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.40, 11.34, 11.97 [18:05:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.12, 11.33, 11.80 [18:21:20] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/96fa19e4752d [18:21:22] [02miraheze/puppet] 07Universal-Omega 0396fa19e - roundcubemail: upgrade to 1.6.0 [18:21:23] [02puppet] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/puppet [18:21:24] [url] Page not found · GitHub · GitHub | github.com [18:21:25] [02puppet] 07Universal-Omega opened pull request 03#2792: roundcubemail: upgrade to 1.6.0 - 13https://github.com/miraheze/puppet/pull/2792 [18:21:25] [url] Page not found · GitHub · GitHub | github.com [18:28:15] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/96fa19e4752d...a23a50b1d0ba [18:28:15] [url] Comparing 96fa19e4752d...a23a50b1d0ba · miraheze/puppet · GitHub | github.com [18:28:16] [02miraheze/puppet] 07Universal-Omega 03a23a50b - Update config.inc.php.erb [18:28:18] [02puppet] 07Universal-Omega synchronize pull request 03#2792: roundcubemail: upgrade to 1.6.0 - 13https://github.com/miraheze/puppet/pull/2792 [18:28:18] [url] Page not found · GitHub · GitHub | github.com [18:37:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.51, 3.13 [18:38:59] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.58, 4.45, 3.53 [18:42:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.61, 3.47, 3.34 [18:44:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.08, 3.85, 3.48 [18:46:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.82, 3.49, 3.39 [18:48:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:48:35] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.96, 3.18, 3.28 [18:49:54] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a23a50b1d0ba...11cc0f2b0f92 [18:49:55] [url] Comparing a23a50b1d0ba...11cc0f2b0f92 · miraheze/puppet · GitHub | github.com [18:49:56] [02miraheze/puppet] 07Universal-Omega 0311cc0f2 - Update init.pp [18:49:57] [02puppet] 07Universal-Omega synchronize pull request 03#2792: roundcubemail: upgrade to 1.6.0 - 13https://github.com/miraheze/puppet/pull/2792 [18:49:58] [url] Page not found · GitHub · GitHub | github.com [18:52:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [18:56:40] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/32895eceaca1...f2f4689599aa [18:56:40] [url] Comparing 32895eceaca1...f2f4689599aa · miraheze/puppet · GitHub | github.com [18:56:41] [02miraheze/puppet] 07Universal-Omega 03f2f4689 - reports: remove recurse_submodules from git clone [19:35:29] [02puppet] 07Universal-Omega closed pull request 03#2792: roundcubemail: upgrade to 1.6.0 - 13https://github.com/miraheze/puppet/pull/2792 [19:35:29] [url] Page not found · GitHub · GitHub | github.com [19:35:30] [02miraheze/puppet] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:35:32] [02puppet] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/puppet [19:35:32] ... [19:35:33] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/f2f4689599aa...88e679959e65 [19:35:34] [url] Comparing f2f4689599aa...88e679959e65 · miraheze/puppet · GitHub | github.com [19:35:35] [02miraheze/puppet] 07Universal-Omega 0388e6799 - roundcubemail: upgrade to 1.6.0 (#2792) [19:37:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.07, 4.80, 3.50 [19:40:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:43:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.50, 3.83, 3.53 [19:46:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [19:47:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.74, 3.21, 3.35 [19:54:19] PROBLEM - mon141 APT on mon141 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:54:35] PROBLEM - db101 APT on db101 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (1 critical updates). [19:55:29] PROBLEM - prometheus131 APT on prometheus131 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:55:29] PROBLEM - mem131 APT on mem131 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:56:44] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:57:18] PROBLEM - matomo131 APT on matomo131 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:58:54] PROBLEM - test131 APT on test131 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:59:27] PROBLEM - gluster121 APT on gluster121 is CRITICAL: APT CRITICAL: 15 packages available for upgrade (1 critical updates). [19:59:35] PROBLEM - es131 APT on es131 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [19:59:42] PROBLEM - graylog121 APT on graylog121 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [19:59:46] PROBLEM - reports121 APT on reports121 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:00:26] PROBLEM - mail121 webmail.miraheze.org HTTPS on mail121 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 289 bytes in 0.013 second response time [20:00:57] PROBLEM - gluster102 APT on gluster102 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (1 critical updates). [20:01:12] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.36, 6.00, 3.51 [20:02:04] PROBLEM - db131 APT on db131 is CRITICAL: APT CRITICAL: 11 packages available for upgrade (1 critical updates). [20:02:21] RECOVERY - mail121 webmail.miraheze.org HTTPS on mail121 is OK: HTTP OK: HTTP/1.1 200 OK - 1213 bytes in 0.045 second response time [20:02:25] PROBLEM - db112 APT on db112 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:03:18] PROBLEM - es141 APT on es141 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [20:03:23] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.04, 3.79, 3.41 [20:03:28] PROBLEM - mw142 APT on mw142 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:04:57] PROBLEM - bast101 APT on bast101 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:04:58] PROBLEM - mw122 APT on mw122 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:05:47] PROBLEM - ldap141 APT on ldap141 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:07:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.71, 3.66, 3.31 [20:07:36] PROBLEM - mw141 APT on mw141 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:08:38] PROBLEM - mw132 APT on mw132 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:09:06] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.47, 2.91, 3.08 [20:09:57] PROBLEM - puppet141 APT on puppet141 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:10:09] PROBLEM - db141 APT on db141 is CRITICAL: APT CRITICAL: 11 packages available for upgrade (1 critical updates). [20:10:42] PROBLEM - phab121 APT on phab121 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:11:50] PROBLEM - mw121 APT on mw121 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:11:53] PROBLEM - mw131 APT on mw131 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:12:17] PROBLEM - cloud10 APT on cloud10 is CRITICAL: APT CRITICAL: 50 packages available for upgrade (1 critical updates). [20:12:17] PROBLEM - cloud12 APT on cloud12 is CRITICAL: APT CRITICAL: 59 packages available for upgrade (1 critical updates). [20:12:21] PROBLEM - mwtask141 APT on mwtask141 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:12:47] PROBLEM - bast121 APT on bast121 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:12:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.51, 3.71, 3.70 [20:13:08] PROBLEM - gluster122 APT on gluster122 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (1 critical updates). [20:13:41] PROBLEM - mail121 APT on mail121 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:16:09] PROBLEM - gluster101 APT on gluster101 is CRITICAL: APT CRITICAL: 15 packages available for upgrade (1 critical updates). [20:16:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.16, 4.12, 3.84 [20:18:19] PROBLEM - jobchron121 APT on jobchron121 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:20:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 3.79, 3.77 [20:20:48] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:21:24] PROBLEM - cp31 APT on cp31 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:21:27] PROBLEM - db121 APT on db121 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (1 critical updates). [20:22:05] PROBLEM - mem101 APT on mem101 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:22:35] PROBLEM - cloud13 APT on cloud13 is CRITICAL: APT CRITICAL: 15 packages available for upgrade (1 critical updates). [20:22:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 12.43, 6.65, 4.79 [20:22:55] PROBLEM - cp30 APT on cp30 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:23:18] PROBLEM - cloud11 APT on cloud11 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:23:47] PROBLEM - cloud14 APT on cloud14 is CRITICAL: APT CRITICAL: 15 packages available for upgrade (1 critical updates). [20:23:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.98, 6.33, 4.32 [20:25:21] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:19] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.686 second response time [20:29:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.95, 3.97, 3.95 [20:31:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.95, 4.31, 4.07 [20:33:51] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.37, 3.44, 3.77 [20:34:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:34:36] PROBLEM - mail121 Puppet on mail121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_roundcubemail] [20:37:48] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.88, 2.50, 3.30 [20:41:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.66, 3.10, 3.96 [20:42:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [20:45:38] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.15, 3.76 [20:46:37] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:49:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.68, 3.50, 3.77 [20:51:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.60, 3.64, 3.77 [20:53:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.36, 3.50, 3.71 [20:56:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:57:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.34, 4.37, 3.96 [21:00:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [21:00:55] !log upgrade roundcubemail on mail121 [21:01:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:01:46] RECOVERY - cloud14 APT on cloud14 is OK: APT OK: 14 packages available for upgrade (0 critical updates). [21:01:56] RECOVERY - puppet141 APT on puppet141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:01:59] RECOVERY - ldap141 APT on ldap141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:02:03] RECOVERY - db131 APT on db131 is OK: APT OK: 10 packages available for upgrade (0 critical updates). [21:02:04] RECOVERY - mem101 APT on mem101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:02:13] RECOVERY - db141 APT on db141 is OK: APT OK: 10 packages available for upgrade (0 critical updates). [21:02:14] !log upgrade the zlib1g package on all servers [21:02:15] RECOVERY - jobchron121 APT on jobchron121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:02:15] RECOVERY - cloud10 APT on cloud10 is OK: APT OK: 49 packages available for upgrade (0 critical updates). [21:02:24] RECOVERY - db112 APT on db112 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:02:25] RECOVERY - cp30 APT on cp30 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:02:25] RECOVERY - mon141 APT on mon141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:02:30] RECOVERY - reports121 APT on reports121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:02:34] RECOVERY - cloud13 APT on cloud13 is OK: APT OK: 14 packages available for upgrade (0 critical updates). [21:02:35] RECOVERY - db101 APT on db101 is OK: APT OK: 23 packages available for upgrade (0 critical updates). [21:02:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:02:40] RECOVERY - phab121 APT on phab121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:02:44] RECOVERY - ns1 APT on ns1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:02:46] RECOVERY - bast121 APT on bast121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:02:47] RECOVERY - ns2 APT on ns2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:02:57] RECOVERY - bast101 APT on bast101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:02:57] RECOVERY - gluster102 APT on gluster102 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [21:03:04] RECOVERY - cloud12 APT on cloud12 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [21:03:08] RECOVERY - gluster122 APT on gluster122 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [21:03:15] RECOVERY - cp31 APT on cp31 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:03:15] RECOVERY - gluster101 APT on gluster101 is OK: APT OK: 14 packages available for upgrade (0 critical updates). [21:03:17] RECOVERY - cloud11 APT on cloud11 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:03:17] RECOVERY - matomo131 APT on matomo131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:03:18] RECOVERY - es141 APT on es141 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:03:22] RECOVERY - gluster121 APT on gluster121 is OK: APT OK: 14 packages available for upgrade (0 critical updates). [21:03:24] RECOVERY - db121 APT on db121 is OK: APT OK: 23 packages available for upgrade (0 critical updates). [21:03:28] RECOVERY - prometheus131 APT on prometheus131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:03:28] RECOVERY - mem131 APT on mem131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:03:36] RECOVERY - es131 APT on es131 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:03:40] RECOVERY - graylog121 APT on graylog121 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:03:40] RECOVERY - mail121 APT on mail121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:06:52] RECOVERY - test131 APT on test131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:06:57] RECOVERY - mw122 APT on mw122 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:07:20] !log also upgrade the zlib1g-dev package on mw*, mwtask141, and test131 [21:07:27] RECOVERY - mw142 APT on mw142 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:07:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:07:35] RECOVERY - mw141 APT on mw141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:07:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.88, 3.21, 3.75 [21:07:53] RECOVERY - mw131 APT on mw131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:07:55] RECOVERY - mw121 APT on mw121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:08:20] RECOVERY - mwtask141 APT on mwtask141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:08:38] RECOVERY - mw132 APT on mw132 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:11:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.49, 4.18, 4.04 [21:13:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.01, 3.77, 3.90 [21:19:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.74, 2.60, 3.34 [21:25:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.67, 3.22, 3.32 [21:26:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:27:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.41, 3.37 [21:29:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.66, 3.61, 3.45 [21:33:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.95, 3.21, 3.31 [21:38:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [21:41:45] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.22, 3.53, 2.49 [21:45:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.85, 3.85, 2.88 [21:53:45] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.87, 3.09, 2.98 [22:07:57] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/d5aef05fe79e [22:07:59] [02miraheze/puppet] 07Universal-Omega 03d5aef05 - roundcubemail: update config and synchronise comments and organisation from upstream [22:08:00] [02puppet] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/puppet [22:08:01] [url] Page not found · GitHub · GitHub | github.com [22:08:08] [02puppet] 07Universal-Omega opened pull request 03#2793: roundcubemail: update config and synchronise comments and organisation from upstream - 13https://github.com/miraheze/puppet/pull/2793 [22:08:08] [url] Page not found · GitHub · GitHub | github.com [22:09:59] PROBLEM - en.clockupwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.clockupwiki.org' expires in 15 day(s) (Sat 10 Sep 2022 21:43:05 GMT +0000). [22:12:07] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9cbc83ed1d29...e7cf9de5070d [22:12:08] [url] Comparing 9cbc83ed1d29...e7cf9de5070d · miraheze/ssl · GitHub | github.com [22:12:09] [02miraheze/ssl] 07MirahezeSSLBot 03e7cf9de - Bot: Update SSL cert for en.clockupwiki.org [22:14:34] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.95, 1.78, 1.95 [22:14:58] PROBLEM - en.clockupwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.clockupwiki.org' expires in 15 day(s) (Sat 10 Sep 2022 21:43:05 GMT +0000). [22:15:07] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:16:47] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/88e679959e65...63f5abeee270 [22:16:47] [url] Comparing 88e679959e65...63f5abeee270 · miraheze/puppet · GitHub | github.com [22:16:48] [02miraheze/puppet] 07Universal-Omega 0363f5abe - icingaweb2: fix typo in backend name [22:18:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 4.07, 2.52, 2.18 [22:18:38] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e7cf9de5070d...c713aa0247bd [22:18:39] [url] Comparing e7cf9de5070d...c713aa0247bd · miraheze/ssl · GitHub | github.com [22:18:40] [02miraheze/ssl] 07MirahezeSSLBot 03c713aa0 - Bot: Update SSL cert for en.clockupwiki.org [22:21:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.40, 4.76, 3.77 [22:29:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.13, 3.56, 3.63 [22:33:25] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.49, 2.98, 3.38 [22:41:38] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.68, 1.67, 1.98 [22:43:41] RECOVERY - en.clockupwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.clockupwiki.org' will expire on Wed 23 Nov 2022 21:18:33 GMT +0000. [22:47:33] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.52, 1.81, 1.91 [22:49:32] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.19, 1.59, 1.82 [22:51:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ba57660cb98c...2216a2ca81a1 [22:51:11] [url] Comparing ba57660cb98c...2216a2ca81a1 · miraheze/mw-config · GitHub | github.com [22:51:11] [02miraheze/mw-config] 07Universal-Omega 032216a2c - T9551: update `$wgPageImagesScores` for houkai2ndwiki [22:52:09] miraheze/mw-config - Universal-Omega the build passed. [22:53:29] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.53, 1.90, 1.88 [22:55:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/mw-config/commit/56b4f28102a8 [22:55:47] [02miraheze/mw-config] 07Universal-Omega 0356b4f28 - CI: Add typos job [22:55:48] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [22:55:49] [url] Page not found · GitHub · GitHub | github.com [22:55:50] [02mw-config] 07Universal-Omega opened pull request 03#4877: CI: Add typos job - 13https://github.com/miraheze/mw-config/pull/4877 [22:55:51] [url] Page not found · GitHub · GitHub | github.com [22:56:21] !log [@test131] starting deploy of {'config': True} to all [22:56:22] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [22:56:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:56:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:57:11] miraheze/mw-config - Universal-Omega the build passed. [22:57:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/56b4f28102a8...b1c3b2510a0d [22:57:27] [url] Comparing 56b4f28102a8...b1c3b2510a0d · miraheze/mw-config · GitHub | github.com [22:57:28] [02miraheze/mw-config] 07Universal-Omega 03b1c3b25 - Update continuousIntegration.yml [22:57:29] [02mw-config] 07Universal-Omega synchronize pull request 03#4877: CI: Add typos job - 13https://github.com/miraheze/mw-config/pull/4877 [22:57:30] [url] Page not found · GitHub · GitHub | github.com [22:58:18] [02mw-config] 07Universal-Omega synchronize pull request 03#4877: CI: Add typos job - 13https://github.com/miraheze/mw-config/pull/4877 [22:58:19] [url] Page not found · GitHub · GitHub | github.com [22:58:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b1c3b2510a0d...6fae7dbd954b [22:58:20] [url] Comparing b1c3b2510a0d...6fae7dbd954b · miraheze/mw-config · GitHub | github.com [22:58:21] [02miraheze/mw-config] 07Universal-Omega 036fae7db - Update typos [22:58:59] RhinosF1 FYI: ^ I'll add that to config also. [22:59:24] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.96, 1.83, 1.96 [22:59:49] CosmicAlpha: the typos job is cool [22:59:56] Even if it's mainly for my sleepy brain [23:00:03] Which on that note is why I'm going to sleep [23:00:25] [02mw-config] 07Universal-Omega synchronize pull request 03#4877: CI: Add typos job - 13https://github.com/miraheze/mw-config/pull/4877 [23:00:26] [url] Page not found · GitHub · GitHub | github.com [23:00:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6fae7dbd954b...cb003d7bc60e [23:00:27] [url] Comparing 6fae7dbd954b...cb003d7bc60e · miraheze/mw-config · GitHub | github.com [23:00:28] [02miraheze/mw-config] 07Universal-Omega 03cb003d7 - Update continuousIntegration.yml [23:02:06] [02mw-config] 07Universal-Omega synchronize pull request 03#4877: CI: Add typos job - 13https://github.com/miraheze/mw-config/pull/4877 [23:02:06] ... [23:02:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/cb003d7bc60e...f1206e13aa8f [23:02:08] [url] Comparing cb003d7bc60e...f1206e13aa8f · miraheze/mw-config · GitHub | github.com [23:02:09] [02miraheze/mw-config] 07Universal-Omega 03f1206e1 - Update continuousIntegration.yml [23:02:43] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f1206e13aa8f...fcde9cc801ae [23:02:43] [url] Comparing f1206e13aa8f...fcde9cc801ae · miraheze/mw-config · GitHub | github.com [23:02:44] [02miraheze/mw-config] 07Universal-Omega 03fcde9cc - Update typos [23:02:46] [02mw-config] 07Universal-Omega synchronize pull request 03#4877: CI: Add typos job - 13https://github.com/miraheze/mw-config/pull/4877 [23:02:46] [url] Page not found · GitHub · GitHub | github.com [23:03:24] miraheze/mw-config - Universal-Omega the build has errored. [23:03:32] miraheze/mw-config - Universal-Omega the build passed. [23:04:27] [02mw-config] 07Universal-Omega closed pull request 03#4877: CI: Add typos job - 13https://github.com/miraheze/mw-config/pull/4877 [23:04:27] [url] Page not found · GitHub · GitHub | github.com [23:04:28] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [23:04:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/mw-config/compare/2216a2ca81a1...2f1bbbf7aad6 [23:04:31] [url] Comparing 2216a2ca81a1...2f1bbbf7aad6 · miraheze/mw-config · GitHub | github.com [23:04:31] [02miraheze/mw-config] 07Universal-Omega 032f1bbbf - CI: Add typos job (#4877) [23:04:33] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [23:04:33] [url] Page not found · GitHub · GitHub | github.com [23:05:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.14, 2.00, 2.00 [23:07:58] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/290ab7a02c79...3e3485dd1834 [23:07:59] [url] Comparing 290ab7a02c79...3e3485dd1834 · miraheze/puppet · GitHub | github.com [23:08:00] [02miraheze/puppet] 07Universal-Omega 033e3485d - irc: minor update to comments [23:09:16] !log [@mwtask141] starting deploy of {'config': True} to all [23:09:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:09:25] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [23:09:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:09:48] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3e3485dd1834...b4a891e1f45a [23:09:49] [url] Comparing 3e3485dd1834...b4a891e1f45a · miraheze/puppet · GitHub | github.com [23:09:49] [02miraheze/puppet] 07Universal-Omega 03b4a891e - prometheus: fix mediawiki role case [23:10:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.98, 3.90, 3.35 [23:12:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.78, 3.37 [23:14:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.67, 3.67, 3.36 [23:16:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.04, 3.39, 3.29 [23:26:22] !log [@test131] starting deploy of {'config': True} to all [23:26:23] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [23:26:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:26:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:39:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.45, 3.05, 2.10 [23:41:45] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.69, 2.92, 2.18 [23:58:43] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.16, 1.83, 2.00