[00:03:38] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.77, 4.40, 3.35 [00:03:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.25, 7.08, 6.42 [00:05:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.11, 6.88, 6.44 [00:07:36] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.42, 6.78, 6.46 [00:09:40] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [00:11:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.60, 3.89, 3.52 [00:13:23] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.30, 3.82, 3.52 [00:13:40] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [00:14:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.14, 6.90, 6.71 [00:16:23] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.63, 6.34, 6.52 [00:17:17] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.89, 3.86, 3.66 [00:18:42] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.88, 10.18, 9.31 [00:19:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.76, 10.81, 9.99 [00:19:47] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.91, 3.91, 3.37 [00:20:41] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.93, 9.27, 9.08 [00:21:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.43, 4.01, 3.71 [00:21:20] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.05, 10.19, 9.87 [00:22:03] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.15, 5.83, 3.08 [00:23:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.12, 3.33, 3.50 [00:24:04] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 6.80, 6.30, 3.60 [00:25:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.72, 3.59, 3.45 [00:26:57] [02puppet] 07AgentIsai opened pull request 03#3207: betawiki -> metawikibeta in deploy-mediawiki - 13https://github.com/miraheze/puppet/pull/3207 [00:27:39] @Site Reliability Engineers can I get a merge? [00:27:43] PROBLEM - swiftobject121 NTP time on swiftobject121 is WARNING: NTP WARNING: Offset 0.4396731257 secs [00:27:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 9.31, 6.51, 4.56 [00:29:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.84, 10.30, 9.83 [00:29:01] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.06, 3.08, 3.39 [00:32:03] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.27, 5.78, 4.00 [00:32:47] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.26, 10.66, 10.02 [00:33:36] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.08, 11.70, 10.25 [00:36:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.42, 11.34, 10.49 [00:37:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.32, 11.18, 10.42 [00:38:04] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 3.68, 6.85, 5.27 [00:39:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.43, 3.44, 3.92 [00:40:04] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.73, 5.17, 4.85 [00:42:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.10, 12.37, 11.00 [00:43:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.13, 11.29, 10.59 [00:45:14] [02puppet] 07The-Voidwalker closed pull request 03#3207: betawiki -> metawikibeta in deploy-mediawiki - 13https://github.com/miraheze/puppet/pull/3207 [00:45:16] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/efb5c8c82f40...be2842b47123 [00:45:18] [02miraheze/puppet] 07AgentIsai 03be2842b - betawiki -> metawikibeta in deploy-mediawiki (#3207) [00:45:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.98, 10.37, 10.34 [00:45:58] Thanks! [00:46:04] np [00:47:47] !log [macfan@test131] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=metawiki --user=MacFan4000 (END - exit=0) [00:47:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:49:04] !log [macfan@test131] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=metawiki --user=MacFan4000 (END - exit=0) [00:49:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:49:47] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.15, 3.46, 3.62 [00:50:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.98, 11.53, 11.28 [00:51:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.72, 9.15, 9.93 [00:51:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.46, 3.10, 3.45 [00:53:11] !log [macfan@test131] sudo -u www-data php /srv/mediawiki/w/maintenance/resetAuthenticationThrottle.php --wiki=metawiki --login --user=MacFan4000 --ip=2603:7081:a02:10c9::16 (END - exit=0) [00:53:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:53:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.38, 4.06, 3.74 [00:54:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.80, 11.22, 11.13 [00:56:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.08, 10.48, 10.88 [00:58:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.53, 11.28, 11.12 [00:59:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.97, 3.14, 2.77 [00:59:57] !log [agent@test131] starting deploy of {'world': True, 'l10n': True} to test131 [01:00:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:00:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.69, 10.78, 10.97 [01:00:48] RECOVERY - swiftobject121 NTP time on swiftobject121 is OK: NTP OK: Offset 0.007049322128 secs [01:01:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.64, 3.99, 4.00 [01:03:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.61, 3.96, 3.97 [01:05:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.77, 3.39, 3.75 [01:07:04] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.88, 4.09, 3.35 [01:08:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.67, 10.43, 10.55 [01:09:02] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.40, 3.69, 3.29 [01:09:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.65, 2.43, 3.26 [01:10:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.97, 11.14, 10.81 [01:10:58] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.19, 2.77, 3.00 [01:13:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.83, 10.47, 9.97 [01:13:34] !log [macfan@test131] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=betawiki --user=MacFan4000 (END - exit=0) [01:13:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:14:19] !log [agent@test131] finished deploy of {'world': True, 'l10n': True} to test131 - SUCCESS in 861s [01:14:22] !log [macfan@test131] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=metawiki --user=MacFan4000 (END - exit=0) [01:14:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:14:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:17:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.60, 9.82, 9.83 [01:18:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.34, 11.14, 10.95 [01:20:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.70, 10.52, 10.76 [01:21:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.02, 10.32, 10.07 [01:23:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.96, 3.84, 3.28 [01:25:18] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.93, 9.15, 9.67 [01:25:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.90, 3.41, 3.19 [01:27:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.12, 4.42, 3.61 [01:29:16] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.747 second response time [01:29:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.80, 3.51, 3.38 [01:31:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.77, 2.84, 3.14 [01:32:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.49, 8.36, 9.70 [01:33:15] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 788 bytes in 0.144 second response time [01:33:54] the check needs updating as it uses beta.betajeze.org [01:34:01] betaheze* [01:34:36] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:39:14] PROBLEM - test131 MediaWiki Rendering on test131 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 6664 bytes in 0.034 second response time [01:41:56] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 10.53, 16.13, 8.08 [01:43:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.77, 3.55, 3.20 [01:45:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.70, 2.84, 2.98 [01:45:56] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 1.11, 7.80, 6.48 [01:47:56] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 1.58, 5.71, 5.86 [01:48:01] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 14797 MB (11% inode=99%); [01:51:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.92, 3.63, 3.18 [01:53:36] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03betawiki-mentions-replace [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/e6c9fdd1c90a [01:53:39] [02miraheze/mw-config] 07AgentIsai 03e6c9fdd - betawiki -> metawikibeta [01:53:41] [02mw-config] 07AgentIsai created branch 03betawiki-mentions-replace - 13https://github.com/miraheze/mw-config [01:53:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.60, 3.59, 3.22 [01:53:46] [02mw-config] 07AgentIsai opened pull request 03#5212: Change mentions of betawiki to metawikibeta - 13https://github.com/miraheze/mw-config/pull/5212 [01:54:23] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03betawiki-mentions-replace [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e6c9fdd1c90a...7741f7c6c61c [01:54:24] [02miraheze/mw-config] 07github-actions 037741f7c - CI: lint code to MediaWiki standards [01:54:27] [02mw-config] 07github-actions[bot] synchronize pull request 03#5212: Change mentions of betawiki to metawikibeta - 13https://github.com/miraheze/mw-config/pull/5212 [01:55:01] miraheze/mw-config - AgentIsai the build passed. [01:55:13] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.270 second response time [01:55:17] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03betawiki-mentions-replace [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7741f7c6c61c...5538613da3aa [01:55:18] [02miraheze/mw-config] 07AgentIsai 035538613 - Update LocalWiki.php [01:55:19] [02mw-config] 07AgentIsai synchronize pull request 03#5212: Change mentions of betawiki to metawikibeta - 13https://github.com/miraheze/mw-config/pull/5212 [01:55:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.98, 3.39, 3.20 [01:56:22] miraheze/mw-config - AgentIsai the build passed. [01:56:24] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03betawiki-mentions-replace [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5538613da3aa...39e8fee49b1a [01:56:27] [02miraheze/mw-config] 07AgentIsai 0339e8fee - Update LocalSettings.php [01:56:30] [02mw-config] 07AgentIsai synchronize pull request 03#5212: Change mentions of betawiki to metawikibeta - 13https://github.com/miraheze/mw-config/pull/5212 [01:57:09] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.96, 3.29, 2.83 [01:57:21] [02puppet] 07MacFan4000 opened pull request 03#3208: change url for beta monitoring - 13https://github.com/miraheze/puppet/pull/3208 [01:57:24] miraheze/mw-config - AgentIsai the build passed. [01:57:33] [02puppet] 07AgentIsai opened pull request 03#3209: Replace mentions of betawiki to metawikibeta - 13https://github.com/miraheze/puppet/pull/3209 [01:58:46] [02puppet] 07AgentIsai synchronize pull request 03#3209: Replace mentions of betawiki to metawikibeta - 13https://github.com/miraheze/puppet/pull/3209 [01:59:06] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.79, 3.12, 2.82 [01:59:12] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 341 bytes in 0.083 second response time [01:59:12] [02mw-config] 07AgentIsai closed pull request 03#5212: Change mentions of betawiki to metawikibeta - 13https://github.com/miraheze/mw-config/pull/5212 [01:59:15] [02miraheze/mw-config] 07AgentIsai deleted branch 03betawiki-mentions-replace [01:59:18] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/mw-config/compare/bd408945d4d6...05772b1c1a21 [01:59:20] [02miraheze/mw-config] 07AgentIsai 0305772b1 - Change mentions of betawiki to metawikibeta (#5212) [01:59:23] [02mw-config] 07AgentIsai deleted branch 03betawiki-mentions-replace - 13https://github.com/miraheze/mw-config [02:00:12] miraheze/mw-config - AgentIsai the build passed. [02:01:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.65, 3.74, 3.37 [02:03:04] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.03, 4.08, 3.31 [02:05:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.05, 3.91, 3.47 [02:05:56] I'm getting an error that suggests that a db server is in read on;y [02:06:01] only* [02:06:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.91, 3.75, 3.37 [02:07:11] paladox: can you look and unset read-only in mariadb? [02:07:25] or @Orduin [02:07:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.27, 3.10, 3.21 [02:08:55] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.14, 3.20, 3.21 [02:09:05] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.13, 10.87, 9.10 [02:10:09] !log [@mwtask141] starting deploy of {'config': True} to all [02:10:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:10:17] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [02:10:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:12:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.62, 10.84, 9.27 [02:13:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.17, 9.87, 9.09 [02:13:07] db121? [02:13:46] I'm guessing so [02:13:57] done [02:14:16] Thanks [02:14:17] !log [void@db121] set read_only=0 [02:14:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:14:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.74, 11.44, 9.68 [02:15:50] Can an SRE merge please? [02:16:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.54, 11.13, 9.81 [02:19:57] [02puppet] 07The-Voidwalker closed pull request 03#3209: Replace mentions of betawiki to metawikibeta - 13https://github.com/miraheze/puppet/pull/3209 [02:19:58] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/be2842b47123...ee80d6925fbf [02:20:01] [02miraheze/puppet] 07AgentIsai 03ee80d69 - Replace mentions of betawiki to metawikibeta (#3209) [02:20:17] I'm in a button pushing mood tonight :) [02:20:42] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=ffoswiki --no-updates --username-prefix=wikia:ffos-database /home/agent/ImportDump/ffoswiki-20230426020250.xml (START) [02:20:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.06, 12.79, 10.69 [02:20:43] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=ffoswiki --no-updates --username-prefix=wikia:ffos-database /home/agent/ImportDump/ffoswiki-20230426020250.xml (END - exit=256) [02:20:44] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=ffoswiki (START) [02:20:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:20:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:20:56] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=ffoswiki (END - exit=0) [02:20:57] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=ffoswiki --active --update (END - exit=0) [02:20:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:21:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:21:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:22:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.07, 6.46, 6.22 [02:22:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.33, 11.44, 10.46 [02:24:02] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14600 MB (10% inode=99%); [02:24:40] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.04, 6.17, 6.13 [02:24:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.75, 9.85, 10.00 [02:26:38] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.70, 3.56, 3.13 [02:26:52] !log [@test131] starting deploy of {'config': True} to all [02:26:53] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [02:26:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:27:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:30:34] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.56, 3.73, 3.35 [02:30:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.22, 7.74, 6.73 [02:32:32] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.60, 4.12, 3.52 [02:32:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.73, 6.86, 6.53 [02:33:15] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:34:03] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.01, 5.10, 2.97 [02:34:30] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.79, 3.93, 3.51 [02:36:03] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 6.39, 5.95, 3.57 [02:38:34] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1015704572 secs [02:40:24] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.29, 3.00, 3.27 [02:42:19] PROBLEM - test131 HTTPS on test131 is CRITICAL: connect to address 2a10:6740::6:406 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:42:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.85, 7.46, 6.99 [02:42:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.68, 10.00, 9.66 [02:43:07] https://www.irccloud.com/pastebin/7lvuFivU/ [02:43:28] seems we still have an issue with loading things twice [02:44:17] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.01, 3.89, 3.58 [02:44:24] I tried to turn the skin off via the managewiki script, but am stopped by the error [02:44:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.42, 7.23, 6.98 [02:44:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.99, 8.91, 9.31 [02:46:15] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.07, 3.19, 3.36 [02:48:34] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09414634109 secs [02:48:40] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.64, 6.49, 6.75 [02:52:12] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.86, 2.94, 2.36 [02:54:12] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.33, 2.65, 2.32 [02:54:44] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:58:01] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.21, 4.12, 3.51 [02:59:26] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [02:59:58] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.09, 3.27, 3.26 [03:01:17] RECOVERY - db101 Backups SQL on db101 is OK: FILE_AGE OK: /var/log/sql-backup.log is 75 seconds old and 275 bytes [03:03:54] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.29, 5.81, 4.39 [03:04:11] that shouldn't be possible [03:04:16] it worked an hour ago [03:04:37] ExtensionProcessor was removed so how is this regression even happening? [03:05:34] !log [agent@test131] starting deploy of {'world': True, 'l10n': True} to test131 [03:05:35] !log [agent@test131] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [03:05:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:05:35] !log [agent@test131] starting deploy of {'world': True} to test131 [03:05:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:05:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:07:51] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.21, 3.96, 3.96 [03:09:57] huh, beta is down [03:13:40] What changed in the last hour? [03:13:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.69, 2.44, 3.26 [03:13:55] Seems puppet failed [03:14:02] and brought down nginx with it [03:14:42] and I don't have perms in Graylog to check [03:14:44] RIP [03:15:02] do you have root on mw? [03:15:11] I do not [03:15:26] oh, other check would be journalctl -xe, but that's a root thing [03:15:38] Yeah, I tried and got an error heh [03:15:46] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:16:28] there's a lot of that extension processor errors present in the logs [03:16:51] probably from jobrunner, but there's a lot of them [03:19:04] sudo journalctl -xe should work even for mw-admins [03:19:05] running commands from /srv/mediawiki-staging works but not from /srv/mediawiki [03:19:25] I'm guessing that /srv/mediawiki doesn't have the latest MM patch applied for some reason [03:19:48] might not be deployed [03:20:33] nginx config failed [03:20:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [03:23:47] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.67, 3.71, 3.34 [03:28:06] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:29:25] I've disabled Splash manually [03:29:28] but Beta is still down [03:30:26] RECOVERY - cp33 APT on cp33 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [03:31:12] same error still appears when attempting to restart nginx [03:31:33] > nginx: [emerg] duplicate listen options for [::]:443 in /etc/nginx/sites-enabled/opensearch-miraheze-org:2 [03:31:59] !log [macfan@test131] starting deploy of {'folders': 'w/extensions/MirahezeMagic'} to all [03:32:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:33:14] hmm, I know paladox has been using test131 for opensearch stuff [03:34:50] MacFan4000: perhaps disable that config for now? [03:35:41] Issue (and solution) seems to be described in full detail here: [03:35:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.77, 3.53, 3.91 [03:35:57] ah [03:39:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.25, 2.41, 3.37 [03:40:42] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03patch-ipv6only [+0/-0/±9] 13https://github.com/miraheze/puppet/commit/372ca281deef [03:40:45] [02miraheze/puppet] 07The-Voidwalker 03372ca28 - nginx: remove use of ipv6only [03:40:46] [02puppet] 07The-Voidwalker created branch 03patch-ipv6only - 13https://github.com/miraheze/puppet [03:41:14] [02puppet] 07The-Voidwalker opened pull request 03#3210: nginx: remove use of ipv6only - 13https://github.com/miraheze/puppet/pull/3210 [03:42:59] !log [agent@test131] sudo -u www-data php toggleExtension.php --wiki metawikibeta --disable splash [03:43:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:43:15] finally disabled [03:43:57] so to disable an extension when scripts return errors, you remove the extension from the ManageWiki cache and then run toggleExtension and it'll work [03:46:42] !log [macfan@test131] starting deploy of {'force': True, 'folders': 'w/extensions/MirahezeMagic'} to all [03:46:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:47:41] meh, will have to wait until nginx is fixed [03:50:23] PROBLEM - www.zetapedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.zetapedia.com' expires in 15 day(s) (Fri 12 May 2023 03:46:27 GMT +0000). [03:51:23] [02puppet] 07The-Voidwalker commented on pull request 03#3210: nginx: remove use of ipv6only - 13https://github.com/miraheze/puppet/pull/3210#issuecomment-1522736165 [03:51:41] PROBLEM - burnout.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'burnout.wiki' expires in 15 day(s) (Fri 12 May 2023 03:47:06 GMT +0000). [03:51:52] PROBLEM - buildabearwiki.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'buildabearwiki.info' expires in 15 day(s) (Fri 12 May 2023 03:48:11 GMT +0000). [03:52:44] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [03:53:28] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1035896838 secs [03:53:48] PROBLEM - wiki.veloren.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.veloren.net' expires in 15 day(s) (Fri 12 May 2023 03:51:46 GMT +0000). [03:54:14] problem includes that `deferred` is a setting that can only be included once for some reason [03:54:17] PROBLEM - swests.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'swests.org' expires in 15 day(s) (Fri 12 May 2023 03:52:01 GMT +0000). [03:55:20] PROBLEM - wikipedia.yogfront.ooo - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikipedia.yogfront.ooo' expires in 15 day(s) (Fri 12 May 2023 03:46:51 GMT +0000). [03:55:55] this one is more difficult to fix, as there's not necessarily an easy way to tell if another server block has already defined it, and probably isn't a default setting [03:56:11] PROBLEM - aresrocket.ml - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'aresrocket.ml' expires in 15 day(s) (Fri 12 May 2023 03:51:37 GMT +0000). [03:56:20] PROBLEM - wiki.aoemods.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.aoemods.com' expires in 15 day(s) (Fri 12 May 2023 03:49:19 GMT +0000). [03:56:26] PROBLEM - mystiverse.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mystiverse.wiki' expires in 15 day(s) (Fri 12 May 2023 03:49:07 GMT +0000). [03:56:41] PROBLEM - legacygt.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'legacygt.wiki' expires in 15 day(s) (Fri 12 May 2023 03:50:37 GMT +0000). [03:56:47] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d4f1930aa9b7...13e49ae09d98 [03:56:50] [02miraheze/ssl] 07MirahezeSSLBot 0313e49ae - Bot: Update SSL cert for burnout.wiki [03:56:55] PROBLEM - pmwiki.cubestudios.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pmwiki.cubestudios.xyz' expires in 15 day(s) (Fri 12 May 2023 03:49:35 GMT +0000). [03:57:10] [02puppet] 07The-Voidwalker commented on pull request 03#3210: nginx: remove use of ipv6only - 13https://github.com/miraheze/puppet/pull/3210#issuecomment-1522739455 [03:57:30] PROBLEM - wiki.yahyabd.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.yahyabd.xyz' expires in 15 day(s) (Fri 12 May 2023 03:53:00 GMT +0000). [03:57:42] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [03:57:55] PROBLEM - wiki.evscope.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.evscope.org' expires in 15 day(s) (Fri 12 May 2023 03:50:02 GMT +0000). [03:58:15] nginx running through manual edit [03:58:19] RECOVERY - test131 HTTPS on test131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 549 bytes in 0.015 second response time [03:58:48] there's a comment on my task regarding it, though if opensearch isn't necessary on test131, we can ignore it (mostly) [03:59:37] PROBLEM - wiki.scpsharp.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.scpsharp.ga' expires in 15 day(s) (Fri 12 May 2023 03:47:17 GMT +0000). [04:00:07] !log [macfan@test131] starting deploy of {'force': True, 'folders': 'w/extensions/MirahezeMagic'} to all [04:00:08] PROBLEM - pastport.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pastport.org' expires in 15 day(s) (Fri 12 May 2023 03:50:54 GMT +0000). [04:00:08] !log [macfan@test131] finished deploy of {'force': True, 'folders': 'w/extensions/MirahezeMagic'} to all - SUCCESS in 0s [04:00:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:00:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:00:34] PROBLEM - wiki.recaptime.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.recaptime.eu.org' expires in 15 day(s) (Fri 12 May 2023 03:49:54 GMT +0000). [04:00:41] PROBLEM - dragdown.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dragdown.wiki' expires in 15 day(s) (Fri 12 May 2023 03:50:30 GMT +0000). [04:01:05] PROBLEM - iol.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'iol.wiki' expires in 15 day(s) (Fri 12 May 2023 03:52:45 GMT +0000). [04:01:21] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 5.17, 3.97, 3.02 [04:02:24] PROBLEM - datawiki.online - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'datawiki.online' expires in 15 day(s) (Fri 12 May 2023 03:51:53 GMT +0000). [04:02:32] PROBLEM - lgbtqia.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.lgbtqia.wiki' expires in 15 day(s) (Fri 12 May 2023 03:50:21 GMT +0000). [04:02:38] PROBLEM - txt-dex.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'txt-dex.com' expires in 15 day(s) (Fri 12 May 2023 03:46:19 GMT +0000). [04:03:21] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 3.16, 3.87, 3.11 [04:03:28] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.08433008194 secs [04:03:35] PROBLEM - wiki.virtualjet.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.virtualjet.net' expires in 15 day(s) (Fri 12 May 2023 03:48:43 GMT +0000). [04:03:39] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/13e49ae09d98...53323f9a21e2 [04:03:42] [02miraheze/ssl] 07MirahezeSSLBot 0353323f9 - Bot: Update SSL cert for legacygt.wiki [04:04:06] oop, task in my last message should be pull request [04:05:10] PROBLEM - roll-of-arms.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'roll-of-arms.com' expires in 15 day(s) (Fri 12 May 2023 03:52:52 GMT +0000). [04:05:21] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 4.80, 4.22, 3.32 [04:05:31] PROBLEM - wiki.cjgh.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cjgh.xyz' expires in 15 day(s) (Fri 12 May 2023 03:49:43 GMT +0000). [04:06:06] MacFan4000: I'm going to disable all extensions apart from those used on Meta on metawikibeta and going to enable them on testwikibeta instead [04:06:31] PROBLEM - mariopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mariopedia.org' expires in 15 day(s) (Fri 12 May 2023 03:50:46 GMT +0000). [04:06:50] PROBLEM - wiki.ravynos.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ravynos.com' expires in 15 day(s) (Fri 12 May 2023 03:47:42 GMT +0000). [04:06:54] PROBLEM - wikiyri.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikiyri.org' expires in 15 day(s) (Fri 12 May 2023 03:48:20 GMT +0000). [04:07:21] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 2.68, 3.74, 3.26 [04:08:05] PROBLEM - wikicompliance.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikicompliance.com' expires in 15 day(s) (Fri 12 May 2023 03:50:09 GMT +0000). [04:08:38] PROBLEM - persist.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'persist.wiki' expires in 15 day(s) (Fri 12 May 2023 03:51:01 GMT +0000). [04:10:29] PROBLEM - entw.ml - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'entw.ml' expires in 15 day(s) (Fri 12 May 2023 03:46:58 GMT +0000). [04:11:05] PROBLEM - hi.famepedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hi.famepedia.org' expires in 15 day(s) (Fri 12 May 2023 03:46:10 GMT +0000). [04:11:40] PROBLEM - wiki.kachow.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.kachow.org' expires in 15 day(s) (Fri 12 May 2023 03:51:29 GMT +0000). [04:11:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/53323f9a21e2...5dbf1ed31b58 [04:11:48] [02miraheze/ssl] 07MirahezeSSLBot 035dbf1ed - Bot: Update SSL cert for wikipedia.yogfront.ooo [04:12:37] PROBLEM - phytinfo.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'phytinfo.cf' expires in 15 day(s) (Fri 12 May 2023 03:52:38 GMT +0000). [04:12:53] PROBLEM - wiki.ssangyongsports.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ssangyongsports.eu.org' expires in 15 day(s) (Fri 12 May 2023 03:47:29 GMT +0000). [04:13:18] PROBLEM - wiki-tintin.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki-tintin.cf' expires in 15 day(s) (Fri 12 May 2023 03:51:09 GMT +0000). [04:13:21] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 8.74, 5.76, 4.17 [04:13:46] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [04:14:07] PROBLEM - royal-wiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'royal-wiki.org' expires in 15 day(s) (Fri 12 May 2023 03:48:51 GMT +0000). [04:14:11] PROBLEM - squareenixwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'squareenixwiki.com' expires in 15 day(s) (Fri 12 May 2023 03:48:59 GMT +0000). [04:18:09] PROBLEM - bn.wikiyri.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bn.wikiyri.org' expires in 15 day(s) (Fri 12 May 2023 03:51:17 GMT +0000). [04:18:53] PROBLEM - wiki.mostly.vet - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mostly.vet' expires in 15 day(s) (Fri 12 May 2023 03:48:03 GMT +0000). [04:19:07] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5dbf1ed31b58...0906638026fe [04:19:08] [02miraheze/ssl] 07MirahezeSSLBot 030906638 - Bot: Update SSL cert for wiki.scpsharp.ga [04:19:09] PROBLEM - www.lgbtqia.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.lgbtqia.wiki' expires in 15 day(s) (Fri 12 May 2023 03:50:21 GMT +0000). [04:19:17] PROBLEM - en.omniversalis.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.omniversalis.org' expires in 15 day(s) (Fri 12 May 2023 03:52:31 GMT +0000). [04:21:48] PROBLEM - knifepointhorror.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'knifepointhorror.wiki' expires in 15 day(s) (Fri 12 May 2023 03:53:11 GMT +0000). [04:22:08] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.22, 6.37, 3.46 [04:22:33] PROBLEM - evil.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'evil.wiki' expires in 15 day(s) (Fri 12 May 2023 03:52:24 GMT +0000). [04:23:23] PROBLEM - wiki.infomedia.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.infomedia.co.uk' expires in 15 day(s) (Fri 12 May 2023 03:52:13 GMT +0000). [04:24:05] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.38, 5.53, 3.53 [04:24:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0906638026fe...55f20b993530 [04:24:44] [02miraheze/ssl] 07MirahezeSSLBot 0355f20b9 - Bot: Update SSL cert for wiki.kachow.org [04:25:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.88, 3.07, 2.38 [04:25:30] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.57, 3.33, 2.83 [04:26:08] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/55f20b993530...b10fa24f215e [04:26:11] [02miraheze/ssl] 07MirahezeSSLBot 03b10fa24 - Bot: Update SSL cert for evil.wiki [04:27:23] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.14, 3.10, 2.48 [04:29:26] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.68, 3.56, 3.08 [04:30:11]